What's causing the problem is a combination of selfish passengers and the intricacies of how passengers estimate which train will be the fastest to a particular station. In situations like these, where you have two lines taking slightly different routes to the same special station, with trains very near to each other, the inaccuracies of our estimation become apparent.
When a train is stopped at a station and has finished disembarking, each boarding tick (0.83s at normal speed, to sync with the audio) it asks the passengers waiting at the station if they want to board. This kicks off a full pathfind for that passenger (we never reuse paths, as the situation can change quickly). When the pathfind is complete, the passenger has a full path to a destination station (if one exists); board train 1 on the orange line, hop off at circle station 5, wait 2.8s, board train 5 on the red line, etc. If the first step is boarding the train in question, then it hops on, otherwise the train will ask the next passenger. If all the passengers waiting at the station want to board different trains then it'll leave.
The complexity is in the estimation of 'how long in the future will train X leave station Y and arrive at station Z' (we have a function called EstimatedTimeFromStationToStation). Estimating the time for the train that's currently at the station is the easiest, but even then we can't know precisely how long it will take; it doesn't know for sure how many passengers are yet to board, so doesn't know when it will depart. So it's inaccurate before we've even left the first station. In the example you've got above, the star passengers will be asking the orange train when it will arrive at the star station after leaving the circle station—the orange train needs to make a call if it will be stopping at the circle station, as this impacts the time dramatically. If it has to stop to drop off even one passenger, the amount of time it takes just to decelerate and accelerate again is several seconds. We use a heuristic based on how many passengers the train is currently holding and how far in the future we're trying to estimate; any more than a couple of stations in the future and we just give up and add a constant time value per station. When we try to be smart about it and guess more intelligently, we only introduce more inconsistencies into the system and passengers flip-flop paths.
This is why passengers suddenly decide to stop boarding the green train and hop on the orange train instead. When the orange train was approaching the triangle station, it was probably using a more conservative estimate for how long it would likely be at the circle station, so the green train was definitely faster (as even a 0.1s penalty would tip it). When the orange train arrives, the algorithm can be more accurate as now it knows it likely won't have to stop at the circle station, so the estimate is more aggressive (see below, †). Then the time estimates will be roughly the same, and the passenger counts on each train will swing the estimate. More passengers on a train will increase the estimated time-to-arrival as it implies more loading / unloading time. As soon as no star passengers decide to take the green train, it'll take off even if more centralised planning would have had stars load onto the green train to leave space for the crosses on the orange train. We do do some work to combat that issue through the reservation system (more on that below), but in general passengers are selfish and don't care at all if their optimal solution makes the situation worse overall. Just like a real commuter!
To avoid the situation where five passengers want to hop onto one free seat on the fastest train, and therefore skip a slightly slower, emptier, train, passengers reserve sets on inbound trains which only they can hop on. They can only reserve seats on trains that are about to arrive at the station they're waiting at (or almost at, if their train is asking its passengers if they want to disembark at the station it's pulling in to). So this would potentially avoid the issue, as both the triangles and crosses can't all reserve seats on the orange train, so you'd think some would flip to the green train. Now it comes down to the ordering of the passengers, and (again) how the time estimation works. Because the star passenger is asked before the cross, it (selfishly) decides to take the orange train, and so reserves a seat. To save time, we reuse the pathfinding result for the subsequent stars at the station, so they reserve on the orange train as well (while seats are available). The crosses do the same, however the seats run out, so the leftover run their own pathfind, but assume they can't use the orange train on the first link as all seats are reserved. They could take the green train, disembark at the star, then embark orange there, however we penalise transfers to encourage passengers to take direct routes (transfers are very difficult to estimate - I wrote about that here:
http://steamcommunity.com/games/287980/announcements/detail/1451702179003810806). The further in the future we're scheduling a transfer, the earlier we have to think we will arrive before the second train; so if it's five stations in the future our ETA must be 2.5s before the train we're transferring to, ten stations in advice 5s before, etc.). In this case the penalty will be enough that the cross doesn't think it will make it, so will have to wait until the orange train goes on another complete circuit and hits the star station again — at which point the best path is to sit tight and get picked up by the orange train at the triangle station, instead of boarding earlier and taking a joyride on the green train (passengers don't want to take up train capacity needlessly). Again, passengers are selfish and don't care about other passengers, overcrowding stations, etc. If the first passenger was a cross rather than a star, you probably wouldn't have noticed anything.
So it's all a very complicated way of saying that having two similar ways of getting to a special station can make the passengers appear confused!
I would like to find the time to work on a more accurate time estimation system, but even if it does get implemented it'll never solve every case perfectly as the estimates will always be that, an estimate. To predict everything perfectly requires we know the optimal path for each passenger ... which is what we're trying to work out! There are changes that we could make to solve the case you've presented, but then we'd introduce different observable problems elsewhere. After a number of years of revising how the pathfinding works we've come to realise there is no perfect answer, and trying to chase it is playing whack-a-mole.
† As an aside, it might have to stop at the circle station after all—a circle passenger could spawn at the triangle station while the train is boarding, or another train could drop one off, etc., (although this would require far fewer passengers to be waiting at the station, as stations are FIFO by passenger type).