Out of order tick dropping in live data plugin

A lot of live tick data was dropped in my live data plugin. I printed the live data information in the debug envirentment. Lines which started with 'Out of order tick diff' are the output of RE itself. Others are the tick data from my plugin.

The data server provides live data for four Exchanges. So every frame of tick data has its own timestamp from its Exchange. Those data from a specified Exchange are in order. But the time between different Exchanges may be different.

It seems that there is a timestamp cache mechanism inside RightEdge. RE will remember the timestamp of the most recently tick data, and compare it with the newly comming tick data, if the newly comming timestamp is smaller, the tick data is dropped.

The possible solution is : RE not only caches the timestamp but also caches the name of this symbol. When a newly tick frame is comming, RE compares both the timestamp and the symbol name. Only the 'out of order' tick with the same symbol name should be dropped.

The following is the debug information:

cu1207 : Time = 14:30:00.500, LasPrice = 57620ru1209 : Time = 14:30:00.500, LasPrice = 26790zn1207 : Time = 14:30:00.500, LasPrice = 15510IF1205 : Time = 14:30:02.500, LasPrice = 2631.6m1209 : Time = 14:30:01.000, LasPrice = 3394Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01m1209 : Time = 14:30:01.000, LasPrice = 3394Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01cu1207 : Time = 14:30:01.000, LasPrice = 57620Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01ru1209 : Time = 14:30:01.000, LasPrice = 26795Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01IF1205 : Time = 14:30:03.000, LasPrice = 2631.6cu1207 : Time = 14:30:01.500, LasPrice = 57620Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02ru1209 : Time = 14:30:01.500, LasPrice = 26795zn1207 : Time = 14:30:01.500, LasPrice = 15515Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02IF1205 : Time = 14:30:03.500, LasPrice = 2631.4m1209 : Time = 14:30:02.000, LasPrice = 3394Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01zn1207 : Time = 14:30:02.000, LasPrice = 15515Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01IF1205 : Time = 14:30:04.000, LasPrice = 2631.8m1209 : Time = 14:30:03.000, LasPrice = 3394Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01zn1207 : Time = 14:30:02.500, LasPrice = 15515Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02IF1205 : Time = 14:30:04.500, LasPrice = 2631.8Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02Out of order tick diff: 00:00:02m1209 : Time = 14:30:03.000, LasPrice = 3394Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01ru1209 : Time = 14:30:03.000, LasPrice = 26795Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01zn1207 : Time = 14:30:03.000, LasPrice = 15515Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01Out of order tick diff: 00:00:01

When running a system, the ticks need to all be in order, even for different symbols. A tick for one symbol updates the time for all the symbols, and hence may send out a new bar for other symbols. If a tick comes in with an earlier time it can't be added back to the bar that was already sent out.

If the difference in time between the ticks is less than 0.5 seconds, RightEdge will modify the time of the "out of order tick" so it is the same timestamp as the latest tick received. That way it can still be processed by the system without causing errors due to being out of order. To avoid dropping ticks, you can do the same thing in your plugin but with a larger time difference tolerance.

Where is the time difference set in TWSPlugin and IQFeed? I'm using both of these feeds along with the Random feed and it is quite common that my live data shows all zeros. Currently, I have to restart the live feed several times in order for data to appear.

I see the local variable ‘accountTimeDiff’ set in the TWS client_CurrentTime method but where can I set a larger time difference tolerance per your post on 4/26 in order to prevent the error, for example, with a time difference of 1 sec? Where is the 0.5 sec set?

I have tracked down the issue to the random tick generator when used with either IQFeed or TWS plugins. There are no issues when using just IQFeed and TWS together. My assumption is that my clock, even though it is synced to a stratum 1, is slightly off compared to the times that are being returned on the other two.

Also, I’m not sure the 0.5 seconds gives enough differential when using multiple live data providers.

What are your thoughts to fix? I need to fix asap because I’m running live systems of it. Note, I technically do not need the random tick data (just the prior day closing price).

With the way RightEdge handles processing incoming events, it can't handle data that comes in out of order. So the workaround when the data is less than 0.5 seconds out of order is to change the timestamp of the out-of-order data so it matches the most recent timestamp received previously.

Would it help if you were able to increase the margin of error to something bigger than 0.5 seconds? Is it a problem that the timestamps on the incoming data would be changed?

I’m not sure changing the timestamps will work. The question then becomes, what do I change it to? If I add/subtract x sec to the timestamp set in the live random plugin, it is very likely that the problem will still be present except that the ticks from the random plugin will be ahead/behind of the other two plugins.

Yes, I believe changing the margin to greater than 0.5 sec should work. Where is this set in the live random plugin?

There are two types of adjustments going on. The first is in the live data plugin. Currently only the TWS plugin does this. TWS doesn't send a timestamp with each tick, so the plugin uses the current system time adjusted by the offset stored in the accountTimeDiff field.

The other adjustment is in RightEdge's live data processing. If it detects out of order ticks, it will drop them if they are more than 0.5 seconds out of order, otherwise it will adjust the timestamp of the tick.

If you are just using one data plugin, then the plugin should just send ticks in the correct order and there won't be any out of order issues. However when you have multiple plugins they tick timestamps may be set differently. With the IQFeed plugin the timestamp is sent by the server for each tick, with TWS the local time plus the offset is used, and with the random data plugin, just the local time is used. So it's quite possible to end up with out of order data.

I'm not sure there's a great solution to this. I could make the 0.5 second margin of correction larger or a user-controlled setting, but then the tick timestamps will be adjusted more which doesn't seem great. If your trading system doesn't depend on reacting quickly to market data, it would be possible to buffer the ticks for longer (for example for 15 seconds) and sort any ticks that come in within the buffer by the timestamp. That would be kind of complicated to write however.