Mind the Gap!
Market Data. Two words on which a significant portion of the world economy could be considered to turn. Getting data on what is happening in the trading markets from the providers to the consumers is a critical operation that can result in the loss of thousands, if not millions, of dollars if it is not done properly. There are many consumers of market data and there are many companies that provide aggregation and forwarding of that data to consumers. These providers install large pipes to absorb data from the wide range of sources and then resell that information to various consumers who use the data for their day-to-day business. There are many benefits to using this approach including:
- Get only the exact data you need for the people who need it
- Save money on having to aggregate the data yourself for the multitude of sources available
When providers send market data to their customers there are a multiple ways that data can be transmitted. Transmission methods can include:
- UDP multicast transmission
- TCP unicast transmission
Both of these approaches has benefits and costs. The most efficient approach from the providers standpoint is the use of UDP multicast packet transmission. A single multicast stream is sent and those who subscribe to the data feed are able to receive the data and process it for their use. This approach is relatively easy to implement and relatively inexpensive both from a network and operational management approach. A single multi-cast stream uses little bandwidth and does not require extensive user control.
Using a TCP-based unicast approach can be the next-best solution. While neither as efficient nor as light weight as UDP multicast, TCP is a more reliable protocol. While UDP makes no guarantee that any (or all) packets will make it from source to destination, with TCP the data is guaranteed to get to the client. Unfortunately there are additional costs in the form of bandwidth, configuration and management if a provider has lots of customers all using TCP unicast streams.
Based on this, why would a provider ever offer a TCP unicast transmission stream as opposed to UDP multicast? The answer comes in the form of the provider’s ability to guarantee delivery of the data to their customers. If the UDP multicast stream is working smoothly and no packets are being dropped then everything is great. The customer gets what they have paid for, and everyone is happy. Unfortunately as soon as packets start getting lost between the provider and consumer, the customer is going to be missing data and the value of the delivery drops off drastically. At some point, enough gaps in the data will result in unhappy customers which will almost always lead to unhappy providers.
You can analyze the market feed data and identify issues by looking for gaps in the information contained within the packets. Both the data provider and the customer have an interest in looking for any gaps that may arise. When a provider has a customer that is not able to get their stream over UDP that customer may request (or require) that the exchange provide a TCP unicast stream to ensure reliable delivery of the market data. As more and more customers have issues, the cost of providing more and more TCP unicast streams increases both in terms of bandwidth and the labor involved in managing multiple streams. This, in turn, can lead to greatly diminished profit margins.
From the customers perspective, if they are experiencing gaps, not only are they not getting a service they paid for but they may be missing crucial information that impacts their ability to do business and their bottom line. From the providers perspective if their data feeds contain gaps, they risk alienating existing customers, damage to their reputation, and issues acquiring new customers.
Riverbed SteelCentral NetShark and Packet Analyzer provides the ability to do real-time market data gap analysis by analyzing the market data protocols (such as ITCH) and looking for breaks in the message sequences. Because a single packet can contain multiple messages, it is important to be able to actually look at the internal header information so you know exactly how many messages were lost (in addition to the number of actual packets lost). Even though many providers are compressing the actual market data (quite successfully with rates of up to 90% reduction), the headers themselves are not compressed and thus the NetShark is still able to successfully analyze the packets.
While being able to generate a graph or table showing gaps in market data is useful, what is even more useful is the ability to generate an alert when such an event occurs. Fortunately this can be accomplished by taking advantage of NetShark’s and Packet Analyzer’s ability to create watches. Watches look for specific occurrences (such as a gap in market data) and can generate an alert for an administrator or manager that a problem has occurred.
Whether you are a provider or consumer of market data feeds the onus is on you to ensure you are providing or getting the data you are supposed to. While it is easy to say missing one or two packets does not matter, how do you know you did not miss the packet that would have made a multi-million dollar difference to your business? Knowing you are transmitting or receiving all the data all the time is critical to business operations. SteelCentral NetShark and Packet Analyzer can help you ensure you ‘mind the gaps’ and know exactly what you are getting/providing.