How to detect stalled HTTP or WebSocket connection?


#1

I have been testing the client libraries on multiple languages and I have noticed a common issue: if the network connection is dropped unexpectedly, it does not appear to detect the loss of connectivity and attempt to reconnect. I tested this by running the examples on my local machine, then disabling the network card for 5 minutes, then turning the network card back on. The connection is gone but the client libraries do not realize it.

How do I ensure that my streaming data connection will be kept alive? Is this too uncommon of a scenario to worry about? We are running all of our production services on Amazon EC2 so we don’t expect this to occur very often. I experienced similar issues when consuming the Twitter streaming API - is this more of an overall issue with these types of connections?


#2

How are you using the client libraries? Are you running the examples, or have you written wrappers and custom code for each of them? Also, specifically which client libraries are you using, and what kind of network configuration are you running them through (are there any corporate firewalls/proxies in your network)?

It is rare to lose network connections in this way, but it is still important to ensure you have all eventualities covered. Using the PHP client library as an example, reconnecting on failures is handled in the onStart() method of the DataSift_StreamConsumer_HTTP class. If it is detected that the connection is not open, we retry the connection. You may want to look at adding a simple try/catch block around your connection code to try to reconnect if any errors are detected. 

When using HTTP Streaming, we send Keep-Alive 'ticks' like the following:

  {"tick":1311694603,"status":"connected","message":"Waiting for data"}

These will be sent if your HTTP connection has been idle for 30 seconds. So if you have not received an interaction, or a tick for 30 seconds, DataSift will send a tick to keep your connection alive. If you do not receive anything over your connection for over 30 seconds, this is a sign that your HTTP connection may have been dropped. 

We also send a number of different 'warning' and 'failure' status messages to make you aware of any potential issues with your connection like the following:

  {"status": "failure", "message": "The username or API key is not correct"}

For WebSocket connections, we will soon be implementing support for WebSocket Ping-Pong messages to allow you to ensure your connection is kept alive.


#3

Our preferred client library is the Node.js HTTP streaming library so the “tick” message solves our problem perfectly! I missed this in the documentation earlier. Thanks!