Multiple streams with messages


#1

Hi,

Can you provide more information on how to consume multiple streams with the HTTP endpoint and messages? As I understand it, this is the expected behavior:

1. Open a socket to to http://stream.datasift.com (does GET or POST matter? I tried both...) 2. Once connected, send a JSON string containing the hash to subscribe to over the socket, i.e.: { "action" : "subscribe" , "hash": "4f3af38ad1c31646561bc814c3e69af5"} For some reason, I wasn't able to get this to work. In most cases, after attempting to send something like the string in step 2, I would receive a TCP FIN and the connection would close. The closest I got to working was passing an unsubscribe message and having datasift tell me I provided the wrong hash to unsubscribe from and disconnecting me immediately. I tried in both ruby as well as using raw netcat commands from the terminal and nothing I tried resulted in receiving data or even acknowledging that I had sent a subscribe command.

Am I missing some sort of formatting or additional command? I’m evaluating DataSift for consuming multiple different streams and option 1 (disconnecting and reconnecting with the list of hashes) is not viable as that would result in a loss of data.

Thanks


#2

Apologies for the late reply.

 

Multi-streaming only works with the subscribe/unsubscribe messages through WebSockets, not the HTTP protocol. Currently the two API Client Libraries we have which support multi-streaming are out Java Client, and our NodeJS Client (take a look at our Using Messages page for more details). 

We have implemented a version of HTTP multi-streaming through the NodeJS Client, although this technically breaks HTTP streaming standards. 

Which client library were you using when you encountered these issues?


#3

I was attempting to modify the ruby client, as well as using just a raw HTTP connection from a C client. Given that the documentation states messages work through HTTP, you may want to update it to reflect that only WebSockets are supported. :frowning: I was definitely a little confused that the multi-streaming docs stated it worked over HTTP, given the standards breakage, but was willing to give it a shot.

This is a shame as I don’t really see another way to consume data without losing some due to disconnects/reconnects to update the hashes. I suppose I could have one giant hash…


#4

If multi-streaming only works with websockets, then what do the classes HttpMulti and HttpMultiThread do?
They seem to be dedicated to multistreaming over the http protocol.


#5

The HTTP multi-streaming classes allow you to subscribe to several streams at once, although you can not use subscribe/unsubscribe messages to add / remove streams to your connection without reconnecting.