Yahoo! Pipes


I am completely mystified as to how to get data out of DataSift.

I don’t have any problem with the CSDL inside Datasift.

But, how to get the data out of DataSift?

I would like to get a feed output for use in Yahoo! Pipes.

How does one access DataSift output in feed form?

Are there any converters for the CSV or JSON output into feed form?

Yahoo! Pipes can handle JSON via the webservice module; but, I need some kind if example to help me with this.

DataSift output ought to be usable in Yahoo! Pipes.


I would recommend reading our Streaming API documentation. You should be able to find examples in our documentation and our API Client Libraries.

We serve our streams in JSON, so it sounds like Yahoo! Pipes should be able to handle this without a problem. 


I was able to get the DataSift REST API working in Yahoo! Pipes using the following modules:

URL Builder => Fetch Data => Create RSS => Pipe Output.

Additionally, I had to add a Regex module to reconstruct the tweet URLS from their constituent parts.

There seem to be some push / pull issues here that I don’t fully understand. What would be the limitations of using the REST API, and how could they be improved upon ??

I guess I fail to understand why I can’t just get a simple feed output of my streams directly from DataSift without having to go through all this bother… It seems to me that DataSift could get A LOT MORE users if it was more straightforward to use.


The full Tweet URL can be found in ['interaction']['link'] in the returned JSON object - you should not need to reconstruct this. 

Exactly what problems are you having setting up this connection? Connecting to DataSift's Streaming API is as simple as opening a conection to, and passing in a stream hash, your username, and your API key.

How to connect to our streaming API is described in detail for both HTTP Streaming and WebSocket Streaming