Getting subscription ids in order to do pull content


#1

What I cannot find among your docs is how two docs are connected with
each other. Let me explain.

I went through http://dev.datasift.com/docs/historics/historics-steps
I went through http://dev.datasift.com/docs/push/steps
I went through http://dev.datasift.com/docs/api/1/pull

Here is where I am stuck at.

After compiling the CSDL, after 'prepare’ing the historic query, after
hitting push/create/ (with the connector as ‘pull’) and after hitting
/historics/start, things are not explained well in the docs.

I started 4 historics jobs.

When I hit the /push/get, it returned the list of active subscriptions.

I left it to run and came back after a day.

When I again hit /push/get, it returns nothing!

Now, I hit /historics/get and get a list containing the 4 jobs that I started.

I am ready to pull in the content.

Apparently, as per Paul’s answer here http://dev.datasift.com/discussions/404-client-error-not-found-pull-request, I need to give the subscription_id to the ‘pull’ command.

But, the list that I got from hitting /historics/get does not have the subscription_id of any of them.

Now how am I supposed to pull?


#2

The Pull connector is slightly different from every other type of Push connector:

Other Push connectors will beign pushing data to your specified endpoint as soon as dta is available (i.e. just after your Historic has started running)

To receive data from the Pull endpoint, you need to issue a /pull API call in order to pull the available data into your application.

So the steps you would need to take here are:

  • /compile your CSDL
  • /historics/prepare to create your Historic
  • /push/create to set up a pull subscription for that Historic
  • Make sure you record the Push subscription ID generated by your /push/create call - you'll need this to /pull your data
  • /historics/start to begin running your Historics
  • Hit /pull for that Push subscription to start pulling data.

Remember: /historics/* API endpoints only ever return information about Historics queries - never anything about any related Push subscriptions. If you have an Historics ID, you can call /push/get passing in the Historics ID in the historics_id parameter to search your Push subscriptions for one that is consuming that Historic. 

Calling /push/get will return a 'hash' value that lets you know what it is consuming - this might be a stream hash, or an Historics ID. 


#3

Jason,

This line “If you have an Historics ID, you can call /push/get passing in the Historics ID in the historics_id parameter to search your Push subscriptions for one that is consuming that Historic”, was really valuable. I could not figure that out from the docs.

Just one more thing. Do the Push subscriptions remain active until there is no data to be returned? I could not find a straightforward answer to this in the docs. Consider I am using the pull connector and did 3 push subscriptions. Will those three be returned when I do a /push/get unti I call a Pull on any one of them? (Or until I delete one of them or one of the queries does not match any data).


#4

Hi

We have moved our discussion forums - your query has been moved to support.datasift.com/hc/communities/public/questions/202165627-Getting-subscription-ids-in-order-to-do-pull-content-