When I hit the /historics/start endpoint, where does DataSift post the responses to?


#1

I successfully created and ran realtime queries using CSDL and the datasift-python library. The example given in the repository here: https://github.com/datasift/datasift-python/blob/develop/examples/live_stream_api.py worked neatly and all I had to do was to insert code within the subscribe_to_hash() function.

However, when I modified the code to follow the step by step Historics guide, I could not get the interactions within the subscribe_to_hash function. Following is how I modified:

@client.on_open
def on_open():
print "Stream client set up, subscribing to stream"
csdl = 'twitter.retweet.text contains “hashtag” OR twitter.text contains “hashtag”'
stream = client.compile(csdl)[‘hash’]
start = 1371580200.0 + (3600 * 5)
end = start + (3600 * 3)

historics = client.historics.prepare(stream, start, end, 'My hashtag historics', ['twitter'])

print "Historics is: ", historics #This got successfully printed.

historics_id = historics['id']

subscription = client.push.create_from_historics(historics_id, 'Python push', 'pull', {})

print(subscription) #This got successfully printed.

print(client.historics.start(historics_id)) #This got successfully printed.

@client.subscribe(stream)
def on_interaction(interaction):
    print(interaction) #Nothing here.

Is there something obviously wrong with this approach or is there something that I need to understand better on how historics work?


#2

The @client.subscribe(stream) method attempts to subscribe to the stream in realtime. Any data matched by your Historics query will have been made available to the Push endpoint you specified in your client.push.create_from_historics call - in this case, it has been made available to the Pull connector.

Take a look at the push_api example for an example of how to consume data from a Pull subscription.


#3

I took a look at it and tried the following:


client = Client(“name”, “key”)
csdl = 'twitter.retweet.text contains “#tag” OR twitter.text contains “#tag”'
stream = client.compile(csdl)[‘hash’]

start = 1371580200.0 + (3600 * 9)
end = start + (3600 * 3)

historics = client.historics.prepare(stream, start, end, ‘My historics’, [‘twitter’])
subscription = client.push.create_from_historics(historics[‘id’], ‘Here we are again’, ‘pull’, {})

client.historics.start(historics[‘id’])

time.sleep(30)

for interaction in client.pull(subscription[‘id’]):
print(interaction)

This did not return anything either.

My understanding was that once you create a subscription, it will keep the data collected so that you can “pull” it at any time you wish. However, when I hit the “get” endpoint, it returns null, which means I don’t have any subscriptions. How can I collect the aggregated data if I can’t get the subscription id?


#4

How silly of me. Apparently, I do not have Twitter as a Data Source activated. Let me try doing that and running these queries once more. Already wasted a few DPUs.


#5

Sweet. I seem to be able to get the data. This query has been answered. A few other have cropped up which I will post as separate questions.

PS: It’d be great if I could see all the questions that I have asked under one dashboard. It is almost impossible otherwise to keep track of one’s discussions over here.

Cheers!


#6

Good to hear you’re up and running! We will be moving to new discussion forum software this month, and you will be able to see all your posts under one dashboard there.