How can I combine multiple streams with the Historics API to save DPUs?


#1

I have many very lightweight queries to retrieve data from the Historics API, all for the same period. I need the data separately. Is there a way to combine multiple 0.1 DPU streams/queries into a single call to the Historics API to avoid getting billed 1 DPU for each of them individually?


#2

Try combining the stream keyword with tags:

If you have the following two streams:

twitter.text contains "tea" - with a stream hash of '123456'

twitter.text contains "coffee" - with a stream hash of '987654'

You could combine them into a 'master stream':

tag "Tea_stream" stream "123456" }
tag "Coffee_stream" stream "987654" }
return {
  stream "123456" or
  stream "987654"
}


#3

Jason, I totally understand the code. Can you please help me understand how it works “under the covers”? Does it actually save DPU? …and $$?

Thanks


#4

John, Take a look at our Understanding Billing page. The simplest stream you can write (in terms of DPU cost) is 0.1 DPUs. The minimum DPU charge for consuming a stream is charged at 1 DPU (currently $0.20 per hour). 

So from Corey's example above, if he were to run several 0.1 DPU streams simultaneously, they would be charged at 1 DPU each. If you use multistreaming to run these streams together, each stream will be rated at it's DPU cost (0.1 DPU in this case), so running 12 streams costing 0.1 DPU each over multistreaming would cost just 1.2 DPUs, rather than the 12 DPUs it would cost to run them all separately.