Behavior of Historics Queue


#1

Hi –

I’m wondering how your queuing system works for Historics API calls. If we had, say, ten jobs in the queue configured for continuous delivery, would some of them be executed in parallel, or should we plan on the set of jobs taking about ten times as long as a single job? Is there a best-case and worst-case runtime for this kind of situation?

Thanks very much,
-Zachary Ernst

Data Scientist
Narrative Science
zernst@narrativescience.com


#2

We actually look in some detail at our Historics fair-scheduling algorithm on our blog post Historics - We Keep making it Better.

In short, the sooner you submit your jobs, the sooner they will complete. Your Historics chunks (days) may run sequentially, some may run in parallel, or they may all run at the same time if the Historics cluster is under particularly low usage.

Our Data Team are working hard to improve DataSift's Historics, and we hope to be able to bring you faster Historics in the not so distant future.