Historics queries very slow (or stuck?)


#1

Hi, I started a historics task yesterday at around midday. After the task spent 4 hours in queue I deleted it and re-added the same task. It got picked up fairly quickly and progressed to 9 per cent, but it has been like that for over 16 hours now with no visible progress made. Is that normal? The estimated delivery time was 8:19pm last night.


#2

The issue is now resolved - progress resumed again.
I never had such a slow historics job before and I run them with very high frequency. Do you perhaps have any stats on the average expected time of completion for historics jobs? I would also be interested in knowing what the longest time we can expect for these tasks to complete as this may affect our projects.


#3

What you experienced is due to a couple of different factors that have to do with the way our historics platform is designed. One of the reasons for slowness on the platform is due to a heavy load at the time of the request. Although we do our best to prioritize requests to maximize efficiency, sometimes requests may take longer than normal due to a large number requests at a given time. Another reason for slowness could be due to the date range chosen for your request. Larger ranges of dates will take longer to run than smaller ranges and may be prioritized lower than smaller or more recent ranges. Due to these reasons it is hard to give a concrete statistics for average completion time or the longest expected completion time.

Regards,
Michael
Datasift Support


#4

Thanks Michael. The task did finish in the end.