I have a process that polls a pull queue every ten seconds.
In that process, it gathers a few subscription IDs, and calls /pull on it.
It then takes the interaction and performs some processing on it.
If it sees there is a X-DataSift-Cursor-Next header with a value, it waits .5 seconds (as per the documentation) and calls again.
It continues to do this until the queue is drained (it’s pulling 20MB at a time, but it never gets that high).
That said, in the past five hours, I’ve had 34 calls fail where it returns a 500 error.
- Is the general logic of cycling through the cursor until there is no next cursor correct? The assumption is that when the queue is drained, the next cursor header will return null.
- Do I lose data when a 500 is returned?
- The documentation for response codes (http://dev.datasift.com/docs/rest-api/response-codes) indicates that we should try again. Should we wait the 1/2 second before the next call before trying again?
- It also says to contact support. I have the times where these failures happened, and can pass those along if need be.