DPU charges in the Ondemand stream scenario after exhaustion of 500k tweets/day


#1

If the 500k tweets/day has been already streamed and a part of the day is still remaining, are DPU charges applicable for the remaining part of the day ?


#2

If you hit the Twitter rate limit, your account will be temporarily restricted, and you will not be able to successfully connect any streams while the rate limit is exceeded. This means that any currently connected streams will also be disconnected, so no DPU charges will be incurred until you are able to reconnect. 


#3

Jason,

We encountered this today, but after receiving 500k tweets in 25h30mins, i.e. 25h30mins since we got the stream connection. Couple of qns :

  1. Since the 500k tweets was hit after the 24hr limit, my understanding was that the streaming would continue. If we reach 500k within the next 24hrs, say at 21hrs or something, then streaming would stop for 3hrs and resume after 24hrs is over.
  2. When you mention, account will be temporarily restricted, how long is this temporary restriction and when can we begin streaming again ? Its’ been about 3hrs now and we still get the same msg of
    "The rate limit for twitter has been exceeded"

#4

The Twitter rate limit is worked out over a rolling 24hr period - from '24 hours ago' until 'now'.

The account restriction will end as soon as you are no longer in violation of the limit, so when the nubmer of Tweets consumed in the last rolling 24 hour period is less than 500K. Hitting the /usage API endpoint and setting the 'period' parameter to 'day' will give you your usage over the last rolling 24 hour period.

If you are on a monthly subscription package, we do allow you to apply for higher Twitter rate limits by contacting Technical Support.


#5

Jason,

That is weird. The understanding from the documentation with the 500k/24hr limit seemed like if 500k was used up in 20hrs, then after 4 hrs, this would get reset to 0 and we’d have the opportunity capture 500k over the next 20-24 hrs. The rolling 24 hr window from ‘24 hrs ago’ until ‘now’ wasn’t clear in the documentation.

In any case, when you say ‘now’, do you mean the the second, minute or hour ?

Here’s something I see on the dashboard after I restarted the stream about 1 hr ago.
“Your account is currently restricted. You will not be able to authenticate with the API for 110 minutes. The rate limit for twitter has been exceeded”.

This seems to restrict it to the closest 2 hrs.

Say from
Start Hr - 00:00, End Hr - 20:00,
Num tweets received : 500k,
Num tweets received between 00:00 - 00:01 (i.e. 1 minute) : 50

At Hr - 24:01, we would be able to connect again and be allowed to capture upto 50 tweets, but not more than that. Is this the way this progresses ?


#6

Jason,

Can you provide answers for the questions so that we can understand the twitter rate limit better ?

Mahesh


#7

Hi Mahesh,

Billing is checked every 5 minutes so as the window rolls, it will be from "24 hours ago" to "now (the most recent 5th minute). 

So, to answer your question, if you consumed 50 tweets between 00:00 - 00:05, you will be able to consume 50 more tweets 24 hours later. 

Regards,
Victor