It would be useful to have some developer streams on standard terms (Like the example in the client library - “football”) so we can develop and test without fearing we’re going to spend lots of money without realizing it.
We do not have any "developer streams", but feel free to take a look at our interaction.sample target as an alternative. You can add this to your filter to reduce the amount of throughout a stream gets to a certain percentage. Here is an example stream using interaction.sample:
does this avoid incurring any cost?
You will still have to pay the license fee and processing (DPU) cost, but the license fee will be greatly reduced, as you would only be receiving a fraction of the interactions which matched your filter.
You could always run tests on some of our free* Data Sources such as Facebook or Digg, which means you would only have to pay the DPU processing cost, and avoid paying for licenses.
For more information about our costs, see our Understanding Billing page.
* (Free at time of writing)
Thanks Jason. I think you could be more small developer friendly. I think you should offer a limited use account which is free before you start having to pay real money.
Right now I’m seeing that I test one stream call using Datasift PHP library and I get charged 20 cents for 5 football tweets. Considering that you have my personal credit card details I am now super worried right now about Datasift and don’t much feel like learning and trialling your API. Sorry to be honest but I hope it helps you see things from my perspective.
I’ve also seen the Billing page which says “Small startup $3000 a month” which also made me fall off my Google Campus chair… Two zeroes too many.
I am just getting the feeling I’m at the wrong party… #runningscared
I think you got the "wrong end of the stick" regarding our pricing models;
- When you create a new account, you are given $10 for free to begin experimenting with the platform.
- We charge $0.20 per DPU hour. This means you can run a stream of up to 1 DPU in complexity for an hour for $0.20.
- Tweets are subject to a license fee (as are some of our other Data Sources). This works out at $0.10 for every 1000 Tweets you receive.
- As well as our subscription models, we also offer a Pay-As-You-Go service, allowing you to top up as little as $10 at a time. This is the ideal pricing model for smaller developers.
You may want to take a quick look at our Billing FAQ, as this covers all of these questions and more.
Thanks Jason, yes I hope so. :o)
Let me show you what’s in my head.
Let’s say each user I have on leaderboarded.com creates a leaderboard with 40 players and tracks 1 hashtag. We track tweets in to each player and tweets out on the tracked hashtag. Fairly standard. They run the leaderboard on a weekly basis. Typically we see maybe 100 tweets on the hastag by the players themselves and around 10 incoming tweets per person. That means:
Total tweets per week = 500 = $0.10 per week (OK no problem)
41 streams (1 DPU each) over a week of 168 hours = 6888 DPU = $1377.60 per week (Yikes!)
That’s just for one small leaderboard with 40 players.
If we scale up to a leaderboard for 2000 players, say at a conference, we’re looking at a crazy cost despite the fact the per player volumes are small.
Have I still got the wrong end of the stick? Is there a way of searching multiple @mentions for many players without incurring high DPU costs?
Sorry I did format this nicely but the statement “Lines and paragraphs break automatically” in the comments help is not actually true…
This doc entry on Optimization - http://dev.datasift.com/docs/advanced/optimization - looks very positive. Need to see if there’s any limit to the number of screen names in a single CSDL search
A stream like the one you have described (following 40 different users, and looking for them to Tweet a certain phrase/hashtag), will cost 0.6 DPUs, so will be charged at $4.80 / day, or $33.60 / week plus license fees. (The minimum cost of running a stream is $0.20 / hr, so any streams costing less than 1 DPU will be rounded to 1 DPU). The example CSDL would be:
twitter.user.screen_name in "user1, user2, user3, ..., user40" AND twitter.text contains "myHashtag"
There are ways of reducing costs a little further, by running a number of streams together as a Multiple Stream. This takes into account the cost of all the streams you are looking to consume, and you are billed on the total DPU cost across all your streams. For example, lets say you are looking to run three streams, costing 0.1 DPUs, 0.6 DPUs and 0.8 DPUs. Consuming each of these streams separately will cost 1 DPU per stream, so 3 DPUs in total. Running them together in a multi-stream, the DPU costs will be added together, and you will be billed on the total. So:
0.1 + 0.6 + 0.8 = 1.5 DPUs
You can run up to 5000 streams simultaneously over one connection, so this could dramatically reduce your DPU costs.
Also, our DPU cost is determined using a sliding scale, so you could monitor up to 10,000 users for only 4 DPUs (or $0.80 per hour). Details about this can be found towards the bottom of our Understanding Billing page.
The only limit on CSDL is that a single CSDL rule must not exceed 1MB in size (documented here http://dev.datasift.com/docs/advanced). This means you can fit approximately 90,000 users into a single rule. You can however link a number of CSDL rules together using our stream keyword (http://dev.datasift.com/docs/advanced/stream-keyword) to create much larger rules.
Thanks Jason, this is really very helpful. :o)
What are the additional license fees you’re talking about? Is that the fee per 1,000 tweets?