Compile large amounts of CSDL: 414 Request-URI Too Large

#1

I’m building an application for a stream that will follow 5000 - 10000 Twitter accounts.

The problem I am having is compiling the CSDL as it is too big for the compiler.

I am using the DataSift C# client. The error is: “The server committed a protocol violation. Section=ResponseStatusLine”

I have tried using http://api.datasift.com/compile : The response is “414 Request-URI Too Large”.

If I reduce the size of the CSDL by removing some of the accounts, both methods work.

How do I compile large amounts of CSDL?

Many thanks.

#2

I’m also interested in the answer to this one as it is linked to my question here: http://dev.datasift.com/discussions/regularly-updating-csdl.

Over time, I could easily end up with over 100,000 tagged queries…

I guess the solution could be to have a separate stream for each one, but that doesn’t seem like a good idea…

Jon

#3

There is a limit on the maximum size of a CSDL query of 1MB - http://dev.datasift.com/docs/advanced

The best way to write very large streams at the moment is to use our Stream keyword to join multiple streams together into a single stream, for example:

tag "APPLE" {stream "f0596f03644177b8bf5b59708a08bfe8"}
tag "Snow"{stream "6740288c3c3d02984d1dc624d77e198f"}
tag "Date" {stream "922aad7bec1836c83ede051b153d3d09"}
return {
    stream "f0596f03644177b8bf5b59708a08bfe8" or
    stream "6740288c3c3d02984d1dc624d77e198f" or
    stream "922aad7bec1836c83ede051b153d3d09"
}

Alternatively, you could use Multiple Streams to access multiple streams simultaneously

#4

Thanks for your response. I’ll bear the limit in mind as the application grows. However, at this time the CSDL is only 10KB. It therefore seems that the problem is the mechanism for submitting the CSDL to the compiler as the permitted querystring length is being exceeded. The C# client seems to fail for the same reason as it also uses the querystring. Is there an option to use POST instead?

#5

POST is the prefered way to send query strings - we are currently updating our C# Client Library to reflect this - the changes should be pushed shortly. 

#6

It compiles successfully now. Thanks for your assistance.

#7

Is it any limitation on the amount of sub streams to combine streams into a master stream?

#8

If you take a look at our Advanced Features page, you will see that we implement a maximum CSDL rule size of 1MB.

This translates to *approximately* 9000 substreams per master stream. If you need to include more than this, you may want to combine this method with Multiple Streaming