Ability to push Source logs to a destination


#1

While you can gain access to the logs of a managed source by calling the API, It would be great if you had the option of being able to have them work under the same rules for data collection.

That is say have the ability to have the log data push to a destination source (such as S3) based and have it respect the destination sources attributes.

For example, have a Instagram source push its log entries out every 30 mins to a S3 bucket/folder destination.

This could be done at an individual source level or at an account level where its a 0/1 option and all logs go to the same location. (Id be happy with either option!)

If it was an account wide option, have it so that the usual useful attributes are in each log item such stream id, name and so on for easy identification.


#2

Certainly an interesting request. One limitation that is immediately obvious is that if a Push subscription generates logs saying something like “Your endpoint is returning 500 errors/unavailable” (DataSift is unable to deliver to your endpoint), then you may never see these notifications if you are dependent on receiving them via a Push subscription.
As for delivering logs from Managed Sources however, I can certainly see the use in a feature like this. I’ll raise it with our Product team for consideration.


#3

Agreed on the cyclic loop problem in regards pushing a log to an source of which the source is the cause of the log… That is where the API would be used. In our case, we would pick up the endpoint failure by way of data reduction over the hour. An alarm would be raised and we would investigate.

I would suggest that the types of log events one would be after could be loosely classified as Infrastructure =Vs “Application” . By that I mean Infrastructure errors are where things have just gone away or there is a fatal error (destination no longer available, platform error, and so on…) where an application error would be around errors reported from the source itself. For instance, and specifically for the case I am thinking about is source token management such as when errors are reported around the tokens used.

I would like to be able to parse over all logs for all FB sources ever X hours and look for token management errors. This could be due to expiration or something else and if need be, send an alert to take action either manually or automajically.

Jason