onStopped not called sometimes when account runs out of money


#1

I was running 5 streams in parallel against trial account using 2.2.0 Datasift java library (WebSockets consumer). After account ran out of money 3 WsThreads stopped (onStop callback called with reason ‘You have insufficient credits …’ which is expected behaviour). However onStop callbacks for 2 WsThreads were not called and I think corresponding threads were still alive.

Could you please take a look at 2.2.0 Datasift java library and investigate if such cases could happen sometimes?

I know the issue is not easy to reproduce. One thing that I noticed about 2 not stopped streams is that they had very low-freqency CSDL filters (could match several tweets per day) while streams that stopped had high-frequency CSDL filters.

Thanks,
Artsiom


#2

I have raised this as an issue against the Github repository. We will look into this as soon as possible.

Problem calling onStop() for some streams when account runs out of credit


#3

I think I found the reason why onStop is not called sometimes. This happens when ‘int b = input.read();’ inside WebSocketsReceiver.run() throws SocketException:

java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:185)
at java.net.SocketInputStream.read(SocketInputStream.java:199)
at de.roderick.weberknecht.WebSocketReceiver.run(WebSocketReceiver.java:72)

To handle this exception WebSocketsConnection.close() method is called at some point. This method should call eventHandler.onClose() at the end. However WebSocketsConnection.close() throws exception before doing eventHandler.onClose():

de.roderick.weberknecht.WebSocketException: error while sending close handshake
at de.roderick.weberknecht.WebSocketConnection.sendCloseHandshake(WebSocketConnection.java)
at de.roderick.weberknecht.WebSocketConnection.close(WebSocketConnection.java)
at de.roderick.weberknecht.WebSocketConnection.handleReceiverError(WebSocketConnection.java)
at de.roderick.weberknecht.WebSocketReceiver.handleError(WebSocketReceiver.java:124)
at de.roderick.weberknecht.WebSocketReceiver.run(WebSocketReceiver.java:102)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at de.roderick.weberknecht.WebSocketConnection.sendCloseHandshake(WebSocketConnection.java)

WebSocketsConnection.close() fails because it tries to use the same socket (to sendCloseHandshake) that thrown ‘Connection reset’ before.
Calling eventHandler.onClose() is very important as it controls reconnection or stopping.

I would fix it myself in my local version but I don’t know much about protocol between datasift server and client in particular about consequences of not sending close handhake in case socket fails with ‘connection reset’ and just reconnecting e.g. like this:

public synchronized void close()
throws WebSocketException {
if (!connected) {
return;
}

    try {
    sendCloseHandshake();
    catch(SocketException e) {
          //just log and ignore
    }


    if (receiver.isRunning()) {
        receiver.stopit();
    }

    closeStreams();

    eventHandler.onClose();
}

Could you please help with this?


#4

Hi,

Could you please take a look at my question above?
Thanks in advance.

Regards,
Artsiom


#5

Hi,

I am also using trial version of datasift. We are able to call unsubscribe() in WS which inturns sends message like _ws.send("{“action”:“stop”}");. But the stream doesn’t look like stopping. I started two streams say (1.Obama and 2.Romney). I am capturing the hash of both keywords and pass it to unsubsribe method(). But it does not work properly. Could you share your experience in proper way of unsubscribing the hash and stopping the stream. Note: We are using Java client.