repeatably error and server crash

[manuprendlair]

Hello,

I experiment some pb with the server. I am not able to use it more than 1 hours without crash.
My config:

• Raspberry pi 2 (no overclocking)
• Debian mini + lxde ( kernel 3.18.8-v7+)
• instrumentpanel
• NMEA strings send by UDP to port 5556 by NemaStudio.

Firstly when I start to sen nmea string to UDP, I have this error who come repeatedly:

[) thread #4 - seda://commonOut] OutputFilterProcessor DEBUG Outputting:{breadcrumbId=ID-blueocean-nav-51270-1430049210217-0-15170, OUTPUT_TYPE=OUTPUT_NMEA, websocket.connectionKey=78c75864-af2b-43e1-ae1b-1b11a873c768}
[) thread #4 - seda://commonOut] OutputFilterProcessor DEBUG Outputting:Message: {"context":"vessels.self","updates":[]}
[ nioEventLoopGroup-4-2] SubscriptionManager DEBUG Adding 9f4a9c12-b632-460c-8ffb-1d4dd502c782/9f4a9c12-b632-460c-8ffb-1d4dd502c782, outputType=OUTPUT_NMEA
[ thread #5 - timer://heartbeat] HeartbeatProcessor ERROR Exception occurred during execution on the exchange: Exchange[Message: {"context":"vessels.self","updates":[]}]
org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[Message: {"context":"vessels.self","updates":[]}]
at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1572)
at org.apache.camel.util.ExchangeHelper.extractResultBody(ExchangeHelper.java:623)
at org.apache.camel.impl.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:471)
at org.apache.camel.impl.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:467)
at org.apache.camel.impl.DefaultProducerTemplate.sendBodyAndHeaders(DefaultProducerTemplate.java:245)
at org.apache.camel.impl.DefaultProducerTemplate.sendBodyAndHeaders(DefaultProducerTemplate.java:379)
at nz.co.fortytwo.signalk.processor.HeartbeatProcessor.process(HeartbeatProcessor.java:81)
at org.apache.camel.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:63)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:448)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:165)
at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:73)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: java.lang.IllegalStateException: Queue full
at java.util.AbstractQueue.add(AbstractQueue.java:98)
at org.apache.camel.component.seda.SedaProducer.addToQueue(SedaProducer.java:230)
at org.apache.camel.component.seda.SedaProducer.process(SedaProducer.java:167)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
at org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:68)
at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:375)
at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:343)
at org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:233)
at org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:343)
at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:184)
at org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:124)
at org.apache.camel.impl.DefaultProducerTemplate.sendBodyAndHeaders(DefaultProducerTemplate.java:233)
... 11 more
[ thread #5 - timer://heartbeat] all ERROR Exchange[ExchangePattern: InOnly, BodyType: null, Body: [Body is null]]

Then after sometime (30 min to 1h...), the server stop responding with this error:

org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[Message: {vessels.kiaOra.navigation.position.latitude=44.053721, vessels.kiaOra.navigation.position.longitude=4.103768, vessels.kiaOra.navigation.position.source=vessels.kiaOra.sources.nmea.0183.RMC, vessels.kiaOra.navigation.position.timestamp=2015-04-27T08:21:20.294Z, vessels.kiaOra.navigation.speedOverGround.source=vessels.kiaOra.sources.nmea.0183.RMC, vessels.kiaOra.navigation.speedOverGround.timestamp=2015-04-27T08:21:20.294Z, vessels.kiaOra.navigation.speedOverGround.value=4.629959999999999, vessels.kiaOra.sources.nmea.0183.RMC.source=unknown, vessels.kiaOra.sources.nmea.0183.RMC.timestamp=2015-04-27T08:21:20.294Z, vessels.kiaOra.sources.nmea.0183.RMC.value=$GPRMC,082113.77,A,4403.21076,N,00406.22608,E,0009.0,180.0,270415,0.0,W,A_16}]
at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1572)
at org.apache.camel.impl.DefaultExchange.setException(DefaultExchange.java:308)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:144)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.FatalFallbackErrorHandler.process(FatalFallbackErrorHandler.java:49)
at org.apache.camel.processor.RedeliveryErrorHandler.deliverToFailureProcessor(RedeliveryErrorHandler.java:896)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:387)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:291)
at org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsumer.java:200)
at org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:147)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "Thread-14" java.lang.OutOfMemoryError: Java heap space
[) thread #0 - seda://inputData] SedaConsumer WARN Error processing exchange. Exchange[Message: $WIMWV,073.7,R,013.0,N,A_22
]. Caused by: [java.lang.OutOfMemoryError - Java heap space]
java.lang.OutOfMemoryError: Java heap space
Exception in thread "qtp26986565-90" java.lang.OutOfMemoryError: Java heap space
[ nioEventLoopGroup-4-2] NioEventLoop WARN Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space

With the help of Roberts I change hawtio.start to false. But I have same error I described.

Emmanuel

[rob42]

This is probably the message buffer stacking up messages until the memory overflows. Possibly the heartbeat message is continuing to be sent after the client goes away.
I'm not seeing this in normal websocket connections, my system runs for 24+ hours without issue.

Will need to test further on TCP/UDP etc but very busy at present

[rob42]

Hmm, in above error:
Caused by: java.lang.IllegalStateException: Queue full
Just realised that the NemaStudio is probably sending, but I dont know if it reads from the UDP port. If not then the heartbeat messages will queue up and overflow.
Guess i will need to deal with the case where a client sends but doesnt read...
Actually that doesnt matter in UDP as its just send/no-ack. Lost messages are normal for UDP.

[rob42]

Closing as old and not repeatable for me. Pls re-open if you still suffer this problem