I am building a dashboard template that we will use as the basis for all of our other logging pages to simplify and standardise the displays.

I am almost finished with Jared's excellent help earlier on another thread.

I am seeing another problem now.

I have a virtual data source set up to test the template and I can see the data points update every 60 seconds as it should. But the dashboard, when viewed will only update once or twice then it stops updating until the web browser is manually refreshed. Then the dashboard will update once or twice again and then stop.

Any suggestions would be greatly appreciated. Let me know if you want to see the mark up.

I am building a dashboard template that we will use as the basis for all of our other logging pages to simplify and standardise the displays.

I am almost finished with Jared's excellent help earlier on another thread.

I am seeing another problem now.

I have a virtual data source set up to test the template and I can see the data points update every 60 seconds as it should. But the dashboard, when viewed will only update once or twice then it stops updating until the web browser is manually refreshed. Then the dashboard will update once or twice again and then stop.

Any suggestions would be greatly appreciated. Let me know if you want to see the mark up.

How do you know the data source is updating correctly? Can you see the point value continue to change on the Data Point Details page for example?

]]>https://forum.infiniteautomation.com/post/20630https://forum.infiniteautomation.com/post/20630Wed, 05 Dec 2018 17:13:57 GMTHello Jared,
Thank you for getting back to me. I can see the data points update while viewing the watchlist. The timestamp and values are changing every 60 seconds.

In the chrome developer tools / console, I reloaded the page and got a lot of these messages:
(247 actually)

[Violation] Added non-passive event listener to a scroll-blocking <some> event. Consider marking event handler as 'passive' to make the page more responsive. See <URL>
mangoUi~ngMango~ngMangoServices.js?v=0f03d20f7b30b4a597f9:39 [Violation] Added non-passive event listener to a scroll-blocking 'touchmove' event. Consider marking event handler as 'passive' to make the page more responsive. See https://www.chromestatus.com/feature/5745543795965952

Let me know if you need me to try something else.

Cheers

Brian

]]>https://forum.infiniteautomation.com/post/20632https://forum.infiniteautomation.com/post/20632Thu, 06 Dec 2018 16:23:44 GMTHello,
I have set up the permissions so that a user with only viewing permissions can see the template in the hopes that I may get a better idea of the problem here. With the new user logged in I actually get no errors on the chrome debugger for the dashboard page.

I do still see the system stop updating the readings after a few minutes.

Checking the ma.log file I see the following error message each time when the updates stop:

ERROR 2018-12-06T13:19:25,087 (org.springframework.web.socket.handler.ExceptionWebSocketHandlerDecorator.tryCloseWithError:90) - Closing session due to exception for JettyWebSocketSession[id=62d19e47-0c62-2360-e522-c454977e1207, uri=ws://redacted:8080/rest/v1/websocket/point-value]
java.lang.IllegalStateException: WebSocketHandler not found for JettyWebSocketSession[id=62d19e47-0c62-2360-e522-c454977e1207, uri=ws://redacted:8080/rest/v1/websocket/point-value]
at org.springframework.web.socket.handler.PerConnectionWebSocketHandler.getHandler(PerConnectionWebSocketHandler.java:113) ~[spring-websocket-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.web.socket.handler.PerConnectionWebSocketHandler.handleTransportError(PerConnectionWebSocketHandler.java:91) ~[spring-websocket-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.web.socket.handler.WebSocketHandlerDecorator.handleTransportError(WebSocketHandlerDecorator.java:80) ~[spring-websocket-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.web.socket.handler.LoggingWebSocketHandlerDecorator.handleTransportError(LoggingWebSocketHandlerDecorator.java:64) ~[spring-websocket-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.web.socket.handler.ExceptionWebSocketHandlerDecorator.handleTransportError(ExceptionWebSocketHandlerDecorator.java:68) ~[spring-websocket-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.web.socket.adapter.jetty.JettyWebSocketHandlerAdapter.onWebSocketError(JettyWebSocketHandlerAdapter.java:131) ~[spring-websocket-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at sun.reflect.GeneratedMethodAccessor1039.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
at org.eclipse.jetty.websocket.common.events.annotated.CallableMethod.call(CallableMethod.java:71) ~[websocket-common-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.websocket.common.events.annotated.OptionalSessionCallableMethod.call(OptionalSessionCallableMethod.java:72) ~[websocket-common-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onError(JettyAnnotatedEventDriver.java:157) ~[websocket-common-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingError(AbstractEventDriver.java:96) ~[websocket-common-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.websocket.common.WebSocketSession.incomingError(WebSocketSession.java:460) ~[websocket-common-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.websocket.common.WebSocketSession.notifyError(WebSocketSession.java:521) ~[websocket-common-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:573) ~[websocket-common-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:391) ~[websocket-common-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) ~[jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) ~[jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

This looks like a java problem, but I may be wrong. Do I need to upgrade my java version?
Your Linux installation page says JDK 1.8 is required but not the build number.

Cheers

Brian

]]>https://forum.infiniteautomation.com/post/20638https://forum.infiniteautomation.com/post/20638Thu, 06 Dec 2018 16:25:17 GMTI am also seeing this in the threads:

Cheers

Brian

]]>https://forum.infiniteautomation.com/post/20639https://forum.infiniteautomation.com/post/20639Invalid DateOK that's interesting, can you download the thread dump and post it for me when you see the blocked thread again (I'm assuming you might have restarted Mango to clear it).

Do I need to upgrade my java version?
Your Linux installation page says JDK 1.8 is required but not the build number.

Its unlikely that this is related to your Java version. We do no recommend any particular build number, the latest is usually advisable.

]]>https://forum.infiniteautomation.com/post/20645https://forum.infiniteautomation.com/post/20645Invalid DateHello Jared,
I will send the thread dump as requested. It was taken immediately after a Mango restart.

I don't know if it is relevant but when I said I could see the virtual points update in the watchlist, I was referring to watchlist.shtm. I have also had a look at ui/watchlist and saw something strange there, the chart along the bottom continues to update but the list above the chart stops updating.

Here is a screen shot of what I mean:

Cheers

Brian

]]>https://forum.infiniteautomation.com/post/20646https://forum.infiniteautomation.com/post/20646Invalid DateOK thanks for the additional details, that makes sense. The old watchlist page does not use websockets, the chart on the new watchlist page also does not. The table does.
]]>https://forum.infiniteautomation.com/post/20647https://forum.infiniteautomation.com/post/20647Invalid DateHello Jared,
The length of time before the web sockets close seems to be random. I have seen the updates stop as soon as 2 minutes after logging in and as long as 7 minutes after logging in.

Is there any way to prevent the web sockets closing? Maybe a setting in a file that can be saved in overrides? I don't see anything in the env.properties file that would solve this.

Cheers

Brian

]]>https://forum.infiniteautomation.com/post/20661https://forum.infiniteautomation.com/post/20661Fri, 07 Dec 2018 12:44:30 GMT@BG we are still looking into it. Could you also send your env.properties file from overrides (reply to the same support email where you sent the thread dump). I want to check a couple of setttings. Also include the type of processor you are using and how many CPU cores.
]]>https://forum.infiniteautomation.com/post/20663https://forum.infiniteautomation.com/post/20663Invalid DateBrian,

You could try adding this line to the classes/log4j2.xml file to enlighten us as to why/when they are closing. There is a debug log message anytime a point value web socket is closed explaining why.

You will need to either reload the log4j configuration from the system setting page or restart Mango for the changes to take effect.

]]>https://forum.infiniteautomation.com/post/20673https://forum.infiniteautomation.com/post/20673Invalid DateHello Jared,
Thank you. I have sent the details as requested. I have also followed Terry's advice and edited the line. I have also put the edited log4j2.xml file in the overrides/classes folder.

Please Post/Send some of the DEBUG log messages generated when the websocket stops updating the page.

In terms of settings you can change, the defaults in your env properties seem fine:

# Ping timeout for response from browser
web.websocket.pingTimeoutMs=10000
#Time socket can be idle before being closed (ms)
web.socketIdleTimeout=70000

Basically the socket will ping/pong back and forth every 10s so that the 70s idle timeout is never hit. However if the ping/pong messages are lost the connection may be closed. That information should come out in the DEBUG messages.

]]>https://forum.infiniteautomation.com/post/20713https://forum.infiniteautomation.com/post/20713Invalid DateHello Terry,
Thank you for your reply. I won't send the entire ma.log for today since it is 157 pages long, Unless you rally wan it of course. :)

There are a lot of logged events from my template creations and normal run of the mill log entries.

I did look through though and found the following items that may help:

ERROR 2018-12-12T09:18:00,593 error.tryCloseWithError:90) - Closing session due to exception for JettyWebSocketSession[id=26978505-c5f1-2ae3-3372-ec1cdbe0b184, uri=ws://redacted.extension:8080/rest/v1/websocket/events]
java.lang.IllegalStateException: WebSocketHandler not found for JettyWebSocketSession[id=26978505-c5f1-2ae3-3372-ec1cdbe0b184, uri=ws://redacted.extension:8080/rest/v1/websocket/events]
at org.springframework.web.socket.handler.PerConnectionWebSocketHandler.getHandler(PerConnectionWebSocketHandler.java:113) ~[spring-websocket-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.web.socket.handler.PerConnectionWebSocketHandler.handleTransportError(PerConnectionWebSocketHandler.java:91) ~[spring-websocket-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.web.socket.handler.WebSocketHandlerDecorator.handleTransportError(WebSocketHandlerDecorator.java:80) ~[spring-websocket-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.web.socket.handler.LoggingWebSocketHandlerDecorator.handleTransportError(LoggingWebSocketHandlerDecorator.java:64) ~[spring-websocket-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.web.socket.handler.ExceptionWebSocketHandlerDecorator.handleTransportError(ExceptionWebSocketHandlerDecorator.java:68) ~[spring-websocket-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.web.socket.adapter.jetty.JettyWebSocketHandlerAdapter.onWebSocketError(JettyWebSocketHandlerAdapter.java:131) ~[spring-websocket-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
at org.eclipse.jetty.websocket.common.events.annotated.CallableMethod.call(CallableMethod.java:71) ~[websocket-common-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.websocket.common.events.annotated.OptionalSessionCallableMethod.call(OptionalSessionCallableMethod.java:72) ~[websocket-common-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onError(JettyAnnotatedEventDriver.java:157) ~[websocket-common-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingError(AbstractEventDriver.java:96) ~[websocket-common-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.websocket.common.WebSocketSession.incomingError(WebSocketSession.java:460) ~[websocket-common-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.websocket.common.WebSocketSession.notifyError(WebSocketSession.java:521) ~[websocket-common-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:573) ~[websocket-common-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:391) ~[websocket-common-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) ~[jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) ~[jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

These were the ones i found searching through the log for: closed or error.
Please let me know if you want more information or if you need me to search through the logs for specific strings.

]]>https://forum.infiniteautomation.com/post/20722https://forum.infiniteautomation.com/post/20722Wed, 12 Dec 2018 20:33:03 GMTHello Terry,
I also just wanted to add that since I have been working from home tonight on my templates, I have not seen the dashboards stop updating.

It looks like the ping/pong failure is the problem. The other messages are expected during normal operation.

FYI on the status code meanings:

Status Code 1001 (Going Away) - This is normal behavior when a page is navigated away from
Status Code 1006 (No Close Frame) - Connection was terminated in an unclean way, it appears the socket was broken.

I find it odd that 10s is not long enough to send/receive the messages so I'm thinking you have a network with packet loss. If you can I would suggest pinging the Mango server to see if that is correct. If not then perhaps increasing the env.properties value of

web.websocket.pingTimeoutMs

will help.

]]>https://forum.infiniteautomation.com/post/20726https://forum.infiniteautomation.com/post/20726Invalid DateHello Terry,Thank you (and your team) for looking into this. Your explanation makes sense now. Our office is on an industrial estate that has a fairly dodgy broadband connection and my home broadband is fine. The packet loss will explain why I am seeing this issue at work. I don't often work on this from home but am making a push to get our Mango 3 implemented.

I will do some ping tests from home and from work to confirm that is the problem. and make the recommended changes to the timeouts.

Again, thank you.
Cheers

Brian

p.s. I wish you all a Merry Christmas.

]]>https://forum.infiniteautomation.com/post/20729https://forum.infiniteautomation.com/post/20729Invalid DateHello Terry,
Unfortunately that has not solved my problem. I have checked the ping and packet loss on our broadband here at work and while it shows some delays there is not packet loss between my desktop and the Mango 3 server.

I have increased the timeout to 30 seconds but am still seeing the page stop updating after a few minutes.

I can't roll this out to our customers until I am sure the connection to the Mango server is reliable and robust.

Any other suggestion would be greatly appreciated.

Cheers

Brian

]]>https://forum.infiniteautomation.com/post/20734https://forum.infiniteautomation.com/post/20734Invalid DateHere is the last log entry at the time of the last disconnect we had:

]]>https://forum.infiniteautomation.com/post/20739https://forum.infiniteautomation.com/post/20739Invalid DateHello Terry,
I think I finally found the culprit. Our anti-virus is causing the drop outs. I turned it off and have run now for 30 minutes with no connection issues.

I am sorry to have bugged you guys about it but thank you for your help and patience.

I replicated the closing of a websocket due to the ping/pong timeout and found that my system handles it gracefully without throwing that java.lang.IllegalStateException. The page stops updating but no errors are produced and I get the same log message about "Didn't receive Pong from Endpoint within..." I replicated this by making the Mango server delay to read the pong message.

I'm wondering if your browser isn't sending the pong message... (This should happen automatically by most every browser and isn't something you need to code into your page)

For yet more information on the connection failure you can add this to your log4j2.xml file