For a few hours now, the heading that appears on the home page of Stack Exchange sites notifying of newly active questions has not been appearing. I have had to manually refresh in order to see new activity.

I have observed this on SF, SU and right here on mSO.

Upon hitting F12 in Chrome and looking around, it appears that the WebSockets connection to sockets.ny.stackexchange.com is failing, though Chrome doesn't tell me exactly why.

On further investigation, I also seem to not be receiving notification of reputation changes, upvote/downvote changes, new comments notifications, flags, or inbox notifications.

1 Answer
1

On our load balancer, we have multiple instances of HAProxy. One of these instances is dedicated to realtime updates (websockets) and somehow failed. After taking a core dump of that instance, I killed the process and started it again to get websockets working again.

Our current next steps are:

Keep investigating to try to figure out exactly what went wrong that caused websockets to stop responding

Set up more extensive monitoring for websockets so we have an alert when websockets stops responding