KMS Network anomaly on 9/1 post-mortem

This morning at 9:21 we received our first help desk ticket that something wasn’t quite right with using the Internet at Kenton Middle School. The second request came at 9:36, after which I posted the first message to @kcsmstech alerting users to the issue.

There was still connectivity, that’s why our monitoring system didn’t register an error. The problem was that network traffic wasn’t going through to the Internet. At first I assumed it was the district Internet gateway, so I started debugging the various services on it, to no avail. I decided to begin I began rebooting all of the network devices between the middle school and the Internet to try to solve the issue as soon as possible:

KMS Router – no change

KHS Switch – no change

District Gateway – no change

Finally, I directed my attention to the wireless bridges that connect the HS and the MS. This is when I found out that the problem was in the fact the data passing on the wireless link was very degraded. I restarted both ends, but it still had issues. After checking a few other things, I decided to restart the wireless bridge at the HS. The link then came up normal, and network access to KMS was restored. Once the link appeared to be up, I ran some transfer tests to verify that the link was up and stable. At 11:48am the alert announcing that the KMS network was working normal was broadcast.