Network outage, June 9-10, 2012

All work was completed by 6:30 PM on June 9. The original schedule remains here for reference.

There will be a major network outage on June 9 and 10, 2012—the weekend after MIT Commencement. This will affect all CSAIL network services, including all Web servers, webmail, and digital telephones, for some part of that weekend. It will not affect card-access locks, analog telephones, or cellular phones.

TIG will be taking all switches in the network down for firmware updates to fix bugs that were discovered during our 2011 network upgrade and subsequent unscheduled outages. The schedule will run approximately as follows. (If work goes faster than expected, we will proceed ahead of this schedule to limit the overall downtime.)

Saturday, June 9

9:00 AM: TIG and vendor personnel arrive to prepare for the updates. Virtualized servers are taken offline. This includes all CSAIL Web servers.

10:00 AM: Core switches are taken down one at a time and upgraded. This may require multiple reboots. The network should remain up, except in some locations that only have service from a single core switch, but will be unstable as the remaining switches adjust to changes in the network topology as the core switches go down and up.

11:30 AM: The switch serving many core services in room 341 will be taken down and upgraded. This will require multiple reboots and will interrupt many critical services. The process will take about an hour.

3:00 PM: Machine-room switches in rooms 399, D471, G474, and G535 will be upgraded. Each room will be down for between half an hour and an hour, depending on the number of devices in each switch that need to be upgraded.

6:30 PM: Work ends for the day.

Sunday, June 10

9:00 AM: TIG and vendor personnel arrive to prepare for the updates.

10:00 AM: Office switches will be upgraded, probably two at a time, starting with the Dreyfoos tower from upper floors to lower, followed by the Stata warehouse floors and the Gates tower. Each area will take between half an hour and an hour.