RECURRENCE: Network and telephone service, down for some users, 11/30/16, 2:52-2:55PM

UPDATE: 3:02PM, 11/30/16

We experienced a brief recurrence of the wireless network outage in WUSOC from 2:52-2:55PM today.

Technicians continue to research underlying cause with vendor.

Some users experienced a brief wireless network interruption this morning at approximately 8:15am, and resolved by 8:30am. Reports and calls to OIT indicate impact was likely limited to the WUSOC building, and a limited area on the Dublin campus.

OIT technicians are working with our vendor, Juniper, to determine the root cause.

Please contact the IT Service Desk at 740-593-1222 if you are experiencing lingering issues.

Should you experience any lingering issues, please contact the IT Service Desk at 740-593-1222.

UPDATE - 11:47AM, 11/15/16

While some users report Concur access having been restored, the vendor's status site still reads that the outage is ongoing. Until we receive confirmation that service is fully restored, please either proceed with caution or (preferably) delay work in Concur until an "all-clear" is posted.

AFFECTED SERVICES:

Concur

IMPACT

Down for all users: Concur's North American data center is experiencing a complete outage. At this time, no further information is available.

OBI Service restored at 8:57AM.Technicians continue to investigate root cause.

AFFECTED SERVICES:

Oracle Business Intelligence (OBI Reporting)

IMPACT

Down for all users; we are restarting OBI services, and anticipate service restoration by 9:15AM.

Periodic updates will be posted until service has been restored.

e-Business, partial outage, 11/2/16, 12:09 pm ||RESOLVLED

1:28 pm: e-Biz: Service restored. The outage was an unintended consequence of preparations for upcoming scheduled maintenance on BobcatBUY. Please report any lingering issues to the IT Service Desk, https://www.ohio.edu/oit/help

1:00 pm: Identified:

eBiz: This outage is related to a problem with one of the servers that gets randomly assigned to a user when they log in. We are in the process of fixing this problem. In the meantime, you way be able to work around this problem with one or more of these steps:

We are seeing a recurrence of DNS errors which, like this morning, appear to be a result of Amazon Web Services problems in a large portion of the country. This is impacting many users' ability to access Box, Qualtrics, SciQuest, and many other websites and services, and is beyond our control.

We will continue to monitor the situation and update the university community as we learn more.

This morning's accessibility issues were determined to be a result of Amazon Web Services problems in the eastern US from 7:31AM until 9:10AM, and are confirmed to be resolved. AWS supports a large array of services, including Box, Qualtrics, and BobcatBuy's platform. For additional information, visit http://status.aws.amazon.com/

We have been receiving reports this morning of intermittent problems accessing off-campus services like Box and Qualtrics. These problems appear to be related to an issue that is affecting a number of Internet providers in the entire eastern US. We will continue to monitor the problem.

At this point, the issue appears to be fixed for Qualtrics, but we continue to receive Box-related reports.

University telephones, down for some users, 10/19/16, 9:00 am

AFFECTED SERVICES:
University telephones

We are currently experiencing trouble with outgoing toll-free 800 calls, as well as trouble receiving calls from off-campus. Callers dialing from off-campus are receiving a busy signal. Our technicians are investigating the trouble.

IMPACT
Down for some users

Periodic updates will be posted until service has been restored.

Network & telephones in Library Annex down for all users,10/17/16, 9:30-9:44AM

*Current Status:* We're investigating an issue in which some users are receiving a certificate prompt in the Outlook client. We’re currently reviewing service logs and data associated with the certificate in an effort to identify the next troubleshooting steps.

*User Impact:* Users receive a certificate prompt in the Outlook client. The prompt does not interrupt connectivity to the service.

*Scope of Impact:* Your organization is affected by this event. Potentially any user accessing their email in the Outlook client is affected.

Service restored. Microsoft was able to confirm that they were able to identify and resolve a problem on their end that was affecting web logins for some Catmail users. Please report any lingering issues to the Service Desk at 740-593-1222 or https://www.ohio.edu/oit/help

AFFECTED SERVICES

Catmail on the web (OWA)

IMPACT

We are receiving reports that some users are unable to log in to catmail.ohio.edu

9/8/16, 5:51 pm - Service restored. Technicians continue to investigate the cause of the problem.

Ebiz Absence Requests and Absence Request approvals are slower than normal.
We're seeing approximately 30-second delays after clicking Submit or Approve (the submissions and approvals are processing, just delayed).

Technicians are investigating; periodic updates will be posted until service has been restored.

AFFECTED SERVICES:
Ebiz Absence Management / EMS

Printing, down for some users, 9/1/16, 8:00am-8:20am

AFFECTED SERVICES:
Departmental printing to some printers; error displayed as a driver issue

IMPACT
Printing was interrupted for some users; issue has been resolved on the Print server. Users still experiencing this issue should reboot their computer and re-attempt. Please advise the Service Desk if issues persist beyond reboot.

DURATION

Issue was initially reported by departments shortly after 8:00am, and resolved on the Print server by 8:20am.

Footprints, down for all users, 08/30/16, 05:22 AM || RESOLVED

08/30/16, 06:00 AM: Service restored.

Please report any lingering issues to the OIT Service Desk at https://www.ohio.edu/oitech or 740-593-1222.

AFFECTED SERVICES:
Footprints

IMPACT
Down for all users

Periodic updates will be posted until service has been restored.

Application for Graduation, down for all users, 8/29/16, 11:40 am || RESOLVED

8/29/16, 11:53 pm - Service restored. The part of the app that verifies the user's address needed to be rebooted.

Fiber optic cable repairs caused the primary firewall to go offline. Service was restored once the firewall was rebooted. Based on user reports during the incident, the outage was brief and did not affect everyone. We are working with our vendor to identify why a repair that should have been non-disruptive affected our primary connection.

The emergency maintenance has been completed. We are currently showing no errors on the Blackboard system. As a root cause has not yet been determined, we and the vendor are going to continue to monitor the system closely.

Additional maintenance will be performed on the system beginning at 12:01am Wednesday, August 3.

AFFECTED SERVICES
Blackboard Learn

IMPACT
Degraded performance: We have received a number of reports of difficulties with Blackboard Grade Center and Discussion Boards, and are working with the vendor to diagnose and resolve the issue. Symptoms reported thus far include difficulties accessing Grade Center, submitting exams, and accessing Discussion Boards.

Periodic updates will be posted until service has been restored.

SPSS, down for all users, 8/1/16, 8:00 am

AFFECTED SERVICES

University-licensed copies of SPSS

IMPACT

Down for all users

ADDITIONAL DETAILS

The university’s site license for SPSS expired today, August 1. We have asked the vendor to renew the license; however, the soonest that may happen will be tomorrow, August 2. Until that renewal gets processed, SPSS will not work.

We apologize for this disruption. In recent weeks, we have been actively negotiating with the vendor about a price increase. We have reached an agreement, but the vendor has declined to offer a good faith extension of the license while the renewal is processed. We will be examining our software licensing procedures for ways to prevent this sort of issue happening in the future.

Network & telephones in Bryan Hall, down for all users, 8/1/16, 9:52am || RESTORED at 1:40pm

Scope of Impact: Customer reports indicated that many users likely experienced impact related to this event. Our analysis indicates that this issue may potentially have affected any of your users attempting to send or receive mail.

Next Steps:
- We're analyzing performance data and trends on the affected systems to help prevent this problem from happening again.
- We're reviewing our code for optimizations and automated recovery options.

We'll publish a post-incident report within five business days.

UPDATE [3:35PM, 6/30]:

A service outage affecting many of Microsoft's Office 365 customers (including Ohio University) caused mail delivery delays. The problem has been resolved and backlogged mail should be delivered over the next several hours.

We will continue to monitor the issue here and with Microsoft. Due to the volume of users affected, the message backlog is extensive and may take time to clear.

AFFECTED SERVICES:
Scan-to-Print functionality from Xerox MFPs (emails of scanned documents not arriving in recipient mailboxes)
This appears to also affect SMS-to-Email messaging.

Please report any lingering issues to the OIT Service Desk at https://www.ohio.edu/oitech or 740-593-1222.

AFFECTED SERVICES:
Network Attached Storage

IMPACT
Degraded performance

Periodic updates will be posted until service has been restored.

Ad Astra, down for all users, 6/14/16, 7:39 pm || RESOLVED

6/15/16 7:04 pm - Service restored. The Ad Astra vendor was able to identify and fix data integrity issues within the application's database. The vendor was unable to determine the cause, and will be analyzing application logs in search of it.

AFFECTED SERVICES

Ad Astra

IMPACT

Down for all users

ADDITIONAL DETAILS

Earlier today, our vendor performed an upgrade. During post-upgrade functional testing, the application did not perform properly and has been down ever since. We have a support case open with the vendor but do not expect to see a resolution until at least the morning of 6/15.

e-Business & related apps, degraded performance, 06/08/16, 01:31 pm

AFFECTED SERVICES:• Oracle E-Business Suite Workflow Mailers

IMPACT:• We are experiencing delays in Absence Management notifications and automated reports normally sent through Workflow Mailerso Supervisors may continue to view or approve/reject absence requests in their Worklist located on the eBusiness Suite Homeo Page: https://ebiz.ohio.edu/

Power outage affecting network on Ridges, West Green & Courtyard, down for all users 6/4/16, 9:40 am || RESOLVED

6/4/16, 1:10 pm - Service restored.

A power outage is affecting network & telephone services in some locations on the Athens campus. Facilities estimates restoration around 1:30 pm. Network will come back online shortly after power is restored.

Note from Concur technical team: Please be advised that we are seeing reports of performance issues with both Travel and Expense this morning in the form of connection issues, slow performance, and problems completing travel transactions. Please continue to log cases as you normally would and I will escalate them accordingly. Our development teams are actively working to resolve any issues that clients are facing.

3:10 pm follow up from Concur: Though Expense and Travel have regained login capabilities, many of you are still experiencing slow performance from the first P-1 we had this morning. As you are now able to access the Support portal, please continue to log performance cases as you normally would. My apologies for any inconvenience this causes you and your users; please know that our development teams continue to work tirelessly to investigate and ensure our systems go back to working as normal.

OBI Reporting, down for all users, 05/26/16, 07:38 am/pm || RESOLVED

5/26/16 7:44 am - Issue was an unintended side effect of automated patching. Will explore options to prevent this problem in the future.

*Current Status:* Engineers are investigating an issue in which some users are experiencing problems with meeting update notifications on their iOS and Android mobile devices. It was determined that a recent update caused this issue. Engineers have developed a fix for the underlying code issue and are now testing it. Engineers anticipate that the solution will be ready to begin deployment within 4-6 hours. Once the deployment is underway, it may take up to 72 hours to complete.

*User Impact:* Users are receiving repeated meeting update notifications on their iOS, Android, and Windows Mobile devices. This is only affecting meetings that were created or updated from an iOS, Android, or Windows Mobile device connecting to the service via Exchange ActiveSync (EAS).

*Scope of Impact:* A very limited number of customers and users appear to be impacted. Some customers have reported that their organization is affected by this issue.

Most OHIO online services, down for all users, 5/13/16, 8:17 am || RESOLVED

5/13/16, 9:45 am - Service restored after rebooting the university’s firewall. Technicians continue to work with engineers from our firewall’s manufacturer to determine whether this outage is related to similar outages in the past and to identify steps to prevent future outages.

Intermittent degraded performance, including slow network access and telephone calls where one party is unable to hear the other.

ADDITIONAL DETAILS

We have received reports of this issue in Baker, Chubb, Cutler, Clippinger, Grover, Stocker and Wilson Hall (administrative building). If you encounter these issues in your location, please report them to the IT Service Desk, https://www.ohio.edu/oitech or 740-593-1222.

University telephones in Chubb Hall, down for some users, 5/9/16, 8:57am

Update, 11:35am:

Technicians are working with our vendor in continued investigation of telephone network issues.

Please be advised that some computers' wired connections may be affected by the same problem affecting the phones; we are looking into the entirety of network reports in the building.

AFFECTED SERVICES:Wilson Hall (College Green) Wired Internet WiFi University telephonesIMPACT Degraded performance and intermittent outages in some areas.Periodic updates will be posted until service has been restored.

Network and telephone service, down for some users, 5/6/16, 1:40pm || RESOLVED

5/6/16, 9:50 pm - Service restored.

Outage was caused by physical damage to a major fiber optic cable. The cable was not entirely severed, which is why the issue was only intermittent. We have bypassed the section until repairs can be completed. In the meantime, network performance should be back to normal.

5/6/16, 8:01 pm - Techs continue to work with vendor to troubleshoot intermittent issues. Will update status in morning, or sooner if issue is resolved.

5/6/16, 4:15 pm: We will be disabling and re-enabling the network in individual buildings in an attempt to isolate the issue.

5/6/16, 3:50 pm: Network and telephones continue to experience intermittent interruptions for some users. Vendor support engineers are assisting with the troubleshooting.

AFFECTED SERVICES:
Wired Internet
WiFi
University telephones

IMPACT
Down for some users

Periodic updates will be posted until service has been restored.

Adobe Connect, degraded performance, 5/6/16, 10:51 pm

5/12/16 - Service currently is available; however, a few student account logins may not work correctly. If you are unable to invite a student to a meeting room, set the meeting to either open access or to allow approved guest access and have that student enter as a guest.

Electrical service restored per Dublin campus and AEP (approx. 1:30PM).

There is currently a power outage on the Dublin campus, affecting buildings A and D. AEP advises this was caused by a substation failure, and are projecting restoration of power as early as 3:00pm today.

Network battery backups report sufficient supply for at least the next few hours, but should they be exhausted, both network and telephone connectivity will be affected.

4/21/16 - The Class Climate email issues we had on Friday April 15th were caused by human error when someone accidentally deleted data from the system. We have rebuilt the lost participant emails to the best of our abilities for the current term. To reduce the likelihood of this happening again, we will be contacting everyone with administrative rights to make sure they understand how this happened and what they should do to avoid it in the future. We also will investigate technical options to restrict access to this particular function.

4/15/16, 3:00 pm - Service restored. Email addresses were added back into the system.

AFFECTED SERVICES

Class Climate

IMPACT

Down for some users

Unable to distribute evaluations because email addresses come back as missing

Network in Academic Research Center, down for some users, 04/15/16, 08:38 am RESOLVED

04/15/16, 09:06 am: Service restored.

Please report any lingering issues to the OIT Service Desk at https://www.ohio.edu/oitech or 740-593-1222.

4/15/16, 4:21 pm - Service restored. Lancaster campus Facilities has confirmed that he electrical power problems that caused this outage have been repaired.

2:55 pm, 4/15/16 - This outage was the result of a partial power outage that is affecting our networking equipment on the Lancaster campus. Power still has not been fully restored, so the outage continues.

4/15/16, 5:03 pm - Service restored. Technicians continue to investigate the root cause of the problem. Please report any lingering issues to the OIT Service Desk at 740-593-1222 or https://www.ohio.edu/oitech

AFFECTED SERVICES

Student printing via Pharos

Faculty/staff printing via Pharos

IMPACT

Degraded performance

Network & telephones at the Ridges and University Courtyard, down for most users, 04/13/16, 2:15PM-2:55PM || RESOLVED

*Next Steps:* Review procedures for validating service updates as the configuration issue was not identified during the testing phase of the development process. A post-incident report will be available on the Service Health Dashboard within five business days.

AFFECTED SERVICES
Mac Mail & iCal

IMPACT
Down for some users

ADDITIONAL DETAILS

Microsoft has identified an issue on their end that is affecting calendar synchronization for some Mac iCal users. They will be rolling out a fix in the next few days. It's unclear if this fix will also resolve similar sync problems with Mac Mail. Outlook and OWA are not affected by this issue.

Periodic updates will be posted until service has been restored.

Bobcat Buy, down for all users, 4/1/16, 11:15AM || RESOLVED

4/1/16, 1:17 PM: Service restored.

Please report any lingering issues to the OIT Service Desk at https://www.ohio.edu/oitech or 740-593-1222.

Degraded performance: users receiving an 'untrusted connection' when trying to access the app

Pharos printing & VDI/VMware Horizon View, 3/19/16, 9:07 am

3/19/16, 11:08 am - Service restored

Additional work will be needed during tonight's maintenance window to ensure that issues do not come back. Our vendor engineers have yet to isolate the root cause of these network problems, so work likely will be needed during future maintenance windows as well.

3/19/16, 10:25 am - We have made some network changes that appear to have eliminated the issues. Please report any failed print jobs or VDI session interruptions to the OIT Service Desk.

AFFECTED SERVICES

Pharos printing

Student printing in Athens campus locations like Alden Library

Faculty/staff printing via Pharos

VDI/VMware Horizon View client

IMPACT

Degraded performance

Student print jobs may fail and result in a "[free print]" error.

Faculty/staff print jobs will fail silently with no error message

Possibility of interrupted connections for VDI sessions

Concur, down for all users, 03/18/16, 10:34 am || RESTORED

3/19/16, 1:45 pm - Service restored

3/19/16, 9:40 am - Concur is still down this morning. Unclear whether the vendor applied their fix last night or not. We have reported the issue to them.

3/18/16, 9:40 pm - Concur remains down, but their technical support states that they have made progress in identifying a fix and intend to apply that fix later tonight.

3/18/16, 12:24 pm - We have received word from our Concur vendor that this a global problem on their end affecting all Concur users. They expect to have the problem resolved before the close of business today. We will send an update when service is restored.

03/18/16, 10:34 am: Technicians continue to troubleshoot. We will update this status every two hours until the outage has been resolved.

3/13/16, 11:57 pm - Service should not be experiencing any network errors at this point; however, additional work still may be needed in the future.

3/11, 6:26 pm - We will be making some changes tonight to our central network equipment. Expect a single, less than 5 minute outage some time during the published 12:05 am - 2:00 am window on 3/12. Engineers will continue to troubleshoot throughout the weekend, but with no expected outages other than those during the already published nightly window.

Friday, 3/11/16, 4:30 pm

Changes made during today's troubleshooting have temporarily resolved the network issues that were affecting Pharos printing and VDI/VMware Horizon View; however, the problem has only been bypassed. Additional work will be needed to identify the root cause and apply a permanent fix.

This is a complicated process: Multiple engineers from our network vendor have been on site working in tandem with OIT staff since March 10. The troubleshooting has involved multiple changes to our network setup followed by intense analysis of the effects of those changes. For example, today's work involved replicating the conditions of last month's outages and then capturing data from several interruptions as they happened.

This detailed analysis is starting to pay off. We have identified likely problem areas and will focus our attention on those areas in the coming days. Doing so may require some brief interruptions during our advertised 12:05 am - 2:00 am windows. We will do our best to minimize any such interruptions.

AFFECTED SERVICES:

ID card swipe student printing

Faculty/Staff printing via Pharos

VDI/VMware Horizon View

IMPACT

Intermittent interruptions

These interruptions are part of ongoing work to identify and repair the problem that caused outages during February.

Monday, 2/29/16, 4:46 pm - Following yesterday's data center maintenance, most of the ID swipe and Pharos printing errors appear to have been resolved; however, we will continue to keep a close eye on the service as students return from spring break.

If you continue to have problems with documents not printing, please report them to your local lab manager or printer support staff. You also can call the OIT Service Desk at 740-593-1222 or submit a request online at https://www.ohio.edu/oitech

This Sunday at 12:05 am, scheduled maintenance in the university data center that may help resolve the problem. In the absence of any new developments, the next update on this outage will be sent on Sunday.

2/25/16, 11:10 am - Technicians continue to troubleshoot. Until this issue is resolved, you usually can fix a failed print job with the following steps:

Students: Print job fails with an error message.

Go to a different ID-swipe printer within 1 hour.

Swipe your ID card and look in your "hold queue" for a print job with the label "[Free Print]".

Select that job, and your document should print.

Faculty/Staff: Print job fails to print, no error message.

Reboot the affected printer.

Re-send the document that failed to print.

AFFECTED SERVICES:

ID card swipe student printing

Faculty/Staff printing via Pharos

IMPACT

Intermittent interruptions

Athens campus printers only

ADDITIONAL DETAILS

Print jobs will fail randomly, often with one of the following errors:

Error releasing print job: Cannot connect to printer

Error releasing print job: Failed to render document

In some cases, students may be charged for the failed job

Restarting the affected printer will sometimes allow the job to complete

9:52 am - Outage was caused by a hardware failure in a piece of data center infrastructure combined with a bug that prevented secondary/backup hardware from taking over seamlessly. Defective unit has been replaced, and our vendor will be sending us replacements for the backup units as well that have been patched to remove the bug that caused the problem.

2/25/16, 10:07 am - We continue to see random print job failures for both student ID card swipe printing and faculty/staff printing via Pharos. IT technicians and vendor tech support continue to look for a solution.

2/22/16, 6:45 pm - OIT and vendor technicians spent today troubleshooting intensively on the Pharos print environment; however, the service continues to experience intermittent interruptions that require restarting individual printers to allow print jobs to complete. We will post an update again tomorrow.

2/21/16, 4:55 pm - Alden Library experienced a period of disruption where printers needed to be restarted before nearly every print job. After several server restarts, service appears to be working normally again. We continue to work with our vendor to identify the cause of this problem.

2/19/16, 2:41 pm - Printers continue to experience occasional interruptions. Restarting the affected printer will restore service. Vendor technical support and OIT staff continue to work on indentifying and eliminating the cause.

9:30 am: Service has been restored. Actual extent of outage was limited to AdAstra and Pharos mobile printing (myprincenter.ohio.edu). Both appear to have been side effects of yesterday's data center outage. Network slowness was caused by a denial of service attack against OARnet, our Internet service provider between 8-8:30 am.

Thursday, 2/11/16, 8:34 am

AFFECTED SERVICES:

Wired & wireless Internet

IMPACT:

Intermittent interruptions

These interruptions are affecting end user connections as well as data center connections, so the problem can appear to be either a complete lack of Internet access or an inability to log into a central service like PeopleSoft SIS

Feb 10, 4:34 pm - 6:06 pm: Most OHIO online services || RESOLVED

Wednesday, 2/10/16, 4:44 pm - 6:06 pm

AFFECTED SERVICES

Most OHIO online services, including

Blackboard Learn

OHIO Front Door

PeopleSoft SIS & My OHIO Student Center

Oracle e-Business

IMPACT

Down for all users

ADDITIONAL DETAILS

Technicians are actively troubleshooting an issue with the university data center

Existing web sessions for cloud hosted services like Catmail, Qualtrics and Box appear to be unaffected; however, if you log out of the service, you may not be able to log back in.

Feb 5, 8:39 am - 8:47 am: Titanium || RESOLVED

Friday, 2/5/16, 8:39 am - 8:47 am

Service restored after restarting file storage that did not properly come back online after an automated system update.

AFFECTED SERVICES

Titanium (Counseling Services app)

IMPACT

Down for all users

February 4, 5:36pm Catmail || RESOLVED

Thursday Februrary 4, 2016 6:00pm: Service restored. Outage was caused by a lack of disk space on the shibboleth server.

Thursday February 4, 2016 5:36pm

AFFECTED SERVICES:

Catmail

IMPACT:

Some users are reporting that they receive a redirect error when attempting to login.

Feb 2: 12:12 pm - 12:18 pm: Identity manager/iTIM || RESOLVED

Service restored at 12:18 pm. Techs have identified the function that caused this outage and have updated the system to prevent future outages.

AFFECTED SERVICES

Identity Manager

Password changes

Account activations

IMPACT

Down for all users

Jan 31, 2:35 pm: Dougan Hall wired & wireless Internet || RESOLVED

Service restored at 4:10 pm. Outage caused by a faulty hardware module. Module has been replaced.

AFFECTED SERVICES

Dougan Hall wired & wireless Internet

IMPACT

Down for all users

Jan 30: Netreg || RESTORED

AFFECTED SERVICES

Network registration web tool

IMPACT

Down for all users

Server did not come back up properly after an automated system update. Service restored after rebooting the affected server.

Jan 30: Internet access, intermittent

10:47 am - OARnet is applying technical measures to filter out the denial of service attack traffic

AFFECTED SERVICES

Internet access

Users on the university network unable to access sites outside of the university

1/26/16, 10:20 am: We have been unable to reproduce the problem and have received no user reports since yesterday. Techs will continue to monitor the telephone system in case the problem reappears.

1/25/16, 3:58: We continue to look into isolated, intermittent cases where callers have difficulty connecting with off campus numbers. Telephone service carriers have been engaged to help troubleshoot.

AFFECTED SERVICES:

University telephones

IMPACT:

Currenlty investigating reports of intermittent problems with 'dead air' on calls between university lines and off campus

Jan 22, 12:15 pm: IMAP access to Catmail, degraded performance

Friday, 1/22/16, 12:15 pm

AFFECTED SERVICES

IMAP access to Catmail

IMPACT

Degraded performance for some users

Microsoft currently is working on fixing this issue. Status update from Microsoft:

*Current Status:* Engineers completed the development phase of the fix. Subsequently, engineers tested the fix and confirmed that it restores service. Engineers are now deploying the fix to the affected environment.

*User Impact:* Affected users are unable to receive email messages when accessing Exchange Online through Internet Message Access Protocol (IMAP). As a workaround, users may be able to access email via Outlook on the web. [Catmail: https://catmail.ohio.edu ]

*Customer Impact:* Analysis of incident scope indicates that a limited number of customers appear to be impacted by this event. However, those customers affected likely have a large number of users experiencing impact. Engineers have received a few isolated customer reports of this issue.

*Preliminary Root Cause:* As part of our efforts to improve service performance, an update was deployed to a subset of code which is responsible for obtaining the subscribed folder list. However, the update caused a code issue that prevented the list from being automatically loaded.

Resolved at 8:00 am. Outage was caused by a lack of system resources on the host server. This only became an issue after deploying several new apps without adding resources. To prevent a recurrence, we will be upgrading the server on Jan 16.

Service resoloved after restarting a wireless controller. Tech noted that the temperature in the closet housing the controller was well below normal operating range for that kind of equipment.

Tuesday, 1/12/16, 6:10 pm - 7:44 pm

AFFECTED SERVICES:

Convocation Center

Wireless Internet (WiFi)

IMPACT:

Services are completely down

Jan 11, 8:49 am - 9:00 am: Network registration tool

Monday, 1/11/16, 8:49 am - 9:00 am - RESOLVED

AFFECTED SERVICES:

Online tool for registering devices on the wired network (NetReg)

IMPACT:

Service is completely down

January 6, 2:30pm-2:55pm Catmail OWA||RESOLVED

3:44 pm - Microsoft has released the following Office 365 outage report. They are listing the problem as resolved, but no root cause.

Incident: MO41617

Final Status: The investigation determined that there was a temporary issue affecting routing of requests to the Microsoft Online Portal. This issue has been resolved and engineers have confirmed that the service is no longer impacted. The root cause is under investigation.

User Impact: Affected users were unable to sign in to the Office 365 service.

Customer Impact: Analysis of incident scope indicated that many customers appeared to be impacted by this event. For those customers affected, this issue potentially affected any user attempting to log in to the service. Some customers reported that their organization was affected by this issue.

Incident Start Time: Wednesday, January 6, 2016, at 19:21 UTC

Incident End Time: Wednesday, January 6, 2016, at 19:50 UTC

Preliminary Root Cause: The root cause of this incident is not known but is under investigation. Based on the impact of this incident the root cause will be provided, if identified, within 30 days.

Next Steps: The following is a list of known action item(s) associated with this incident. As part of the Office 365 problem management process, additional engineering actions may be identified to improve the overall service.

Action: Review the monitoring infrastructure for improvements as this event was reported by customers before the investigation prompted a communication to be published to the Service Health Dashboard.

Action: Identify the underlying root cause and develop a solution to prevent reoccurrences.

10:25 am - Service restored. All outages were caused by a portion of our central storage system not properly coming back online after unrelated, scheduled maintenance earlier this morning. Restarting the affected system resolved the outage.

Issue Recurrence at approximately noon. Technicians are once again looking into the issue.

Issue Resolved at approximately 11am.

The web for that allows users to submit a Facilities work order through the Facilities web site currently is not working. This is a side effect of the weekend's CommonSpot upgrade. Technicians are working to restore the feature.

Network down in Herrold 215A & 271. Tech has been dispatched to investigate an unresponsive piece of network hardware.

Dec 19, 9:54 pm: VDI Blast || RESOLVED

Issue resolved after server reboot. Security cert was recently replaced that requried a restart, but the automatic restart did not happen properly.

VDI Blast is unavailable for some Engineering users. Server reboot scheduled for 12/20 at 12:01 am to address the issue. Less than 1 minute interruption for anyone in a VDI session when the reboot happens.