Planning Center Status - Incident Historyhttp://status.planningcenteronline.com
StatuspageTue, 26 Sep 2017 01:01:44 -0700S3 experiencing increased error rates
<p><small>Sep 14, 13:20 PDT</small><br><strong>Resolved</strong> - Amazon has indicated that their issues are resolved. Planning Center is fully functional.</p><p><small>Sep 14, 12:58 PDT</small><br><strong>Monitoring</strong> - The S3 issues appear to have been resolved by Amazon. We're still monitoring the situation.</p><p><small>Sep 14, 12:09 PDT</small><br><strong>Investigating</strong> - Amazon has noted increased error rates for their S3 service: http://status.aws.amazon.com/
<br />We're currently monitoring the situation.</p> Thu, 14 Sep 2017 13:20:47 -0700http://status.planningcenteronline.com/incidents/s8gftjm0q0db
http://status.planningcenteronline.com/incidents/s8gftjm0q0dbScheduled Database Maintenance
<p><small>Aug 28, 23:01 PDT</small><br><strong>Completed</strong> - Our database maintenance is complete. We thank you for your patience.</p><p><small>Aug 28, 22:10 PDT</small><br><strong>Update</strong> - Scheduled maintenance for all Planning Center apps has started. We will keep you up to date!</p><p><small>Aug 28, 22:00 PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug 28, 11:35 PDT</small><br><strong>Scheduled</strong> - Planning Center will be unavailable the night of Monday, August 28th at 10pm Pacific Time while we perform database upgrades. We expect this to take about two hours. During this time, All services will be unavailable.</p> Mon, 28 Aug 2017 23:01:19 -0700http://status.planningcenteronline.com/incidents/1t2xtfwnbdzn
http://status.planningcenteronline.com/incidents/1t2xtfwnbdznElevated Response Times
<p><small>Sep 6, 08:55 PDT</small><br><strong>Postmortem</strong> - A detailed report of the incident, along with the corrective measures taken, has been posted to our [blog](https://planning.center/2017/sunday-morning-reliability/).</p><p><small>Aug 27, 10:20 PDT</small><br><strong>Resolved</strong> - Systems are back to full functionality. A detailed report, along with the specific corrective measures we'll be taking, will be posted in the coming days.</p><p><small>Aug 27, 09:41 PDT</small><br><strong>Update</strong> - Lists in Planning Center People are still disabled as we keep an eye on our server loads.</p><p><small>Aug 27, 08:55 PDT</small><br><strong>Monitoring</strong> - Fixes have been implemented and systems are recovering. We're monitoring the situation closely.</p><p><small>Aug 27, 07:19 PDT</small><br><strong>Identified</strong> - We're experiencing increased response times and are currently looking into the issue.</p> Sun, 27 Aug 2017 10:20:39 -0700http://status.planningcenteronline.com/incidents/0lnvb2rqbs7x
http://status.planningcenteronline.com/incidents/0lnvb2rqbs7xOutage
<p><small>Aug 23, 19:05 PDT</small><br><strong>Resolved</strong> - The temporary outage has been resolved!</p> Wed, 23 Aug 2017 19:05:13 -0700http://status.planningcenteronline.com/incidents/p669hrkdxf1g
http://status.planningcenteronline.com/incidents/p669hrkdxf1gSlow Response Times
<p><small>Aug 23, 11:10 PDT</small><br><strong>Resolved</strong> - Response times have stabilized and People Lists have been re-enabled. We will continue to monitor, but at this point we are fully operational.</p><p><small>Aug 23, 09:31 PDT</small><br><strong>Monitoring</strong> - A change has been made and we are monitoring the results. Site response times are recovering.</p><p><small>Aug 23, 08:44 PDT</small><br><strong>Investigating</strong> - We currently experience site-wide slowness and are investigating.</p> Wed, 23 Aug 2017 11:10:47 -0700http://status.planningcenteronline.com/incidents/qqww4hkxfv78
http://status.planningcenteronline.com/incidents/qqww4hkxfv78Scheduled Database Maintenance
<p><small>Aug 22, 22:39 PDT</small><br><strong>Completed</strong> - The database maintenance is complete. We thank you for your patience.</p><p><small>Aug 22, 22:00 PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug 22, 10:16 PDT</small><br><strong>Scheduled</strong> - Planning Center will be unavailable on August 22 at 10pm Pacific Time while we do some database maintenance. We expect this to take an hour. All services will be unavailable.</p> Tue, 22 Aug 2017 22:39:37 -0700http://status.planningcenteronline.com/incidents/mv3sblldpt3w
http://status.planningcenteronline.com/incidents/mv3sblldpt3wElevated Response Times
<p><small>Aug 22, 07:56 PDT</small><br><strong>Postmortem</strong> - We sincerely apologize for the increased latency which occurred yesterday, August 20th, from 6:48AM PDT to 8:20AM PDT. We know you depend on us to do your work in your ministry, and we take that responsibility very seriously.
<br />A full summary of the day, including steps we plan on taking moving forward, are as follows.
<br />
<br />ISSUE SUMMARY
<br />
<br />From 6:48AM PDT to 8:20AM PDT, requests to all applications were met with slow response times and users had an especially difficult time using Check-ins. The root cause was an increase in load on our database server which in turn slowed all requests.
<br />
<br />TIMELINE
<br />
<br />6:48AM PDT We began receiving alerts of increased latency across several applications.
<br />
<br />7:11AM PDT We identified a change made to Check-Ins earlier this week which caused the increased load on the database.
<br />
<br />7:35AM PDT The change was reverted from Check-Ins and we began monitoring for recovery.
<br />
<br />8:20AM PDT Load began decreasing and response times improved.
<br />
<br />ROOT CAUSE
<br />
<br />A slow SQL query was deployed earlier in the week that triggered each time Check-Ins searches the list of People. The performance issue did not become apparent until increased traffic on Sunday morning.
<br />
<br />RESOLUTION AND RECOVERY
<br />
<br />At 8:20AM PDT load decreased and the issue was resolved.
<br />
<br />CORRECTIVE AND PREVENTATIVE MEASURES
<br />
<br />After a complete review and analysis of our performance yesterday, we will take the following steps to prevent this from happening again.
<br />
<br />We are reviewing critical queries in Check-Ins and optimizing them to ensure they can perform well during high traffic periods.
<br />We will evaluate our current infrastructure to ensure we are using the most appropriate hardware for our volume of requests.
<br />Planning Center is committed to continually improving our technology and operational processes to prevent outages.
<br />
<br />We appreciate your patience and again deeply apologize for the impact to you and your church.
<br />Thank you for your business and continued support.</p><p><small>Aug 20, 09:28 PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug 20, 08:41 PDT</small><br><strong>Investigating</strong> - We're experiencing increased response times and are currently looking into the issue.</p> Sun, 20 Aug 2017 09:28:00 -0700http://status.planningcenteronline.com/incidents/t21r59dy2jmd
http://status.planningcenteronline.com/incidents/t21r59dy2jmdSlow response times
<p><small>Aug 20, 07:41 PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug 20, 07:00 PDT</small><br><strong>Investigating</strong> - We're experiencing some slower than normal response times for some customers. We're investigating now.</p> Sun, 20 Aug 2017 07:41:05 -0700http://status.planningcenteronline.com/incidents/8dl748z8qphy
http://status.planningcenteronline.com/incidents/8dl748z8qphyPhone and email responses may be delayed
<p><small>Mar 15, 16:25 PDT</small><br><strong>Resolved</strong> - Our help desk is up and running! Our response time should be back to normal now. Thanks again for your patience today!</p><p><small>Mar 15, 13:01 PDT</small><br><strong>Identified</strong> - Our help desk software is currently down. All phone and email responses may be delayed, we will respond to all requests as soon as possible!</p> Wed, 15 Mar 2017 16:25:36 -0700http://status.planningcenteronline.com/incidents/qqp1vbfnyjjv
http://status.planningcenteronline.com/incidents/qqp1vbfnyjjvConnectivity Issues
<p><small>Mar 14, 19:55 PDT</small><br><strong>Resolved</strong> - Looks like Cox has fixed their connectivity issue.</p><p><small>Mar 14, 16:37 PDT</small><br><strong>Identified</strong> - California and Arizona residents with Cox may be unable to access to Planning Center. We will provide updates as soon as we have more information.</p> Tue, 14 Mar 2017 19:55:44 -0700http://status.planningcenteronline.com/incidents/4zmdm11ky6wx
http://status.planningcenteronline.com/incidents/4zmdm11ky6wxSlowness in All apps
<p><small>Mar 1, 17:24 PST</small><br><strong>Postmortem</strong> - We would like to sincerely apologize for the outage which occurred yesterday, February 28th, from 9:48AM PST to 3:30PM PST. We understand how much you depend on us to do your work in your ministry, and we take that responsibility very seriously.
<br />
<br />A full summary of the day, including steps we plan on taking, is follows.
<br />
<br />Thank you for letting us serve your church.
<br />
<br />####Issue Summary
<br />From 9:48AM PST to 3:30PM PST Requests to Service, Check-ins, Giving, People, Resources, Registrations, and Groups were met with slow response times and an inability to use many features on Services and People. The root cause was an inability to read files from, or write files to our hosting provider (Amazon Web Services).
<br />
<br />####Timeline
<br />9:48AM PST We began receiving alerts of increased errors across several applications.
<br />
<br />9:55AM PST We identified the source of the errors as our inability to interact with the file storage system.
<br />
<br />10:00AM PST We began disabling features that relied on file storage in an attempt to keep Services and People available.
<br />
<br />12:00PM PST Other AWS services began to fail as a result of the sheer size of the outage. In an attempt to reduce load on our background servers we issued a restart to some of them. The restart failed, and we were unable to replace those servers the way we normally would.
<br />
<br />12:10PM PST Faced with the loss of some very important servers, we made the decision to take the sites down in order to prevent damage to user data while we processed a large backup of queued tasks.
<br />
<br />1:15PM PST We were able to fix one of the background servers and re-purpose some other servers to help process background tasks.
<br />
<br />1:35PM PST We felt comfortable enough with the current server load to bring the sites back up with some features still disabled.
<br />
<br />3:30PM PST The underlying issues were resolved by our hosting provider and we were able to restore all degraded features to full functionality.
<br />
<br />####Root Cause
<br />We were unable to handle the volume of errors resulting from an inability to read or write files from AWS.
<br />
<br />#### Resolution and Recovery
<br />At 3:30 PST we were able to restore functionality only when the underlying issue was resolved.
<br />
<br />#### Corrective and Preventative Measures
<br />After a complete review and analysis of our performance yesterday, the following action will be taken to improve our resilience to failures in third party products we rely on.
<br />
<br />- We are doing a complete review of all third party integrations, and will take steps to more gracefully deal with changes in availability of service providers.
<br />
<br />- We will modify our internal architecture to better handle slow response times from different parts of our system.
<br />
<br />Planning Center is committed to continually improving our technology and operational processes to prevent outages. We appreciate your patience and again apologize for the impact to you and your church.
<br />
<br />Thank you for your business and continued support.</p><p><small>Feb 28, 15:41 PST</small><br><strong>Resolved</strong> - All services are restored to full functionality.</p><p><small>Feb 28, 13:55 PST</small><br><strong>Update</strong> - Though we're back up, files uploads, chord charts, printing plans, & activity feeds are still down.</p><p><small>Feb 28, 13:35 PST</small><br><strong>Update</strong> - Most of Planning Center is back up, though file uploads still won't work.</p><p><small>Feb 28, 12:10 PST</small><br><strong>Update</strong> - Unfortunately we're having to put the site into maintenance mode. We'll get it back as soon as possible.</p><p><small>Feb 28, 11:48 PST</small><br><strong>Update</strong> - Amazon recently posted:
<br />
<br />Increased Error Rates
<br />Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue.
<br />http://status.aws.amazon.com</p><p><small>Feb 28, 11:47 PST</small><br><strong>Update</strong> - Our current issues are tied to Amazon AWS. You can follow their status at http://status.aws.amazon.com</p><p><small>Feb 28, 10:27 PST</small><br><strong>Update</strong> - The Activity tab in PCO People has been disabled, which seems to speed other things up.</p><p><small>Feb 28, 10:25 PST</small><br><strong>Update</strong> - We've disabled the Activity tab in Planning Center People. This should help speed up other things.</p><p><small>Feb 28, 10:13 PST</small><br><strong>Update</strong> - Planning Center lyrics & chord PDFs are generated on Amazon S3. We've turned off generation of Lyric & Chord PDFs to try and speed up the rest of Services. However, the S3 outage will affect anything related to files. You won't be able to access any files until Amazon restores S3 service.</p><p><small>Feb 28, 10:06 PST</small><br><strong>Monitoring</strong> - Many websites are experiencing difficulties at the moment. Amazon S3 helps power a huge number of sites, not just Planning Center. We're trying to turn off some features in Planning Center Services that rely heavily on S3 to try and reduce the requests we send them. This will hopefully allow other processes to speed up.</p><p><small>Feb 28, 09:59 PST</small><br><strong>Identified</strong> - The current slowness seems to be caused by issues with Amazon's S3 file storage. We're trying to work around it.</p><p><small>Feb 28, 09:48 PST</small><br><strong>Investigating</strong> - We're experiencing some slowness in all Planning Center apps and are investigating the cause.</p> Tue, 28 Feb 2017 15:41:08 -0800http://status.planningcenteronline.com/incidents/j6glpb1g49kj
http://status.planningcenteronline.com/incidents/j6glpb1g49kjPush notifications interrupted
<p><small>Feb 26, 09:21 PST</small><br><strong>Resolved</strong> - The problem has been identified and addressed. Notifications are coming through again.</p><p><small>Feb 26, 08:27 PST</small><br><strong>Investigating</strong> - Most push notifications are either delayed or not getting sent. We're investigating now.</p> Sun, 26 Feb 2017 09:21:03 -0800http://status.planningcenteronline.com/incidents/69f6ggdz2pt6
http://status.planningcenteronline.com/incidents/69f6ggdz2pt6Outage
<p><small>Feb 23, 15:33 PST</small><br><strong>Resolved</strong> - Services is back online. Sorry for the outage!</p><p><small>Feb 23, 15:28 PST</small><br><strong>Investigating</strong> - Services is having connectivity issues. We're working on resolving it as quickly as possible.</p> Thu, 23 Feb 2017 15:33:58 -0800http://status.planningcenteronline.com/incidents/02s755dfns4f
http://status.planningcenteronline.com/incidents/02s755dfns4fScheduled Database Maintenance
<p><small>Feb 7, 23:06 PST</small><br><strong>Completed</strong> - Our database maintenance was completed successfully. Thank you for your patience.</p><p><small>Feb 7, 22:00 PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb 6, 15:51 PST</small><br><strong>Scheduled</strong> - Planning Center will be unavailable on February 7 at 10pm Pacific Time while we do some database maintenance. We expect this to take about an hour. All services will be unavailable.</p> Tue, 07 Feb 2017 23:06:32 -0800http://status.planningcenteronline.com/incidents/6z6y0njltzbl
http://status.planningcenteronline.com/incidents/6z6y0njltzblDelayed job processing
<p><small>Jan 6, 10:47 PST</small><br><strong>Resolved</strong> - All caught up now! Thanks for hanging in there with us.</p><p><small>Jan 6, 10:23 PST</small><br><strong>Monitoring</strong> - Some operations like sending email, and refreshing lists are backed up. The problem has been addressed and we're getting caught up now. We should be back to normal in 20 minutes or so.</p> Fri, 06 Jan 2017 10:47:14 -0800http://status.planningcenteronline.com/incidents/zn6sb8t5nwd6
http://status.planningcenteronline.com/incidents/zn6sb8t5nwd6Planning Center Registrations Urgent Database Maintenance
<p><small>Nov 19, 18:57 PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov 19, 18:19 PST</small><br><strong>Scheduled</strong> - We need to perform some urgent database maintenance for Planning Center Registrations, and will need to take the site offline for a few minutes. We regret the timing of this, we'll be back up just as quick as we can.</p> Sat, 19 Nov 2016 18:57:17 -0800http://status.planningcenteronline.com/incidents/gr39zr3vg34q
http://status.planningcenteronline.com/incidents/gr39zr3vg34qPayments failing in Registrations
<p><small>Nov 16, 07:13 PST</small><br><strong>Resolved</strong> - The issue with accepting payments has been fixed in Registrations.</p><p><small>Nov 16, 07:01 PST</small><br><strong>Identified</strong> - We are aware of an issue accepting payments in Registrations. We are working on a fix now.</p> Wed, 16 Nov 2016 07:13:09 -0800http://status.planningcenteronline.com/incidents/g0wkzqzdsc04
http://status.planningcenteronline.com/incidents/g0wkzqzdsc04Outage
<p><small>Nov 5, 07:28 PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov 5, 06:10 PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov 5, 05:47 PDT</small><br><strong>Update</strong> - Major services have been restored, double checking all services currently.</p><p><small>Nov 5, 05:17 PDT</small><br><strong>Identified</strong> - We have identified the source of the outage and are working for a quick resolution.</p><p><small>Nov 5, 04:40 PDT</small><br><strong>Investigating</strong> - Investigating reported outage.</p> Sat, 05 Nov 2016 07:28:12 -0700http://status.planningcenteronline.com/incidents/0pth3z9cbb1r
http://status.planningcenteronline.com/incidents/0pth3z9cbb1rDelayed processing in Services
<p><small>Oct 31, 14:39 PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct 31, 14:29 PDT</small><br><strong>Update</strong> - We're continuing to process the backlog of tasks in Services. We expect to be all caught up within the next 30 minutes.</p><p><small>Oct 31, 13:49 PDT</small><br><strong>Identified</strong> - We've identified an issue causing delayed processing of backend tasks in Services and are working to resolve.</p> Mon, 31 Oct 2016 14:39:54 -0700http://status.planningcenteronline.com/incidents/cqchkynbfnjj
http://status.planningcenteronline.com/incidents/cqchkynbfnjjElevated Error Rates
<p><small>Oct 18, 17:38 PDT</small><br><strong>Resolved</strong> - The issue should be resolved and back to normal. Thanks for your patience.</p><p><small>Oct 18, 17:14 PDT</small><br><strong>Identified</strong> - The issue is identified and we're working on a fix.</p><p><small>Oct 18, 17:08 PDT</small><br><strong>Investigating</strong> - Our systems have notified us of elevated error rates and our team is investigating.</p> Tue, 18 Oct 2016 17:38:24 -0700http://status.planningcenteronline.com/incidents/kwmlbc7jg0gl
http://status.planningcenteronline.com/incidents/kwmlbc7jg0glPhone and email responses may be delayed
<p><small>Oct 13, 14:02 PDT</small><br><strong>Resolved</strong> - Our help desk is back up! All phone and email responses are back to normal.</p><p><small>Oct 13, 11:14 PDT</small><br><strong>Identified</strong> - Our help desk software is currently down. All phone and email responses may be delayed, we will respond to all requests as soon as possible!</p> Thu, 13 Oct 2016 14:02:32 -0700http://status.planningcenteronline.com/incidents/1m2d9crl592s
http://status.planningcenteronline.com/incidents/1m2d9crl592sScheduled Database Maintenance
<p><small>Oct 3, 22:34 PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct 3, 22:01 PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct 3, 11:19 PDT</small><br><strong>Scheduled</strong> - We're upgrading our database servers again this evening. We're getting pretty good at this, expected downtime is roughly 30 minutes. All Planning Center applications will be unavailable during this time.</p> Mon, 03 Oct 2016 22:34:49 -0700http://status.planningcenteronline.com/incidents/1qvfkwbc5lbt
http://status.planningcenteronline.com/incidents/1qvfkwbc5lbtDelayed email delivery
<p><small>Sep 25, 14:43 PDT</small><br><strong>Resolved</strong> - Email delivery is back to normal now.</p><p><small>Sep 25, 13:53 PDT</small><br><strong>Identified</strong> - Our email provider is experiencing delays in delivering email.</p> Sun, 25 Sep 2016 14:43:27 -0700http://status.planningcenteronline.com/incidents/jbcqq2jksrx1
http://status.planningcenteronline.com/incidents/jbcqq2jksrx1Increased latency due to database load
<p><small>Sep 25, 08:33 PDT</small><br><strong>Resolved</strong> - This week we made a two line, seemingly benign code change that had unexpected performance implications when placed under the high load of Sunday morning traffic. We've reverted the changes and our server loads (specifically our MySQL servers) are back to their typical load levels, resulting in normal response times. Kudos to our development, operations, and support teams for quickly identifying the issue and implementing the fix.</p><p><small>Sep 25, 08:20 PDT</small><br><strong>Monitoring</strong> - The issue has been identified and a fix has been made. We're keeping an eye on things to ensure we're back to 100%.</p><p><small>Sep 25, 07:31 PDT</small><br><strong>Investigating</strong> - We are investigating a slow down in some apps due to increased load on our database.</p> Sun, 25 Sep 2016 08:33:16 -0700http://status.planningcenteronline.com/incidents/7q5cxgvplwcr
http://status.planningcenteronline.com/incidents/7q5cxgvplwcrScheduled Maintenance
<p><small>Sep 12, 22:30 PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep 12, 22:01 PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep 12, 12:13 PDT</small><br><strong>Scheduled</strong> - We will have downtime tonight for some maintenance beginning at 10pm PDT. Expected time frame is approximately 30 minutes. All apps will be inaccessible during this time.</p> Mon, 12 Sep 2016 22:30:46 -0700http://status.planningcenteronline.com/incidents/fmp381ccqyw4
http://status.planningcenteronline.com/incidents/fmp381ccqyw4