tag:status.podio.com,2005:/historyPodio Status - Incident History2018-03-19T14:38:51-04:00Podiotag:status.podio.com,2005:Incident/16482162018-03-19T14:38:50-04:002018-03-19T14:38:51-04:00GlobiFlow experiencing slowness<p><small>Mar 19, 14:38 EDT</small><br><strong>Resolved</strong> - GlobiFlow is operating a normal speed at this point. We'll be scheduling additional maintenance in the future to continue to stabilize performance on GlobiFlow side.</p><p><small>Mar 19, 12:14 EDT</small><br><strong>Identified</strong> - We are working through general slowness on the GlobiFlow side and will keep customers updated here: http://status.globiflow.com/</p>tag:status.podio.com,2005:Incident/16107762018-02-20T18:08:39-05:002018-02-20T18:08:40-05:00File upload errors for some customers<p><small>Feb 20, 18:08 EST</small><br><strong>Resolved</strong> - We consider this issue resolved and will be updating our monitoring for this area of Podio to include additional metrics to ensure we can catch a similar issue in the future before occurring.</p><p><small>Feb 20, 17:39 EST</small><br><strong>Monitoring</strong> - We found the root cause for large file uploads and large app updates failing and will be monitoring the fix.</p><p><small>Feb 20, 17:18 EST</small><br><strong>Update</strong> - We are getting closer to a root cause for the file upload errors. Currently some customers are seeing large file upload failing and some specific apps are not able to save template edits. We'll update as we have more news to share.</p><p><small>Feb 20, 16:18 EST</small><br><strong>Investigating</strong> - We are investigating errors with files upload for some customers. Stay tuned.</p>tag:status.podio.com,2005:Incident/16103742018-02-20T13:56:46-05:002018-02-20T13:56:46-05:00We are seeing intermittent errors with GlobiFlow<p><small>Feb 20, 13:56 EST</small><br><strong>Resolved</strong> - GlobiFlow has worked through the queue and is back to normal operations: http://status.globiflow.com/incidents/jyydn8p93f33</p><p><small>Feb 20, 11:34 EST</small><br><strong>Monitoring</strong> - Please follow along here: http://status.globiflow.com/</p>tag:status.podio.com,2005:Incident/15861772018-02-01T01:22:44-05:002018-02-01T01:22:44-05:00Delays for workflows including calculations in apps<p><small>Feb 1, 01:22 EST</small><br><strong>Resolved</strong> - We are considering this incident solved. We are continuing to work to improve our architecture in order to decrease delays for web-hooks and calculations in the future.</p><p><small>Feb 1, 01:14 EST</small><br><strong>Monitoring</strong> - We have found the bottleneck causing delays for workflows specifically for apps including calculations. We are monitoring to ensure the delay is fully resolved.</p><p><small>Feb 1, 00:59 EST</small><br><strong>Identified</strong> - Team is investigating a delay with workflows and web-hooks specific for apps containing calculations.</p>tag:status.podio.com,2005:Incident/15766972018-01-29T13:18:33-05:002018-01-29T13:18:34-05:00Scheduled Maintenance - Item Logging Database Migration<p><small>Jan 29, 13:18 EST</small><br><strong>Completed</strong> - We have successfully enabled the feature item activity in production again after the successful last migration. Podio should now be fully functional.</p><p><small>Jan 28, 19:45 EST</small><br><strong>Verifying</strong> - The last maintenance has completed successfully over the weekend and we will be verifying the work done Monday in order to enable the item activity log again. You can expect the feature working fully again Monday Jan 29 around 12pm ET // 18.00 CET. Thanks for your patience.</p><p><small>Jan 25, 16:44 EST</small><br><strong>In progress</strong> - As an update on this maintenance, the team has initiated the process and it is expected to run over the next few days and through the weekend to completion. We will provide an update here, no later than Monday, confirming when the maintenance is complete and the item logging functionality is back online.</p><p><small>Jan 24, 16:58 EST</small><br><strong>Scheduled</strong> - The Podio operations team has kicked off maintenance that is necessary to resolve issues with how item changes are being logged to the activity log within the Podio application. As of right now, item activity is not being logged properly.
<br />Maintenance will run over the course of the next few days and will not impact the Podio service or it's availability. We will provide daily updates on progress as it is available.</p>tag:status.podio.com,2005:Incident/15745612018-01-24T16:53:49-05:002018-01-24T16:53:49-05:00Item update errors<p><small>Jan 24, 16:53 EST</small><br><strong>Resolved</strong> - At this point - we are confident that the adjustments made within the maintenance have been successful in bringing the Podio application back online and making it available for customers.
<br />We do still have an open task of performing maintenance to address an open issue where actions are not being logged in our activity log. We will be creating a separate maintenance item to track work on this.
<br />Again, our apologies for the inconvenience caused by this maintenance. We will provide full RCA on this issue as soon as it is available.</p><p><small>Jan 23, 11:41 EST</small><br><strong>Update</strong> - In an effort to resolve and immediately improve underlying issues that are leading to poor performance and behavior within the Podio application, our Development team will be performing maintenance on the application beginning at 12:30pm ET. While this maintenance is underway, the Podio application will be unavailable. We apologize for whatever inconveniences this may cause.</p><p><small>Jan 23, 10:48 EST</small><br><strong>Update</strong> - We are working to prepare a fix however this will take a bit longer to ensure we get things right for you. We apoligise any inconvenience this is causing your work-day.</p><p><small>Jan 23, 10:39 EST</small><br><strong>Identified</strong> - We have identified the main cause and will be implementing a fix to solve.</p><p><small>Jan 23, 10:23 EST</small><br><strong>Investigating</strong> - We are investigating an issue with item update errors. We will notify here as soon as we know more.</p>tag:status.podio.com,2005:Incident/15746662018-01-24T16:53:21-05:002018-01-26T22:20:14-05:001.23.2018 - Emergency Podio Maintenance - Application Will Be Unavailable<p><small>Jan 26, 22:17 EST</small><br><strong>Postmortem</strong> - Provided below is the full root-cause analysis of the events and circumstances that lead to the unavailability of the Podio service beginning at 12:30pm ET, January 23, 2018 to approximately 3:15pm ET on January 24th, 2018.
<br />
<br />**What Occurred?**
<br />
<br />At approximately 10:00am ET, on January 23rd, 2018 Podio operations noticed a flat line in our item creation metrics and began receiving reports of 500 errors as items were attempting to be created. Upon investigation, a large inflow of database exceptions in the `Item` table logs were observed. Each exception referenced an inability to create a new record in the database due to duplicate key constraint with value `2147483647`. The team recognized each number to be `2 ^ 31 - 1` which is the maximum value of a signed `int`. At this point, Podio is effectively in *read-only* mode.
<br />
<br />The Operations team determined the maximum value had been reached for the database cardinality limits for item revisions. The best mechanism to resolve this issue would be to adjust the mechanics that the platform used to write data to the specific tables that were having issues. The solution was to increase the size of the datatype for the column from `Int` to a `BigInt`, which would require maintenance to multiple database tables based on their relationships and mechanics. The Development team went on to prioritize the list of tables and features that were affected while the Operations team conducted testing on the changes to each column of one of the tables to get an estimate of the time it would take to run this statement with a hard cut-off time at 12:30pm ET. During this period, all backend jobs have been stopped to not process and fail in order to keep data safe while Podio.com was still serving requests. Customers were able to view items but not make changes to them.
<br />
<br />Our team made a risk assessment and decision based on the initial analysis and testing. Based on this, we know there were four impacted tables, and had to migrate at least the two impacted tables to the new mechanics. In order to most effectively do this without risk of any data loss, it was necessary to shut down the entirety of the Podio service. There were two primary reasons -
<br />
<br />1. The team recognized the item feature to be critical. If Podio had been kept online even in a read-only mode, requests would be hitting the database, and the database would have taken a harder hit with the migration running in parallel. Because of the size of the tables in questions, development and operations decided that it would be best to eliminate user traffic to the database to reduce the risk of resource competition and hopefully to help the table updates complete more quickly.
<br />
<br />2. Additionally, there were relational concerns of customers using the site and adding data to parts of the database while some tables could not receive writes, increasing the risk of relational errors in the data after the issue had been resolved.
<br />
<br />Upon making this decision, the team implemented the procedure in a test environment before making the final call to bring the service offline for the emergency migration.
<br />
<br />While the process and migration completed, our Operations team ran through the appropriate plan to bring the application back up upon completion. Further, we took the opportunity while these updates were running to apply maintenance and security patches that we had in our pipeline. We also made the decision to run the migration for the last two tables after getting the application online, with the intention to manually remove the feature, the item activity log, related to the problematic tables in order to enable a safe migration and ensure data integrity throughout the process.
<br />
<br />The Podio database is enormous with table sizes containing more than 500GB of data per table, this means changes were progressing at a slow but steady rate, as we were waiting for the database changes to be completed.
<br />
<br />At approximately 14:44pm ET, the following day, January 24th, 2018, the second migration completed. At this time the operations team ran through several procedures to ensure the Podio application was successfully able to read and write with the new mechanics. At approximately 15:15pm ET the application was once again widely available. The job workers were started and the traffic started flowing as soon as we went online.
<br />
<br />During initial testing around 15:17pm ET it was found that the file upload feature was not working as expected and was failing to upload files. Another deploy was done at 15:30pm ET in order to restart all possible services which could have been left in a stale state, but it did not solve the problem. The team started investigating more into the code and logs from across the different services and it was observed that services handling file requests were not running in majority of the servers. At around 16:15pm ET a force restart of service was applied across all the servers handling file services, and it resolved the issue.
<br />
<br />As expected, the impact of the new mechanism being used on the migrated tables, left some operations in a somewhat degraded state. This included the activity log of item adjustments. Additional maintenance was scheduled immediately to migrate the appropriate tables to the new mechanics to alleviate these degraded features as well. We expect the item revision activity log functionality to be back in service after the last two tables have completed migration with the application in online state during the period of Jan 24 to Jan 28.
<br />
<br />At this point, Podio is now accessible to our customers with approximately 99% of all features enabled and available for all customers. All the tasks in our queues have been processed with no loss of data.
<br />
<br />Overall, we do want to apologize to our customers for the huge impact this event had for you, your team and your business. We know you depend on us for a reliable and available service at all times and it is our highest priority to live up to your expectations and improve our availability in the future. We will conduct additional in-depth analysis in the following week to ensure we learn from this event.
<br />
<br />In order to improve, we are evaluating our risks and policies and will implement new processes the following days and weeks that will prevent a similar incident in the future. We are still pursuing the completion of the immediate actions required to restore service fully and to complete the analysis needed to be the reliable platform you need for your team and business long-term.</p><p><small>Jan 24, 16:53 EST</small><br><strong>Completed</strong> - At this point - we are confident that the adjustments made within the maintenance have been successful in bringing the Podio application back online and making it available for customers.
<br />We do still have an open task of performing maintenance to address an open issue where actions are not being logged in our activity log. We will be creating a separate maintenance item to track work on this.
<br />Again, our apologies for the inconvenience caused by this maintenance. We will provide full RCA on this issue as soon as it is available.</p><p><small>Jan 24, 15:22 EST</small><br><strong>Verifying</strong> - At this time, the Podio application is once again available for use. Please be gentle while we get things back up and running. We are continuing to monitor the service and the adjustments made to ensure the appropriate systems are functioning properly.
<br />
<br />As stated previously, there will be some features such as items writing to the item activity log, that may not work properly. We will continue to keep users apprised as we work to ensure ALL Podio functionality is back online and working properly.</p><p><small>Jan 24, 14:50 EST</small><br><strong>Update</strong> - Our table migrations have fully migrated and we are making the appropriate moves to bring the service back online. We are anticipating the service to be available shortly. As soon as we have a full update around the service becoming available we will post it here.</p><p><small>Jan 24, 13:50 EST</small><br><strong>Update</strong> - As of 1:45pm ET, both tables that we have been migrating are at 100% and the migration is finalizing. Once the migration has fully finalized, our Ops team will begin working to make the site available. We should be able to provide a firm ETA on availability once the migration has finalized. We will update this site as soon as the information is available.</p><p><small>Jan 24, 11:01 EST</small><br><strong>Update</strong> - As of right now, we are at 100% complete for one migration and close to 90% completion for the other. This information is assisting us in gauging a time of completion. At this point we are anticipating the maintenance to complete between 1:00pm-2:00pm ET.
<br />Upon completion of the maintenance, we anticipate the service becoming available once more with some features being temporarily unavailable. One of those anticipated features will be item revision records being logged in the item activity log.
<br />
<br />We will continue to provide updates as they are available, anticipating the next one around 1:30pm ET.</p><p><small>Jan 24, 06:39 EST</small><br><strong>Update</strong> - Update: In order to overcome a capacity problem in our main database, we are currently running a migration to expand the limit for 2 primary tables. The migration is currently slowing down compared to plan. As of right now, we are at 100% complete for one migration and slowly reaching 70% completion for another. This information is assisting us in gauging a time of completion. We do anticipate this will take a few extra hours longer than anticipated due the migration slowdown.
<br />
<br />The current estimate for completion will be around 12:00pm Eastern Time / 5 PM GMT / 18.00 CET.</p><p><small>Jan 24, 04:56 EST</small><br><strong>Update</strong> - Update: In order to overcome a capacity problem in our main database, we are currently running a migration to expand the limit for 2 primary tables. The migration is progressing slowly, but according to plan. As of this time we are at 100% complete for one migration and at 60% completion for another, which is assisting us in estimating an approximate time to resolution.</p><p><small>Jan 24, 03:34 EST</small><br><strong>Update</strong> - To offer a bit more background about the work being completed at this point: In order to overcome a capacity problem in our main database, we are currently running a migration to expand the limit for 2 primary tables. The migration is progressing slowly, but according to plan. Our current estimate is that we are at 90% completion for one migration and 50% completion for another, which is allowing us to estimate approximate time to full resolution. It is important to note, that there is absolutely no compromise to data integrity throughout this process.
<br />
<br />Additionally, our team completely understands the critical importance to our customers of Podio being reliable and available at all times. We are taking every measure possible to ensure we do not see a similar situation in the future. When the incident is resolved, we will provide a full and detailed overview of the main problem, implemented solution and the long term measures we are making to ensure this issue does not reoccur. The team is working as diligently as possible to bring Podio back online. We appreciate your patience during this process.</p><p><small>Jan 24, 03:13 EST</small><br><strong>Update</strong> - Our operations team continues to work on the necessary procedures, and at this time we are continuing to estimate that the maintenance should end this morning (9 am Eastern Time / 2:00 PM GMT / 3:00 PM CET), pending the outcome of the activities currently underway.
<br />We will provide another update at that time, or sooner should further information become available.</p><p><small>Jan 23, 21:15 EST</small><br><strong>Update</strong> - Our operations team continues to work on the necessary procedures, and at this time we are continuing to estimate that the maintenance should end tomorrow morning (Eastern Time), pending the outcome of the activities currently underway.
<br />We will provide another update at that time, or sooner should further information become available.</p><p><small>Jan 23, 16:39 EST</small><br><strong>Update</strong> - Our operations team continues to perform the necessary maintenance to resolve the issues with the Podio application and bring it back online.
<br />At this point in time, we are anticipating the maintenance to continue being performed overnight and up through tomorrow Jan 24 (until 9 am Eastern Time - 2:00 PM Wed, Greenwich Mean Time (GMT)). Should that timeline change, we will immediately update this status page to reflect the change.
<br />It is important to note, that there is absolutely no compromise to data integrity throughout this process. The operations team is working to resolve capacity issues that are impacting functionality. We will provide further updates as soon as they are available and we apologize for the inconvenience caused.</p><p><small>Jan 23, 14:30 EST</small><br><strong>Update</strong> - Maintenance is ongoing and we continue to make the appropriate changes in hopes of bringing the Podio application back online as soon as possible. At this time we do not have an ETA for full resolution and when the app should be available again. We will be sure to provide more information via this page, as soon as it is available.</p><p><small>Jan 23, 12:30 EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan 23, 11:35 EST</small><br><strong>Scheduled</strong> - In an effort to resolve and immediately improve underlying issues that are leading to poor performance and behavior within the Podio application, our Development team will be performing maintenance on the application beginning at 12:30pm ET.
<br />While this maintenance is underway, the Podio application will be unavailable. We apologize for whatever inconveniences this may cause.
<br />
<br />At this time, we are unsure of the exact amount time needed to make the appropriate adjustments, however we will update our customers whenever possible to provide as much information as often as we can. The date listed here is simply a placeholder until we have an ETA.</p>tag:status.podio.com,2005:Incident/15672922018-01-16T17:38:42-05:002018-01-16T17:38:42-05:00Slow background tasks causing delays for calculations, exports and workflows<p><small>Jan 16, 17:38 EST</small><br><strong>Resolved</strong> - We are back to normal service levels and consider this incident closed.</p><p><small>Jan 16, 16:40 EST</small><br><strong>Update</strong> - We are still processing the backlog of tasks from earlier today, we currently see 15 min delay and expect to be back to normal within the next 30 min.</p><p><small>Jan 16, 13:36 EST</small><br><strong>Monitoring</strong> - We expect the queue delay for calculations, exports and imports to be back to normal within the next hour.</p><p><small>Jan 16, 13:28 EST</small><br><strong>Identified</strong> - We deployed a fix and see good results from the initial logs. We expect the queue delay for calculations, exports and imports to be back to normal within the next hour.</p><p><small>Jan 16, 13:11 EST</small><br><strong>Update</strong> - We are working on a fix for the delay in the queue and will update as soon as we identified the issue. The queue delay is currently causing presence to fail and delay notifications (users will appear offline when online).</p><p><small>Jan 16, 11:17 EST</small><br><strong>Investigating</strong> - We are aware of the slow background tasks causing delays for calculations, exports and workflows. We are working on a solution and will keep you updated.</p>tag:status.podio.com,2005:Incident/15598062018-01-09T10:28:55-05:002018-01-09T10:28:56-05:00Slow response time in Web and API<p><small>Jan 9, 10:28 EST</small><br><strong>Resolved</strong> - We are back to normal service levels and will continue to monitor the fix. We have a clear understanding of the issue and will work on a longterm solution.</p><p><small>Jan 9, 10:15 EST</small><br><strong>Monitoring</strong> - We are aware of slow response times for web and API affecting GlobiFlow and causing errors. We have a solution and will be updating shortly.</p>tag:status.podio.com,2005:Incident/15582062018-01-08T14:52:06-05:002018-01-09T04:10:35-05:00Delay in search indexing<p><small>Jan 8, 14:52 EST</small><br><strong>Resolved</strong> - Our team has been monitoring the search service and the queue for rendering items ready for search. At this point, we are moving this issue to 'Resolved' as we believe that items are indexing at a normal rate and the service is fully available. Should any other strange behavior be noticed, please do not hesitate to contact Podio support.</p><p><small>Jan 8, 13:27 EST</small><br><strong>Monitoring</strong> - At this time, we believe the adjustments made by our engineers should have resolved issues with our search functionality being available. We are still monitoring the service as it indexes items and makes them available for searching on accounts. We expect this indexing process to complete within the next several hours. We will continue to keep this page updated once we have further information and believe the issue to be fully resolved.</p><p><small>Jan 8, 12:19 EST</small><br><strong>Update</strong> - Our engineers have been continuing to work and address the issues with our search functionality. At this time, we are seeing the search service work appropriately. We are continuing to work and perform the appropriate actions to ensure all items are indexed appropriately and appear in search results where appropriate. We will update this status as further information is available.</p><p><small>Jan 8, 10:41 EST</small><br><strong>Update</strong> - Our development team continues to work to mitigate the problems impacting our Search functionality and it's availability. We will provide updates as often as possible as we work to restore the functionality to appropriate service levels.</p><p><small>Jan 8, 08:01 EST</small><br><strong>Identified</strong> - We have identified the issue causing delays for Search and are working on a solution to decrease latency.</p><p><small>Jan 8, 07:30 EST</small><br><strong>Monitoring</strong> - We see a slight delay in search indexing and will be monitoring closely.</p>tag:status.podio.com,2005:Incident/14478242017-11-03T22:00:18-04:002017-11-03T22:00:18-04:00Scheduled Maintenance – November 3, 2017; 9:00pm EDT(GMT-4) – November 3, 2017; 10:00pm EDT(GMT-4)<p><small>Nov 3, 22:00 EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov 3, 21:00 EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct 27, 15:03 EDT</small><br><strong>Scheduled</strong> - Podio will be conducting maintenance on the evening of Friday, November 3rd starting at 9pm EDT (GMT-4). Maintenance should last no longer than 1 hour and end at 10:00pm EDT (GMT-4).
<br />
<br />During this maintenance, some customers may experience increased latency within the application. Not all customers will be impacted. We apologize for any inconvenience this may cause. This site will be updated once maintenance is complete.
<br />
<br />Provided is the start time for other major time zones globally -
<br />
<br />Los Angeles (US)
<br />6:00pm PDT , Friday, November 3rd
<br />London (UK)
<br />2:00am BST, Saturday, November 4th
<br />Dubai (UAE)
<br />5:00am GST, Saturday, November 4th
<br />Sydney (AU)
<br />12:00pm AEST, Saturday, November 4th</p>tag:status.podio.com,2005:Incident/14592532017-11-03T14:56:08-04:002017-11-03T14:56:09-04:00Delays in Calculations and Exports<p><small>Nov 3, 14:56 EDT</small><br><strong>Resolved</strong> - We are considering this issue resolved as we have seen significant improvements and are no longer seeing delays in calculations and exports. We apologize for any inconveniences caused with this issue.</p><p><small>Nov 3, 13:52 EDT</small><br><strong>Monitoring</strong> - We continue to see improvements that were causing delays in calculations and exports. We are continuing to monitor and will update once we have identified the issue is resolved.</p><p><small>Nov 3, 12:15 EDT</small><br><strong>Investigating</strong> - We are currently investigating a delay in calculations and exports, as these requests are taking longer than normal to process. We will provide updates once we have identified the cause. We apologize for any inconveniences this may cause our customers.</p>tag:status.podio.com,2005:Incident/14539152017-11-01T07:27:58-04:002017-11-01T07:27:59-04:00Increased response times for the API<p><small>Nov 1, 07:27 EDT</small><br><strong>Resolved</strong> - We consider this incident solved. Performance has been back to normal the last hour.</p><p><small>Nov 1, 06:37 EDT</small><br><strong>Update</strong> - We are fully recovered and back to normal performance.</p><p><small>Nov 1, 06:32 EDT</small><br><strong>Monitoring</strong> - We resolved the issue and will continue to monitor. We will also conduct additional analysis to understand how to present the specific issue in the future.</p><p><small>Nov 1, 06:23 EDT</small><br><strong>Identified</strong> - We believe we identified the issue and working on a solution</p><p><small>Nov 1, 05:58 EDT</small><br><strong>Investigating</strong> - Team is investigating the delay in API response time</p>tag:status.podio.com,2005:Incident/14459962017-10-26T11:44:11-04:002017-10-26T11:44:11-04:00Investigating Delayed Webhooks<p><small>Oct 26, 11:44 EDT</small><br><strong>Resolved</strong> - We have identified the issue causing delays in Webhooks and considering this resolved. We will continue to monitor and will update should we identify any other issues.</p><p><small>Oct 26, 11:41 EDT</small><br><strong>Monitoring</strong> - We have identified the issue and queue is decreasing. We will monitor until delays are fully resolved.</p><p><small>Oct 26, 11:22 EDT</small><br><strong>Investigating</strong> - We are currently investigating a delay in Webhooks. We will continue to update as updates are available.</p>tag:status.podio.com,2005:Incident/13937232017-10-03T16:29:41-04:002017-10-11T07:22:29-04:00Slow response times and partial outage<p><small>Oct 6, 03:55 EDT</small><br><strong>Postmortem</strong> - We have identified the incident with slow response times on Tuesday as a capacity problem and applied fixes Tuesday Oct 03 to resolve. We have been monitoring closely the 7 days and can confirm the fix is valid today Oct 11. Not only did our capacity addition help performance but we made changes to allow for more capacity to be added fairly easily in the future.</p><p><small>Oct 3, 16:29 EDT</small><br><strong>Resolved</strong> - We are seeing improvements in the issues that were causing slow response time which was limiting access to Podio. We will continue to monitor through the evening and early morning hours and will provide updates as needed, but we are considering this issue to be resolved at this time. We apologize for any inconveniences this may have caused.</p><p><small>Oct 3, 15:44 EDT</small><br><strong>Monitoring</strong> - We continue to see improvements in response time. We are continuing to monitor this issue and will provide updates as they are available.</p><p><small>Oct 3, 13:41 EDT</small><br><strong>Update</strong> - We are still investigating slow response times. We will continue to update as we have more details on the root cause.</p><p><small>Oct 3, 10:52 EDT</small><br><strong>Update</strong> - We are re-connecting Podio to hopefully fix the issues with slow response times.</p><p><small>Oct 3, 10:34 EDT</small><br><strong>Update</strong> - We are still investigating the root cause and apoligise for the inconvenience. We know you want to get back to work.</p><p><small>Oct 3, 08:53 EDT</small><br><strong>Investigating</strong> - We are continuing to see slow response times and are investigating.</p>tag:status.podio.com,2005:Incident/13934592017-10-03T06:10:00-04:002017-10-03T08:28:39-04:00Podio Service Running Slowly<p><small>Oct 3, 06:10 EDT</small><br><strong>Resolved</strong> - Our response times are back to normal and has been the last 1.5 hour. We are monitoring and will update as we know more.</p><p><small>Oct 3, 05:03 EDT</small><br><strong>Monitoring</strong> - Our response times are back to normal levels and has been the last 30 min. We are still monitoring and will update when the root cause has been identified and solved.</p><p><small>Oct 3, 04:04 EDT</small><br><strong>Investigating</strong> - We are aware of elevated response times again today and are working to resolve the issue. We're investigating the cause and will post updates here as soon as we have them.</p>tag:status.podio.com,2005:Incident/13924622017-10-02T16:42:13-04:002017-10-02T16:42:13-04:00Podio Service Running Slowly<p><small>Oct 2, 16:42 EDT</small><br><strong>Resolved</strong> - Response times are back to normal levels.</p><p><small>Oct 2, 13:04 EDT</small><br><strong>Investigating</strong> - We are aware of elevated response times and are working to resolve the issue. Updates to follow.</p>tag:status.podio.com,2005:Incident/13818752017-09-28T22:35:37-04:002017-09-28T22:35:38-04:00Emergency Maintenance - September 28, 9pm - 11pm ET<p><small>Sep 28, 22:35 EDT</small><br><strong>Completed</strong> - We have completed our scheduled maintenance, without any incidents.</p><p><small>Sep 28, 21:00 EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep 26, 16:05 EDT</small><br><strong>Scheduled</strong> - The Podio application will undergo emergency maintenance on September 28th from 9:00pm ET - 11:00pm ET.
<br />
<br />During this time, we anticipate approximately 30 minutes of downtime. We will provide updates as necessary throughout this maintenance period.</p>tag:status.podio.com,2005:Incident/13838372017-09-27T13:29:44-04:002017-09-27T13:29:45-04:00Podio Service Running Slowly<p><small>Sep 27, 13:29 EDT</small><br><strong>Resolved</strong> - Response times are back to normal levels. The scheduled maintenance tomorrow should help to alleviate the recurring service degradation. Thank you for your patience.</p><p><small>Sep 27, 11:11 EDT</small><br><strong>Investigating</strong> - We are aware of elevated response times and are working to resolve the issue. Updates to follow.</p>tag:status.podio.com,2005:Incident/13811822017-09-26T13:15:07-04:002017-09-26T13:15:08-04:00Podio Service Running Slowly<p><small>Sep 26, 13:15 EDT</small><br><strong>Resolved</strong> - Response times are back to normal levels. We are aware that this is a recurring issue and will be performing several database upgrades over the coming weeks to improve performance.</p><p><small>Sep 26, 10:23 EDT</small><br><strong>Investigating</strong> - We've received reports of the Podio service running slowly. Our engineering team is aware and we will provide updates as they become available.</p>tag:status.podio.com,2005:Incident/13713302017-09-21T14:11:13-04:002017-09-21T14:11:14-04:00Podio.com running slowly<p><small>Sep 21, 14:11 EDT</small><br><strong>Resolved</strong> - All services operational and the queues are back down to normal levels. More details on the incident to follow.</p><p><small>Sep 21, 12:50 EDT</small><br><strong>Investigating</strong> - Podio.com is running slowly - we are investigating the cause of the delays.
<br />
<br />Thank you for your continued patience while we look into this.</p>tag:status.podio.com,2005:Incident/13665372017-09-19T13:11:56-04:002017-09-19T15:55:10-04:00Latency in calculations and triggered events<p><small>Sep 19, 15:55 EDT</small><br><strong>Postmortem</strong> - On 9/19/17, reports of elevated wait times in Globiflow-based processes were reported to Podio Ops at 9:55am EDT, and monitoring tools recorded API errors. As task load increased unexpectedly on the affected instance, subsequent processes became queued. After a manual throttle of the affected tasks was unsuccessful, the API server was restarted at 10:59am, leading to an almost immediate reduction in queue time and overall performance, effectively resolving this incident.
<br />
<br />Further investigation revealed that the impacted instance had been scheduled for maintenance earlier today, however a discrepancy in the backing instance configuration resulted in the instance remaining deployed, and contributed to the inability of the system to gracefully failover as intended.
<br />
<br />Our Operations team has made a number of recommendations for long-term remediation, particularly as it applies to the backing services configuration, and the failover mechanics for instances that are no longer deployed.</p><p><small>Sep 19, 13:11 EDT</small><br><strong>Resolved</strong> - All services operational.</p><p><small>Sep 19, 11:42 EDT</small><br><strong>Update</strong> - An unexpected increase in workload activity created an operation backlog; we have taken steps to block some of that activity, which has mitigated the latency issue.
<br />
<br />Separately, we are investigating occurrences of the front end web application being intermittently unavailable, and will continue to post updates here as that work progresses.
<br />
<br />Again, thank you for your patience.</p><p><small>Sep 19, 10:41 EDT</small><br><strong>Investigating</strong> - Our operations and development teams are aware of latency issues in both Podio and Globiflow, resulting in delays and/or timeouts when events/flows are triggered, as well as in populating calculated fields.
<br />
<br />Our team has identified the potential root cause, and is working to make changes on the backend to remedy the latency.
<br />
<br />We apologize for any inconvenience this may have caused, and will update this page as information is available.</p>tag:status.podio.com,2005:Incident/13489832017-09-06T17:11:07-04:002017-09-06T17:11:08-04:00Podio.com intermittently unavailable<p><small>Sep 6, 17:11 EDT</small><br><strong>Resolved</strong> - All services operational.</p><p><small>Sep 6, 13:18 EDT</small><br><strong>Monitoring</strong> - We have identified the issue causing intermittent availability and the site is now functioning normally. We are closely monitoring to prevent further disruption.</p><p><small>Sep 6, 12:58 EDT</small><br><strong>Investigating</strong> - We are seeing intermittent issues accessing podio.com and are working to identify the root cause. We apologize for any inconvenience this issue may have caused, and will provide additional information as it's available.</p>tag:status.podio.com,2005:Incident/13042702017-07-21T12:51:11-04:002017-07-21T12:51:12-04:00Performance issues<p><small>Jul 21, 12:51 EDT</small><br><strong>Resolved</strong> - Issue has been resolved.</p><p><small>Jul 21, 12:02 EDT</small><br><strong>Monitoring</strong> - Performance issues have been identified and corrective measures in place. Performance has been steadily improving and we are continuing to monitor</p><p><small>Jul 21, 09:51 EDT</small><br><strong>Investigating</strong> - We are currently investigating a performance issue that is increasing delay times in calculations. Updates to follow</p>tag:status.podio.com,2005:Incident/12484442017-05-22T06:32:01-04:002017-05-22T06:32:02-04:00help.podio.com is currently down<p><small>May 22, 06:32 EDT</small><br><strong>Resolved</strong> - help.podio.com is back again! Our provider is still experiencing a service degradation and service might not be 100 % stable until they are fully recovered. We are sorry for the inconvenience.</p><p><small>May 22, 06:26 EDT</small><br><strong>Monitoring</strong> - Our knowledge base and support channel provider is experiencing issues and the help pages are down. Our provider is aware of the problem and working on a fix. You can continue to write us via support@podio.com and we'll get back to you as soon as possible.</p>