Pivotal Web Services Status - Incident Historyhttps://status.run.pivotal.io
StatuspageMon, 21 Jan 2019 14:31:17 -0800Current Bills unavailable on AppsManager
<p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>15:45</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>14:20</var> PST</small><br><strong>Identified</strong> - There is an unplanned maintenance window on the PWS Developer Console on Jan 16 from 2:00 PM Pacific Time. The expected duration is 2 hours. Deployed apps will continue to operate without impact and Cloud Foundry functions will continue to work via the Command Line Interface 'cf' tool. Current bills may be unavailable via the Developer Console during this maintenance period. If you have any questions, please contact support@run.pivotal.io.</p> Wed, 16 Jan 2019 15:45:03 -0800https://status.run.pivotal.io/incidents/g98p535h4cnj
https://status.run.pivotal.io/incidents/g98p535h4cnjAppsManager is unavailable
<p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>11:44</var> PST</small><br><strong>Resolved</strong> - AppsManager functionality has returned. The issue has been resolved. If you continue to have issues, please contact support@run.pivotal.io</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>11:00</var> PST</small><br><strong>Investigating</strong> - AppsManager is unavailable. Your app(s) performance should not be impacted. We are looking into it. If you need to manage your apps, please use the CF CLI. If you are experiencing broader issues, please contact support@run.pivotal.io</p> Wed, 19 Dec 2018 11:44:02 -0800https://status.run.pivotal.io/incidents/8m2c3kgnwjw5
https://status.run.pivotal.io/incidents/8m2c3kgnwjw5Intermittent issue provisioning ClearDB databases
<p><small>Jan <var data-var='date'> 2</var>, <var data-var='time'>09:52</var> PST</small><br><strong>Update</strong> - This issue is only affecting certain spaces. A workaround is to temporarily use a different space or to delete the space where the ClearDB service instance cannot be created and recreate the space.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>14:35</var> PST</small><br><strong>Identified</strong> - We've received several support tickets indicating and have seen in our logs that there is an issue provisioning ClearDB service instances. We have escalated the issue to the involved 3rd parties and will update this incident as we progress towards a resolution.</p> Wed, 02 Jan 2019 09:52:18 -0800https://status.run.pivotal.io/incidents/w4lzf45bp9nc
https://status.run.pivotal.io/incidents/w4lzf45bp9ncSSO errors while accessing management pages for marketplace services
<p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>04:42</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>15:32</var> PST</small><br><strong>Monitoring</strong> - We have rolled back the UAA update and the issues around single sign have been resolved.</p> Thu, 13 Dec 2018 04:42:02 -0800https://status.run.pivotal.io/incidents/h465lsqzvf4k
https://status.run.pivotal.io/incidents/h465lsqzvf4kScheduled Maintenance on the p-redis service on PWS
<p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>09:00</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>07:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>06:42</var> PST</small><br><strong>Scheduled</strong> - Redis instances will be unavailable for a brief period of time, estimated at 3-5 minutes, during the maintenance window.</p> Mon, 03 Dec 2018 09:00:25 -0800https://status.run.pivotal.io/incidents/nmtk65wqv9vh
https://status.run.pivotal.io/incidents/nmtk65wqv9vhAppsManager is unavailable
<p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>10:11</var> PST</small><br><strong>Resolved</strong> - AppsManager functionality has returned. The issue has been resolved. If you continue to have issues, please contact support@run.pivotal.io</p><p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>10:09</var> PST</small><br><strong>Investigating</strong> - AppsManager is unavailable. Your app(s) performance should not be impacted. We are looking into it. If you need to manage your apps, please use the CF CLI. If you are experiencing broader issues, please contact support@run.pivotal.io</p> Wed, 21 Nov 2018 10:11:59 -0800https://status.run.pivotal.io/incidents/bxvyg5731brb
https://status.run.pivotal.io/incidents/bxvyg5731brbBilling and Account Management temporarily unavailable
<p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>03:30</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>01:30</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>13:37</var> PST</small><br><strong>Scheduled</strong> - We will be making changes to the database for the usage service, which will make some billing and usage information unavailable</p> Wed, 07 Nov 2018 03:30:02 -0800https://status.run.pivotal.io/incidents/9m228pkwdnz9
https://status.run.pivotal.io/incidents/9m228pkwdnz9CF Push and App Availability outage
<p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>06:45</var> PST</small><br><strong>Resolved</strong> - We were alerted to application push failures at 2018-11-06 06:48 UTC</p> Tue, 06 Nov 2018 06:45:49 -0800https://status.run.pivotal.io/incidents/d0h1lxgw75yr
https://status.run.pivotal.io/incidents/d0h1lxgw75yrSSL Service: Unable to create service instances
<p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>17:04</var> PST</small><br><strong>Resolved</strong> - We discovered that a quota was reached for available SSL/TLS certificates. That issue has now been resolved and the SSL Service is restored to operational.</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>13:30</var> PST</small><br><strong>Identified</strong> - We have identified an issue affecting the SSL Service which is preventing new service instances from being created. Existing service instances are not affected.</p> Mon, 05 Nov 2018 17:04:27 -0800https://status.run.pivotal.io/incidents/z8flqq7bg6rp
https://status.run.pivotal.io/incidents/z8flqq7bg6rpSAML login error in Pivotal Partner and Support portals
<p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>01:45</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p> Sat, 03 Nov 2018 01:45:56 -0700https://status.run.pivotal.io/incidents/461j9ql1lxrb
https://status.run.pivotal.io/incidents/461j9ql1lxrbApplication Staging (cf push, etc.) Failures
<p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>14:37</var> PDT</small><br><strong>Resolved</strong> - An official stemcell containing a fix has been deployed to the platform since last week. We have not observed any issues and are closing this long-running incident.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>11:45</var> PDT</small><br><strong>Monitoring</strong> - The update to mitigate the Linux kernel regression has been deployed to the entire platform, as of a few days ago. Platform stability has improved. We'll continue to monitor until a permanent resolution is deployed.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>15:45</var> PDT</small><br><strong>Identified</strong> - We’ve identified a potential Linux kernel regression around networking performance. We initially suspected that the impact was limited to Docker applications. However, while our initial mitigation of separating Docker and Buildpack applications lowered the rate of failure, we continue to see failures downloading resources in the Buildpack specific infrastructure.<br /><br />We are rolling out an update that we believe mitigates the regression to a portion of the platform. We intend to monitor the update for a short time before applying it to the remainder of the platform.</p><p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>17:53</var> PDT</small><br><strong>Monitoring</strong> - Our mitigation has successfully controlled the error rate and only Docker apps should experience turbulence. We are monitoring the platform for further symptoms. A permanent resolution to Docker-based app issues will be forthcoming.</p><p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>08:39</var> PDT</small><br><strong>Update</strong> - We have finished applying a temporary mitigation and the platform appears to have stabilized as of 2018-10-07 02:00 UTC for buildpack-based applications. The root cause is still being investigated but is related to an OS update adversely affecting Docker-based applications. Docker-based applications have been moved to separate infrastructure as to not negatively impact buildpack-based applications so we expect Docker-based applications to continue to have intermittent issues until a fix can be put in place.</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>10:14</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and we are installing a temporary mitigation while we address the underlying cause.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>06:59</var> PDT</small><br><strong>Investigating</strong> - We've rolled out some fixes, but are continuing to see staging failures and are investigating.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>17:32</var> PDT</small><br><strong>Monitoring</strong> - The deploy to mitigate this issue has completed and we are seeing successful application pushes. We'll continue to monitor.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>16:19</var> PDT</small><br><strong>Identified</strong> - We believe we have identified the issue and are deploying a fix now.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>15:31</var> PDT</small><br><strong>Investigating</strong> - We are currently seeing failures when staging and or starting application (e.g., `cf push`) and are investigating.</p> Tue, 30 Oct 2018 14:37:38 -0700https://status.run.pivotal.io/incidents/lwn85syb3q2j
https://status.run.pivotal.io/incidents/lwn85syb3q2jScheduled Maintenance for PWS
<p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>14:14</var> PDT</small><br><strong>Resolved</strong> - The maintenance is complete and the Pivotal Accounts page is working as expected.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>12:31</var> PDT</small><br><strong>Investigating</strong> - Beginning 21:00 UTC today, we will be performing maintenance tasks on the Pivotal Accounts page (https://login.run.pivotal.io/). We expect the maintenance to complete in less than 5 minutes, and should not impact any services other than the Pivotal Accounts.</p> Wed, 24 Oct 2018 14:14:56 -0700https://status.run.pivotal.io/incidents/2pzlh3p0qry1
https://status.run.pivotal.io/incidents/2pzlh3p0qry1Intermittent Unknown HTTP Failures
<p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>08:08</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved. We believe the issues we experienced were due to a brief period of network connectivity issues within the AWS us-east-1 region.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>12:14</var> PDT</small><br><strong>Monitoring</strong> - We are still looking into the cause of this, but the API has stabilized.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>11:58</var> PDT</small><br><strong>Investigating</strong> - We are seeing unknown http errors with PWS api and are currently investigating.</p> Thu, 18 Oct 2018 08:08:30 -0700https://status.run.pivotal.io/incidents/9956w0vx2p2c
https://status.run.pivotal.io/incidents/9956w0vx2p2cApplication Staging Failures due to TLS Certificate Expiration
<p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>01:00</var> PDT</small><br><strong>Resolved</strong> - From approximately 2018-10-03 21:00 to 2018-10-04 00:30 UTC, application developers deploying to Pivotal Web Services (PWS) would have experienced failures in all attempts to push or scale applications. The underlying cause was that a set of TLS certificates used by platform-internal scheduling and placement components had expired, resulting in communication failures between those components. The monitoring PWS has in place to warn of upcoming certificate expiration missed the certificates in question; that monitoring has been updated.</p> Wed, 03 Oct 2018 01:00:37 -0700https://status.run.pivotal.io/incidents/b43j3c93ymp9
https://status.run.pivotal.io/incidents/b43j3c93ymp9Intermittent Application Staging (cf push, etc.) Failures
<p><small>Sep <var data-var='date'>25</var>, <var data-var='time'>08:55</var> PDT</small><br><strong>Resolved</strong> - For a period of 5 to 10 minutes on 2018-09-25, while performing a software upgrade on the platform, a database migration led to issues with internal networking. This would have impacted a subset of actions such as `cf push`. The issue was mitigated as the platform update proceeded.</p> Tue, 25 Sep 2018 08:55:02 -0700https://status.run.pivotal.io/incidents/z84yrh90bzjy
https://status.run.pivotal.io/incidents/z84yrh90bzjyApps Manager Scheduler Integration Services degradation
<p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>11:04</var> PDT</small><br><strong>Resolved</strong> - The Apps Manager app recently had a defect precluding the Scheduler integration from working. During this service degradation, the "Enable Scheduler" feature did not work. It was still possible to create instances of the Scheduler and bind them to apps through the general service creation and binding workflow in Apps Manager, and via the CF CLI. The issue is now resolved and steps are being taken to ensure that this issue does not happen in the future.</p> Fri, 21 Sep 2018 11:04:06 -0700https://status.run.pivotal.io/incidents/41jtc1df3ctg
https://status.run.pivotal.io/incidents/41jtc1df3ctgOccasional Failures During Application Start
<p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>10:51</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>09:18</var> PDT</small><br><strong>Identified</strong> - We have identified an issue affecting part of our application execution pool that is causing occasional failures during application start.</p> Fri, 21 Sep 2018 10:51:17 -0700https://status.run.pivotal.io/incidents/5m4g5q6pynnm
https://status.run.pivotal.io/incidents/5m4g5q6pynnmSpring Cloud Services degradation
<p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>16:00</var> PDT</small><br><strong>Resolved</strong> - The Spring Cloud Services (SCS) service broker had an outage yesterday from 4:00 PM PST (23:00 UTC) September 19th until approximately 3:00 AM PST (10:00 UTC) September 20th. During the outage, no SCS service instances could be create, updated, bound, unbound or deleted. Existing instances operated normally throughout this time. The issue is now resolved and steps are being taken to ensure that this issue does not happen in the future.</p> Wed, 19 Sep 2018 16:00:12 -0700https://status.run.pivotal.io/incidents/559d0j1s4jgm
https://status.run.pivotal.io/incidents/559d0j1s4jgmKnown issue: Pushing a Docker app with a long password fails
<p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>14:18</var> PDT</small><br><strong>Resolved</strong> - This issue has been resolved in a recent software update, and deployed to Pivotal Web Services.</p><p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>11:18</var> PDT</small><br><strong>Identified</strong> - We've discovered an issue pushing a Docker image based application to Pivotal Web Services when the container registry utilizes a long password (greater than 255 characters). The API team is working on a fix.</p> Thu, 06 Sep 2018 14:18:24 -0700https://status.run.pivotal.io/incidents/gtb4s3gb074d
https://status.run.pivotal.io/incidents/gtb4s3gb074dsome requests to API returns "status code: 500, error code: 10001, message: An unknown error occurred".
<p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>07:04</var> PDT</small><br><strong>Resolved</strong> - This incident is resolved.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>22:23</var> PDT</small><br><strong>Monitoring</strong> - We have applied a fix for the issue and are monitoring for any further errors.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>20:01</var> PDT</small><br><strong>Identified</strong> - We are fixing this problem</p> Thu, 30 Aug 2018 07:04:30 -0700https://status.run.pivotal.io/incidents/f56gn8fqxt0x
https://status.run.pivotal.io/incidents/f56gn8fqxt0xAppsManager is unavailable
<p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>09:44</var> PDT</small><br><strong>Resolved</strong> - AppsManager functionality has returned. The issue has been resolved. If you continue to have issues, please contact support@run.pivotal.io</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>09:44</var> PDT</small><br><strong>Investigating</strong> - AppsManager is unavailable. Your app(s) performance should not be impacted. We are looking into it. If you need to manage your apps, please use the CF CLI. If you are experiencing broader issues, please contact support@run.pivotal.io</p> Thu, 16 Aug 2018 09:44:32 -0700https://status.run.pivotal.io/incidents/qvt7mt5wgw7r
https://status.run.pivotal.io/incidents/qvt7mt5wgw7rElevated error rates on requests to UAA
<p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>08:12</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>08:12</var> PDT</small><br><strong>Update</strong> - Our monitoring indicates that the issue is resolved and the affected services are once again stable.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>12:45</var> PDT</small><br><strong>Monitoring</strong> - We have identified and corrected the cause of these errors. We are continuing to monitor to ensure that the issue has been resolved.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>08:26</var> PDT</small><br><strong>Investigating</strong> - We have observed an increase in errors on request to UAA. We are investigating to determine the cause and correct the issue.</p> Wed, 15 Aug 2018 08:12:35 -0700https://status.run.pivotal.io/incidents/2b8dty7vrvf6
https://status.run.pivotal.io/incidents/2b8dty7vrvf6Elevated error rates during application startup.
<p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>10:14</var> PDT</small><br><strong>Resolved</strong> - We haven't seen an occurrence of this timeout in several weeks. As always, we are continuously monitoring the platform for any irregularities. Please contact support@run.pivotal.io if you need assistance.</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>08:21</var> PDT</small><br><strong>Identified</strong> - We have observed further occurrences of slow droplet downloads during application startup. This manifests as applications taking longer than normal to start or timing out when attempting to push, start, or restart an application. We've observed that retrying will sometimes result in a successful application start. <br /><br />If retries are not working, affected users may be able to work around the issue by setting a local environment variable `CF_STARTUP_TIMEOUT=10` to increase the app start timeout to 10 minutes.</p> Mon, 06 Aug 2018 10:14:38 -0700https://status.run.pivotal.io/incidents/v114p5t6zhc0
https://status.run.pivotal.io/incidents/v114p5t6zhc0Service Broker Authentication Errors (~2 hours)
<p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>21:15</var> PDT</small><br><strong>Resolved</strong> - Beginning Thursday August 2nd 2018, 21:22 UTC through 23:00 UTC, a service broker on Pivotal Web Services (PWS) was misconfigured, resulting in errors provisioning, updating, deleting, and binding service instances managed by this broker.</p> Thu, 02 Aug 2018 21:15:55 -0700https://status.run.pivotal.io/incidents/lnt454c7qdvw
https://status.run.pivotal.io/incidents/lnt454c7qdvwBilling and Account Management still unavailable
<p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>17:46</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>11:38</var> PDT</small><br><strong>Investigating</strong> - We will be upgrading our billing and account management engine. The web pages for managing your account will be unavailable. Users will still be able to manage their apps through Apps Manager. Users will still be able to manage their apps through the API. Customer applications will continue to run, and will continue to be routable.<br /><br />We are extending this maintenance window</p> Mon, 23 Jul 2018 17:46:37 -0700https://status.run.pivotal.io/incidents/jt3qqw3jxnlm
https://status.run.pivotal.io/incidents/jt3qqw3jxnlm