tag:status.cloudsight.ai,2005:/historyCloudSight API Status - Incident History2018-02-21T18:16:17-08:00CloudSight APItag:status.cloudsight.ai,2005:Incident/15891662018-02-04T07:14:34-08:002018-02-04T07:14:34-08:00Randomly Blocked Transactions on AWS RDS<p><small>Feb 4, 07:14 PST</small><br><strong>Resolved</strong> - We are still working with AWS on our database having blocked transactions, but this issue has not reoccurred in the past 16 hours. We will actively continue to monitor and engage AWS as necessary.</p><p><small>Feb 3, 14:52 PST</small><br><strong>Monitoring</strong> - We have detected an issue in AWS Relational Database Service which is causing blocked transactions in our database, resulting in intermittent service. We have opened a case with AWS regarding this issue and are waiting for a response.</p>tag:status.cloudsight.ai,2005:Incident/15690972018-01-17T18:18:45-08:002018-01-17T18:18:45-08:00Latency Issues at AWS<p><small>Jan 17, 18:18 PST</small><br><strong>Resolved</strong> - After working with Amazon, this issue appears to be resolved. We will actively monitor this situation and escalate again if necessary.</p><p><small>Jan 17, 16:22 PST</small><br><strong>Identified</strong> - We are actively working with Amazon to help them identify and fix this issue.</p>tag:status.cloudsight.ai,2005:Incident/14433322017-10-25T07:49:20-07:002017-10-25T07:49:20-07:00Upstream issue with AWS ELB<p><small>Oct 25, 07:49 PDT</small><br><strong>Resolved</strong> - The upstream AWS ELB issue has been resolved by Amazon and the ELB has been stable for ~15 minutes. We are continuing to monitor.</p><p><small>Oct 25, 07:30 PDT</small><br><strong>Monitoring</strong> - The upstream issue with AWS ELB appears to have been resolved by Amazon. We are continuing to monitor.</p><p><small>Oct 25, 07:19 PDT</small><br><strong>Update</strong> - We are currently investigating this issue.</p><p><small>Oct 25, 07:19 PDT</small><br><strong>Investigating</strong> - We are seeing high latency times with our AWS load balancer for api.cloudsightapi.com. We are working with AWS to resolve this issue as quickly as possible.</p>tag:status.cloudsight.ai,2005:Incident/13673902017-09-21T21:14:53-07:002017-09-21T21:14:53-07:00CloudSight NoSQL Database Maintenance<p><small>Sep 21, 21:14 PDT</small><br><strong>Completed</strong> - Our maintenance is complete. All systems are functioning as expected and we are continuing to monitor.</p><p><small>Sep 21, 20:48 PDT</small><br><strong>Verifying</strong> - The maintenance is completed and the APIs are restarting. We are monitoring.</p><p><small>Sep 21, 20:00 PDT</small><br><strong>In progress</strong> - Our scheduled maintenance has begun.</p><p><small>Sep 20, 20:05 PDT</small><br><strong>Update</strong> - The scheduled maintenance for this Friday is moving from 2:00 a.m. UTC to 3:00 a.m. UTC. During this time the CloudSight API will not be available, and CamFind and TapTapSee will not be functional. We anticipate this maintenance to take no longer than 45-60 minutes.</p><p><small>Sep 19, 15:29 PDT</small><br><strong>Scheduled</strong> - CloudSight's primary NoSQL database will be undergoing maintenance this Friday starting at 2:00 a.m. UTC. During this time the CloudSight API will not be available, and CamFind and TapTapSee will not be functional. We anticipate this maintenance to take no longer than 45-60 minutes.</p>tag:status.cloudsight.ai,2005:Incident/13600722017-09-14T13:07:13-07:002017-09-14T13:07:37-07:00Request Rate issues with AWS S3<p><small>Sep 14, 13:07 PDT</small><br><strong>Resolved</strong> - AWS S3 has been stable for the past 30 minutes. We are continuing to monitor this issue.</p><p><small>Sep 14, 12:20 PDT</small><br><strong>Monitoring</strong> - Our API is slowly recovering from the AWS S3 issues. We are continuing to monitor.</p><p><small>Sep 14, 12:13 PDT</small><br><strong>Identified</strong> - We are observing issues with Amazon's Simple Storage Service (AWS S3) affecting our API. We are in communication with Amazon.</p>tag:status.cloudsight.ai,2005:Incident/13545322017-09-12T05:46:56-07:002017-09-12T05:46:56-07:00Intermittent upstream network degradation<p><small>Sep 12, 05:46 PDT</small><br><strong>Resolved</strong> - Network conditions have been stable for several hours.</p><p><small>Sep 11, 20:51 PDT</small><br><strong>Monitoring</strong> - Upstream network conditions have returned to normal. Will continue monitor conditions for stability.</p><p><small>Sep 11, 20:36 PDT</small><br><strong>Identified</strong> - We've identified some issues with our upstream network provider and are working with them to resolve the issue. API users may experience elevated expiry rates.</p>tag:status.cloudsight.ai,2005:Incident/13341112017-08-21T10:24:40-07:002017-08-21T10:24:40-07:00Intermittent slowness with AWS ELBs<p><small>Aug 21, 10:24 PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug 21, 09:51 PDT</small><br><strong>Monitoring</strong> - AWS ELB latency issues appear to have been resolved. We are continuing to monitor.</p><p><small>Aug 21, 09:30 PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:status.cloudsight.ai,2005:Incident/13314032017-08-17T13:21:40-07:002017-08-17T13:21:40-07:00Intermittent API Outage<p><small>Aug 17, 13:21 PDT</small><br><strong>Resolved</strong> - We are continuing to monitor as well.</p><p><small>Aug 17, 13:03 PDT</small><br><strong>Monitoring</strong> - We have identified an issue with our background processing and have implemented a solution. We are now monitoring.</p><p><small>Aug 17, 12:50 PDT</small><br><strong>Investigating</strong> - We are currently investigating.</p>tag:status.cloudsight.ai,2005:Incident/13113812017-07-29T12:32:14-07:002017-07-29T12:32:15-07:00Slow Loading from AWS<p><small>Jul 29, 12:32 PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul 29, 12:02 PDT</small><br><strong>Investigating</strong> - We are currently investigating slow loading from AWS.</p>tag:status.cloudsight.ai,2005:Incident/12278292017-05-05T20:00:42-07:002017-05-05T20:00:42-07:00CloudSight API Outage<p><small>May 5, 20:00 PDT</small><br><strong>Resolved</strong> - The CloudSight API has been stable for several hours - this incident has been resolved.</p><p><small>May 5, 16:19 PDT</small><br><strong>Monitoring</strong> - The CloudSight API issues have been resolved and we are currently monitoring.</p><p><small>May 5, 15:55 PDT</small><br><strong>Investigating</strong> - We are currently investigating an intermittent outage with the CloudSight API.</p>tag:status.cloudsight.ai,2005:Incident/11910082017-04-07T15:27:51-07:002017-04-07T15:27:51-07:00CloudSight API Down<p><small>Apr 7, 15:27 PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr 7, 15:00 PDT</small><br><strong>Monitoring</strong> - A large running SQL batch query caused an out of memory issue on the database, causing problems with our app. We have failed over the database and are back to normal operations. We are monitoring the database and application performance at this time.</p><p><small>Apr 7, 14:46 PDT</small><br><strong>Investigating</strong> - We are investigating.</p>tag:status.cloudsight.ai,2005:Incident/11819312017-04-03T20:52:23-07:002017-04-03T20:52:23-07:00CloudSight API Database Maintenance<p><small>Apr 3, 20:52 PDT</small><br><strong>Completed</strong> - Our database maintenance is completed.</p><p><small>Apr 3, 20:00 PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr 2, 14:16 PDT</small><br><strong>Scheduled</strong> - The CloudSight API database will be undergoing account maintenance. Unfortunately, we do not have a definite end to this maintenance window, but hope the maintenance should be minimal</p>tag:status.cloudsight.ai,2005:Incident/11818992017-04-02T14:13:27-07:002017-04-02T14:13:27-07:00High Latency on CloudSight API<p><small>Apr 2, 14:13 PDT</small><br><strong>Resolved</strong> - A hanging SQL query caused increased locking on our database, resulting in queries exponentially taking longer than expected. This query has been addressed and we will monitor to ensure that the locks do not recur.</p><p><small>Apr 2, 13:09 PDT</small><br><strong>Investigating</strong> - We are currently experiencing high latency on the CloudSight API. We are investigating.</p>tag:status.cloudsight.ai,2005:Incident/11536192017-03-09T19:48:39-08:002017-03-09T19:48:39-08:00AWS RDS is down<p><small>Mar 9, 19:48 PST</small><br><strong>Resolved</strong> - The CloudSight API is back to operational, though there are still underlying issues at Amazon with their RDS service. We are waiting on them to respond and resolve the problem on their end before we declare our dependency on their system as fully operational.</p><p><small>Mar 9, 18:52 PST</small><br><strong>Monitoring</strong> - We have failed over to our hot standby AWS RDS database and are currently monitoring and working with Amazon support on this issue</p><p><small>Mar 9, 18:44 PST</small><br><strong>Identified</strong> - The problem has been identified as a continued issue at Amazon within their RDS environment. We are working with them to resolve this issue so the CloudSight API does not continue to have nightly database issues.</p><p><small>Mar 9, 18:43 PST</small><br><strong>Investigating</strong> - We are currently investigating a state where our AWS RDS instance has gone down. We are working with Amazon RDS support at this time.</p>tag:status.cloudsight.ai,2005:Incident/11524162017-03-08T22:53:52-08:002017-03-08T22:53:52-08:00Degraded Performance in API<p><small>Mar 8, 22:53 PST</small><br><strong>Resolved</strong> - AWS has found initial indications of a root cause and require additional time to confirm. As mentioned previously, once the AWS team shares their findings with CloudSight, we will write a postmortem. The CloudSight API has stabilized.</p><p><small>Mar 8, 22:06 PST</small><br><strong>Monitoring</strong> - AWS is investigating a periodic issue with degraded performance in one of our databases.</p>tag:status.cloudsight.ai,2005:Incident/11510422017-03-07T23:19:08-08:002017-03-07T23:19:08-08:00Degraded performance for CloudSight API<p><small>Mar 7, 23:19 PST</small><br><strong>Resolved</strong> - AWS is continuing to investigate a root cause, but the CloudSight API has been stable for several hours at this point. A postmortem will be written to include AWS' findings once they are shared with our engineering team.</p><p><small>Mar 7, 21:05 PST</small><br><strong>Update</strong> - The CloudSight API has returned to normal operation, but we have yet to receive a full diagnosis from AWS regarding the issue with RDS. We are continuing to monitor our systems (although they are fully operational) and work with AWS to determine a true root cause.</p><p><small>Mar 7, 20:31 PST</small><br><strong>Monitoring</strong> - We've identified a problem upstream with AWS RDS and have failed over to our hot standby database. We are continuing to monitor our services.</p><p><small>Mar 7, 20:06 PST</small><br><strong>Investigating</strong> - We are currently investigating degraded performance with the CloudSight API. We are currently investigating.</p>tag:status.cloudsight.ai,2005:Incident/11485972017-03-05T23:05:57-08:002017-03-06T11:39:53-08:00Investigating Slowness into CloudSight.ai and CloudSight API<p><small>Mar 6, 11:39 PST</small><br><strong>Postmortem</strong> - An upstream issue with Amazon Web Services (AWS) Relational Database Service (RDS) caused recurring CPU spiking on our master database instance, resulting in service disruptions for both the CloudSight API and cloudsight.ai.
<br />
<br />CloudSight engineers failed over to our hot standby database resolving the issue, yet are still working with Amazon to determine the root cause of the failure of our former master database.</p><p><small>Mar 5, 23:05 PST</small><br><strong>Resolved</strong> - The failover to our backup database at AWS appears to have resolved the issue with the CloudSight API and cloudsight.ai. We are continuing to work with AWS to determine the root cause of the primary database failure and will update via a postmortem on 7 March 2017. At this time, we are resolving this issue as our services have returned to normal operation. However, we are continuing to work with AWS to determine a true root cause.</p><p><small>Mar 5, 22:20 PST</small><br><strong>Monitoring</strong> - CloudSight has failed over to our backup database and CloudSight.ai and the CloudSight API are now working again. We are continuing to work with AWS to identify a root cause.</p><p><small>Mar 5, 21:51 PST</small><br><strong>Investigating</strong> - We are currently investigating an apparent outage with our database hosted at AWS, causing downtime for the CloudSight API and CloudSight.ai site.</p>tag:status.cloudsight.ai,2005:Incident/11417312017-02-28T15:45:31-08:002017-03-01T18:21:58-08:00CloudSight API Outage<p><small>Mar 1, 18:21 PST</small><br><strong>Postmortem</strong> - Approximately 17:30 UTC on 28 February, 2017, CloudSight detected issues with receiving blank images to our system and not being able to provide tags to images already in our system.
<br />
<br />It was determined that issues upstream with Amazon Web Services (AWS) Simple Storage Service (S3) had experienced a significant outage, which was preventing images submitted to the CloudSight API and by CloudSight apps (CamFind and TapTapSee) from being received and stored on AWS S3, and returned to our main application for tagging said images.
<br />
<br />We worked closely with AWS' support to monitor the situation, and created additional instances of our application so that when services became available, approximately at 22:00 UTC on 28 February, 2017, CloudSight was able to handle the queued volume of images to be tagged.
<br />
<br />CloudSight experienced a high volume of images once AWS service was restored, but was back to normal operational capacity at approximately 23:00 UTC on 28 February, 2017.
<br />
<br />We have continued to monitor this situation and monitor AWS services used by CloudSight, and will do so through 2 March, 2017.</p><p><small>Feb 28, 15:45 PST</small><br><strong>Resolved</strong> - All components for the CloudSight API and cloudsight.ai portal have returned to an operational status based on average throughput and response times. There may still be some outlying performance that are beyond our control, and that we will continue to monitor. However, as of now, the CloudSight API is considered to be fully operational.</p><p><small>Feb 28, 14:08 PST</small><br><strong>Monitoring</strong> - The CloudSight API is returning to normal operation, but still experiencing slightly higher than normal latency. We are continuing to monitor.</p><p><small>Feb 28, 13:21 PST</small><br><strong>Investigating</strong> - Additional third party services have been disrupted again. We are working again with these services to restore operations as quickly as possible.</p><p><small>Feb 28, 13:13 PST</small><br><strong>Monitoring</strong> - Our third party services appear to be mostly restored, and CloudSight API operations are returning to normal, but at a slower than average pace. We will continue to monitor and update as our API recovers and returns closer to average speed.</p><p><small>Feb 28, 11:48 PST</small><br><strong>Update</strong> - Our third party vendors have identified additional issues causing outages, affecting several parts of our applications, one of which continues to be our CloudSight API. We are continuing to monitor and work with our vendors to resolve these issues.</p><p><small>Feb 28, 11:08 PST</small><br><strong>Update</strong> - Additional third party services are having issues, causing additional outages for CloudSight. We are continuing to work with our third party to resolve these issues as quickly as possible.</p><p><small>Feb 28, 10:24 PST</small><br><strong>Identified</strong> - We have identified the issue upstream and our third party service has acknowledged they have discovered the incident and are investigating. We are working closely with them to stay up to date and get this issue resolved as quickly as possible.</p><p><small>Feb 28, 10:05 PST</small><br><strong>Update</strong> - Issues impacting the CloudSight API appear to be upstream from our application. We are continuing to investigate the outage and working jointly with our third party providers to determine a root cause.</p><p><small>Feb 28, 10:04 PST</small><br><strong>Investigating</strong> - At 9:31 a.m. PST, services for CloudSight's API were interrupted. We are currently investigating.</p>tag:status.cloudsight.ai,2005:Incident/10221332016-12-14T01:57:46-08:002016-12-14T01:57:46-08:00Intermittent "500 Internal Server Error"<p><small>Dec 14, 01:57 PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec 14, 01:37 PST</small><br><strong>Monitoring</strong> - Upstream has identified the root cause and has implemented a change to fix. We're monitoring to ensure a successful resolution.</p><p><small>Dec 14, 00:46 PST</small><br><strong>Investigating</strong> - We're currently experiencing an issue with one of our upstream provider's services. We're currently investigating the issue with them to determine the root cause.</p>tag:status.cloudsight.ai,2005:Incident/9442802016-10-09T21:30:59-07:002016-10-09T21:30:59-07:00Scheduled Maintenance<p><small>Oct 9, 21:30 PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct 9, 20:45 PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct 9, 20:33 PDT</small><br><strong>Scheduled</strong> - Service may be degraded for a periods of time during the maintenance window.</p>tag:status.cloudsight.ai,2005:Incident/7970202016-09-17T16:20:34-07:002016-09-17T16:20:34-07:00System response times have returned to normal<p><small>Sep 17, 16:20 PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p>