I think I've seen something similar online and was way over my pay grade but agree with you. Xero with their 862,000 subscribers probably can't justify the expense unlike another AWS client calling themselves Netflix whose "NZ content" is streamed from a lot closer to home.

Expense varies with the level of redundancy you want:

"Cold", where you have a backup of your data and server images outside the region in something like AWS Glacier, wouldn't be too expensive. The time to get the environment running could be days, it depends how much preparation is done.

"Pilot light", where you have a minimal mirror environment running and can scale it up, costs more, but not too much.

"Active - active", where services run across two sites, is more complex and expensive, with data synchronisation issues. You also have to decide whether you size each environment for the full load in case one fails or half the current load with the ability to scale if one site goes down.

Highly critical things like NASA mission control and anything where lives can depend on it might use multiple active sites. Accounting software, well, customers can probably wait an hour for it to scale up.

There are also very good reasons to rely on external services. There's no way a small organisation can get access to the security and features of AWS or other cloud providers themselves economically. I run a small server for my business websites, it costs me about NZ$10/month for a small VM. For that I get firewalls (security groups and NACLS), backups (disk snapshots, plus I sync data and database to dropbox), analytics (CloudWatch, free), near real time alerts of problems (Cloudwatch again), a plethora of services I could use, plus the ability to spin up my website in any of around 30 different data centers with very little effort.

It's interesting to see how many services are poorly architected. Yes S3 stores data across multiple AZs in order to increase durability and availability, but it seems something must be shared, or they deployed to all AZs in the same time period. Cross region replication is recommended for high reliability.

TLDR: many places put all their eggs in one basket.

It's coming back up now, but may be an hour before it's fully working.

--

Update at 12:52 PM PST: We are seeing recovery for S3 object retrievals, listing and deletions. We continue to work on recovery for adding new objects to S3 and expect to start seeing improved error rates within the hour.

Surprised at Xero - having only migrated to AWS late last year would've thought they would have designed more resiliency

It's interesting that there are no reports of outages on amazon.com. Werner Vogels the Amazon.com CTO famously said "Everything fails all the time", and they have more AWS experience than anyone (completely separate business units), so I assume they've architected for failure.

Ive read reports stating S3 buckets were down across multiple regions so not sure how cross region replication would have helped here?

Amazon are pretty transparent on outages, there's nothing in their status history (bottom of the page), and I haven't read anything about a multi region outage. Can you share link to a reputable source?

At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.

We've all done it. Just not on something that takes down a large amount of the internet.

At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.

We've all done it. Just not on something that takes down a large amount of the internet.

"The Amazon Simple Storage Service (S3) team was debugging an issue causing the S3 billing system to progress more slowly than expected."