The solution is to frequently and automatically test backups by:
The first part sounds hard, but isn’t.

One of the issues of yesterday’s GitLab.com “database incident” is that most of their database backups weren’t being tested, and when they needed a restore, they discovered that most of the backup methods hadn’t been working.

Automated backup testing isn’t difficult — it’s one simple shell script, called by every night, piping its results to .

Two of the best pieces I’ve read about Apple recently:
They may seem at odds, but I agree with a lot from both, and every Apple enthusiast and commentator would be well-served by reading and considering both.

Apple can — and, I believe, does — have both a well-performing CEO and significant problems in its products that have me both worried and saddened as an Apple customer and enthusiast.

Not only was it truly mind-blowing at the time, but in retrospect, so much of modern computing was invented for the first iPhone and revealed to the world in that hour.

I’ve seen a lot of major product launches and technology changes in my life and career so far, but nothing else I’ve seen has ever come close to the surprise, magic, and magnitude of the first iPhone, and I don’t expect it to be surpassed in my lifetime.

Everything about the iPhone seemed impossible to the technology world of early 2007.
“You can’t make a good phone without buttons.”
“You can’t fit a desktop-class OS on a phone.”
“There’s no way that’s a full-blown web browser.”
“That has to cost a thousand dollars.”
Yet over the course of an hour, Steve destroyed every rule we thought we knew.