Disaster recovery / business continuity / “backups” are always an interesting subject for very large scale cloud environments. Many of the old data-center strategies that grumpy old sysadmins (that’s me!) relied upon don’t hold water anymore. I mentioned a couple of years ago that S3 isn’t a backup, and that’s true in isolation. AWS’s vaunted “11 9’s of durability” solely apply to disk durability math; disasters, human error, and the earth crashing into the sun aren’t accounted for in that math.
What Cloud Providers Tell You vs. What Actually Happens
Cloud providers love to talk about their redundancy, their availability zones, and their durability numbers. But here’s what they don’t emphasize enough: most “disasters” aren’t actually externally triggered disasters – they’re mundane mistakes made by sleep-deprived humans who thought they were in the staging environment, or well-intentioned folks making a small configuration mistake that compounds when something else intersects with that error.
The Human Element: Your Biggest Threat
Let’s be honest: the chance that you’ll fat-finger something into oblivion is orders of magnitude more likely than the odds of a simultaneous failure across multiple AWS availability zones. You’ll delete the wrong object from a bucket, run that terrifying production script in the wrong terminal window, or (my personal favorite) discover that your production environment credentials somehow made it into your staging configuration. This is why privilege separation isn’t just a nice-to-have – it’s a must-have. The folks who can access your backups shouldn’t have access to production data, and vice versa. Why? Because when (not if) credentials get compromised or someone goes rogue, you don’t want them to have the keys to both your castle and your backup fortress. “Steven is trustworthy” may be well and good, but the person who exploits Steven’s laptop and steals credentials absolutely is not.
The Multi-Cloud Backup Conundrum
“But Corey,” you might say, “shouldn’t we just back up everything to another cloud provider?” Well, yes, but not for the reason you think. You should maintain a “rehydrate the business” level of backup with another provider not because it’s technically superior, but because it’s easier than explaining to your board why you didn’t when everything goes sideways. Remember: your cloud provider – and your relationship with them – remains a single point of failure. And while AWS’s durability math is impressive, it won’t help you when someone accidentally deletes that critical CloudFormation stack or when your account gets suspended due to a billing snafu.
The Reality of Restore Operations
Here’s something they don’t tell you in disaster recovery school: most restores aren’t dramatic, full-environment recoveries. They’re boring, single-object restores because someone accidentally deleted an important file or overwrote some crucial data. Embrace this reality. Design your backup strategy around it. This means: – Making common restore operations quick and simple – Maintaining granular access controls – Keeping detailed logs of what changed and when – Testing restore procedures regularly (and not just during that annual DR test that everyone dreads)
The “Back Up Everything” Trap
Here’s a controversial opinion: backing up everything in S3 to another location is both fiendishly expensive and completely impractical. Instead: – Figure out what data actually matters to your business – Determine different levels of backup needs for different types of data – Don’t waste resources backing up things you can easily recreate – Document what you’re NOT backing up (and why) so future-you doesn’t curse present-you
The Uncomfortable Truth About DR Planning
If there’s one constant about disasters, it’s that they never quite match our carefully crafted scenarios. The decision to activate your DR plan is rarely clear-cut. It’s usually made under pressure, with incomplete information, and with the knowledge that a false alarm could be just as costly as a missed crisis.
This is why your DR strategy needs to be:
- Flexible enough to handle partial failures
- Clear about who can make the call
- Tested regularly (and in weird ways – not just your standard scenarios)
- Documented in a way that panicked people can actually follow
The Bottom Line
Your DR strategy needs to account for both the dramatic (multi-region failures) and the mundane (someone ran `rm -rf` in the wrong directory). Build your systems assuming that mistakes will happen, credentials will be compromised, and disasters will never look quite like what you expected. And remember: the best DR strategy isn’t the one that looks most impressive in your architecture diagrams – it’s the one that actually works when everything else doesn’t.
