Before These Unprecedented Times struck, my business partner and I hosted two Q&A sessions. We were asked all kinds of questions about AWS’s multitude of services, which we also answer below, lightly edited to make us sound way more intelligent than we already did.
How does RDS pricing compare to other managed database service providers that work on top of AWS or other providers? Do you have any recommendations?
My default is to first take a look at where data needs to live. You might not need a relational database at all.
We’ve seen S3 become a lot more intelligent and queryable. While the latency isn’t what you’re ever going to see around a performant RDBMS, it’s certainly not bad, and you can get pretty far down the road with that.
When you deviate into a traditional database, there are a few different things to consider. Firstly, there’s the NoSQL option, and DynamoDB is an absolute marvel if that supports your use case.
Otherwise, you’ll have to figure out whether to run MySQL on top of RDS, run it on top of EC2, or have a third-party manage something on top of EC2 instead.
My default would be to go to RDS because there’s a lot of things it sweeps away that you don’t have to worry about. And the management overhead costs are generally not significant, either. You’ll pay for what you use and there’s a fee built into it. But you’ll understand how it works and how it breaks, which is important in your database layer.
The next option is running on top of EC2, which gives you a lot of knobs to fiddle with. And it’s entirely likely you’re going to break things in a massive way. Of course, do this if you need to do it. But do it for the right reason—not that you might have to move to a different cloud provider at some ambiguous point in time.
The third option is using a third-party service—something that hangs out in another AWS account. But that can introduce latencies and you won’t have the same visibility. Go down this path cautiously. That’s not to say there aren’t any terrific offerings here. But you now have something else in the critical path between you and AWS. And that introduces a whole bunch of risk—not just technical stuff, but business risks, too. I would want to see a massive capability story here before entertaining this option.
Do you know any good ways of analyzing spend on RDS Charged Backup Storage?
Here’s how RDS Charged Backups work: Up to a certain point, they’re free based upon usage and provisioned capacity. Then it starts costing you money.
When you start seeing those fees, it usually indicates something is awry. Maybe you’re not aging things appropriately, or maybe you have 20 copies of something when you only need a couple. You can generally chase this down with Cost Explorer.
Again, those charges may be an indicator that something is amiss. But there are probably bigger fish to fry. I’ve yet to see a story where RDS Charged Backups were a meaningful portion of the bill. If that’s you, please reach out and let me know–I’d love to see it!
Does anyone actually understand Aurora IO and how it reduces costs?
Nope!
There’s a method you use to calculate what the Aurora IO charge is going to be. It’s called suck it and see. That’s not nearly as scandalous as it sounds—it was the way you’d figure out if a power cable was live back in the day.
Of course, you wouldn’t actually suck on a power cable to see if it blew you through the wall. That would be spectacularly dangerous. You’d have an apprentice do it instead.
It’s the same kind of approach here. You end up modeling the use case and seeing what it’s going to cost after you’ve been charged.
The reason this comes up is because with the Aurora pricing model it’s 20 cents per 1 million I/O operations. But nobody knows how many IO operations the database will actually do because an Aurora IO operation doesn’t map cleanly to a MySQL or Postgres IO operation.
A lot of AWS billing models tie directly to things that are impossible to calculate in advance. Tell me to spin up a single web server running WordPress and serve traffic, and I won’t be able to tell you what that monthly bill is going to be with any level of accuracy until we start seeing what the traffic to the site looks like.
Are you a fan of AWS Config and conformance packs? Have you seen clients use it successfully?
Config, yes—especially with the latest pricing changes. It makes it AWS-like in that it doesn’t tell you how much it’ll cost until after the fact, but it doesn’t charge you a fixed fee per rule anymore, which is handy–and almost always means you’ll pay less under the new model than the old.
I haven’t seen adoption of the conformance packs yet.
But to be fair, the reason most people are using Config is you get it “for free” (it is not free) as a mandatory part of Control Tower or any form of organization-wide governance model.
I’m able to pull the consumption/cost data from Cost Explorer based on a single dimension (e.g., BU, Service, and Instance). Is there a way to pull the data on a multidimensional/hierarchical basis?
The answer is no.
We solved this problem at The Duckbill Group by ingesting the Cost & Usage Report into Athena and using Tableau to slice and dice it because QuickSight is terrible. This has worked super well for us.
If you’re looking for multidimensional querying and visualization, BI tools are the way to go. Unfortunately, Cost Explorer doesn’t support this. If it did, it’d be an absolutely killer feature.
How do I think about storage? We end up backing up machines four times per month and we keep adding space to drives for random customer data. How do I approach this from a policy perspective?
This isn’t always the right answer, but I’ll say it: Store less.
In seriousness, there are multiple storage offerings that more or less map across every cloud provider out there: file, block, and object. The cloud collectively took a vote a long time ago, and it became very clear that object store was the clear winner, and optimized for that. Look at S3, AWS’s object store. The economics are great! There’s no cross-AZ transfer, you get an obnoxious number of nines of durability guaranteed, and you have a per-request charge. So, you’re spending something like 2.3 cents per month per gigabyte vs. 10 cents per month per gigabyte if you were to instead store data on a gp2 volume.
But wait–the economics of block storage volumes get oh so very much worse. Take it a step further: Now you have to store that data that’s on your EBS volume in multiple places (assuming you care even slightly about durability), like three AZs, now you’re spending 30 cents per gigabyte instead of the object store’s 2.3 cents.
But wait, it gets even worse: you have to pre-provision EBS volumes before you can use the storage, which inherently results in overprovisioning the volume (because nobody likes a disk volume that’s 100% full!) and paying a multiple of what your data storage costs.
Lastly, you get to pay 2¢ per GB every time data crosses an AZ boundary. So assuming three availability zones, each GB costs you an additional 4¢ in replication charges.
So yeah, use object storage wherever possible.
That’s not viable for everything. Some things demand EBS volumes. I get it, that’s okay–but take some time to figure out what your usage pattern looks like.
You almost never want to be using io1 (or now, io2) volumes. If you see those volume types on your bill, it’s usually a misconfiguration. It’s faster than gp2, but that’s usually not where you’re smacking into limits. Instead, the bottlenecks that you’re going to run into with gp2 are almost always an instance limit on EBS-optimized volumes that you’ll hit first based on the size, generation, and family of your instance. AWS has a whole ridiculous table of what these limits are, and it’s eye-popping if it applies to you.
Bottom line? There are a lot of opportunities to explore storage optimization further. Just make sure that you’re aimed at the right problem before you dive into it. Getting rid of EBS and replacing it with S3 entirely is the right path in isolation. But there will always be constraints that mean there are given workloads for which that won’t work.
Have another question burning on your mind? Ask me on Twitter.
