Handling Secrets with AWS

Let’s say you’re deploying an application onto AWS. Very few applications are entirely self-contained; we’ve collectively built an ecosystem that’s intertwined with dependencies between a variety of services and platforms. To communicate between them, your application invariably is going to need to store various credentials in some form; usually these are username/password pairs or API keys.

Let’s further assume that you’re not a dangerous lunatic who hardcodes those secrets into your application code. You want to find a way to maturely and sensibly store those secrets in ways that are centralized (so you don’t have to update every server / container / function whenever one changes), secure (so they remain secret), and accessible (in practice, there’s little difference between a service going down and you losing your credentials to talk to the service). There are a number of ways to do this with native AWS services, that I will divide into three categories: bullshit, reasonable, and controversial. There’s a fourth category of using things outside of AWS such as Hashicorp’s excellent Vault product, or another cloud provider’s secret storage service; I’m considering things like that out of scope for what I want to talk about today.

Bullshit answers

Let’s begin with the bullshit options. At their heart, secrets are fundamentally key/value pairs. I want to store something called “ROOT_PASSWORD” and the value of the secret itself, such as “Kitty!”.

You could use Route 53 TXT records, which is a fun gag but seriously do not do this even if you like the chaotic energy of misusing a DNS service as a database; security by obscurity is a bad plan when it comes to sensitive credentials.

You might also be tempted to do something “clever” like using resource tags within AWS. This is to be avoided! AWS doesn’t consider tagging to be sensitive; it’s “metadata” to them, which means they’re going to be treating it with a level of care you probably won’t like. It’ll show up in the AWS console, in logs, in billing files, in services like the little known but greatly admired Tag Editor that you almost certainly haven’t scoped access to, and so on. The AWS tagging documentation explicitly tells you not to do this in a scary red box; I promise, you want to pay attention when AWS tells you not to do something.

Reasonable answers

AWS has two offerings explicitly aimed at solving this problem.

The first is Systems Manager Parameter Store; despite the extra words inherent in its name, like most of the Systems Manager sub-services it absolutely rocks. This is my go-to answer for managing secrets, specifically via its SecureString parameter type. I either teach my code to make Parameter Store calls when it needs the relevant credential, or else I render it and deliver it to the container or Lambda function at deploy time as an environment variable; both of these patterns have been “approved” by AWS in various ways. It has the added bonus of being completely free, at least until you get into whatever the hell “advanced parameters” are.

Parameter Store has a couple of flaws, though. First, its rate limits are set rather low; 40 transactions per second may seem like a lot until you realize that in some scenarios such as spinning up a whole bunch of jobs via AWS Batch or kicking off a flurry of Lambda functions could easily overwhelm it. Additionally, it has no support for cross-account access, nor does it have any inherent ability to replicate secrets between regions; if you want such things, you’re building some scaffolding yourself to support it.

AWS’s answer to those flaws came in the form of AWS Secrets Manager, a top level service aimed at this problem. Don’t let people steer you otherwise; there is no inherent security downgrade if you use Parameter Store instead. Secrets Manager doesn’t offer improved security directly; instead it offers a bunch of things that you’d otherwise have to manage or build yourself.

It automatically can invoke Lambda functions to rotate passwords for just about anything on-prem or off, it features native support for RedShift, RDS, and DocumentDB (which is Amazon Basics MongoDB) so you don’t have to worry about rotating those secrets, and it sets a default quota of a whopping 5,000 transactions per second. It also features cross-region secret replication, easier audit trails without additional work on your part, and frankly a way better name. I’m not joking about the name; it’s super hard to see a service called “Secrets Manager” and not think that if you’re using anything else to manage credentials that you’re somehow doing it wrong.

The big and arguably only downside to Secrets Manager is significant: its cost. Secrets Manager charges a whopping 40¢ per secret per month; for context, that’s more than it costs to store 17GB of data within S3 for the same time period. Additionally it charges 5¢ per every 10,000 API calls; combined with its ability to do 5K transactions per second, the numbers rapidly become rather terrifying.

Controversial answers

Given the API rate limits and lack of cross-region or cross-account access inherent to Parameter Store, and the frankly baffling economics of secret storage in Secrets Manager, I’ve had success leveraging storing secrets that are going to be accessed at high volume in either DynamoDB or S3, depending upon what the specific requirement is. This obviously isn’t the first thing I’d reach for, and you’d have to build a bit of logic around things like versioning and rotation unless you want to chase a whole bunch of problems down, but now that both services support global replication and more favorable economics it’s certainly not a terrifying pattern at all.

That said, yes, I know; it’s clearly on the wrong side of the “Build vs. Buy” debate, gives you a bunch of extra things to maintain for no direct value in return, and should really only be used at significant scale at a point when the other options I’ve highlighted become unacceptable for one reason or another.

The Takeaway

Fundamentally, there’s not a bad path here short of the blatant irresponsibility of treating secrets as if they weren’t sensitive in the first place. This is a pretty good representation of how I think about managing secrets within AWS. If you think I’ve gotten it wrong, please let me know; I’m positive there are edge cases I haven’t considered.