Ask a Cloud Economist: Cost Attribution in AWS

Transcript
Corey: This episode is sponsored in part by our friends at Linode. You might be familiar with Linode; they’ve been around for almost 20 years. They offer Cloud in a way that makes sense rather than a way that is actively ridiculous by trying to throw everything at a wall and see what sticks. Their pricing winds up being a lot more transparent—not to mention lower—their performance kicks the crap out of most other things in this space, and—my personal favorite—whenever you call them for support, you’ll get a human who’s empowered to fix whatever it is that’s giving you trouble. Visit linode.com/screaminginthecloud to learn more, and get $100 in credit to kick the tires. That’s linode.com/screaminginthecloud.

Pete: Hello, and welcome to AWS Morning Brief. I am Pete Cheslock.

Jesse: And I'm Jesse DeRose.

Pete: We're back again, and we're here to answer an audience question. So, every once in a while people tweet at us—you can tweet me @petecheslock. Jesse, what is your Twitter handle?

Jesse: @Jessie_DeRose.

Pete: Yeah, mine is just petecheslock. I do feel bad for the other Pete Cheslock, who actually does live in Boston as well because taking all of his profile names.

Jesse: You should change yours to @therealpetecheslock, or he should change his to @therealpetecheslock, and then it'll just be an ongoing escalating battle.

Pete: That's very true. So, occasionally on the Twitters, we get questions asked of whatever around Amazon cost management, things like that. And we wanted to actually take this opportunity to answer one of the more interesting questions that we received. Because granted, sometimes we get questions and they're pretty boring, so we don't answer them. We just focus on the fun ones, [laugh]—

Jesse: [laugh].

Pete: —selfishly, but we got this question that was really interesting. It had to do with someone who is essentially starting over within Amazon Web Services, meaning they were going to be redeploying their application into a series of new AWS accounts. And they asked us, “What are the most recent best practices—” I hate that term, but the important things you should do and consider when you're deploying into Amazon, into AWS. And we kind of sat back, we thought to ourselves, “Wow, how often does someone have that opportunity?” Right, Jesse?

Jesse: Yeah. Not in any of my experience has that happened for me. I'm very, very envious of these people.

Pete: Yeah, I had that opportunity one time, where we were essentially doing that, like, net-new, starting over. But this was years ago, where there wasn't a lot of insight into this, and we didn't have the features like we have today where Amazon organizations—AWS Organizations—allows such an easy way to create accounts and get started with multiple accounts. So, anyway, we want to take this opportunity to talk about what we believe and what we see as the things that you should focus on, what you should optimize for when getting started, when creating, kind of, net-new in AWS.

Jesse: Yeah, there's a lot of different things that you can optimize for in AWS, and it really depends on what your business goals are; what do you ultimately want to accomplish when you are deploying your application into the cloud? But one of the big ones that we see, selfishly, here at Duckbill Group is cost optimization. And so we wanted to talk a little bit more about cost allocation and cost attribution—which are essentially the same thing, we may use the terms interchangeably in this conversation—to talk about how you can think about cost attribution, why you should think about cost attribution and some of the best ways to go about implementing that in AWS as you're building these new accounts, this new space.

Pete: Yeah, and that being said, I really like people to really think when they create these things. Again, what are you optimizing for? Some people might say, “Oh, well, we want to optimize for security.” And that's great. You absolutely should do that.

Jesse: Sure.

Pete: Security is a first principle, something to absolutely focus on. But what if I told you that the other, probably, most important thing in AWS is—and something if you're not doing it today, you're going to be asked to do it in the future—is accurate cost attribution. And what if you could do both highly secure accounts, and segment based on security, but also get this cost attribution? That is, I think, what we're going to dive into today.

Jesse: Yeah, I think that there's a lot of big conversations around engineers, and multiple other teams when you start talking about the DevOps movements, the DevSecOps movements, all these movements of the software engineers who are actually writing the code and the engineers or the operations folks who are—maybe—managing the infrastructure, maybe deploying the code, maybe the software engineers are deploying the code, it really depends on your team setup. But there's this, kind of, idea that the engineering teams that are working with this code, and then there's all these other teams in the company that have other things that are their top priority, and starting to bridge that gap to have conversations with finance to better understand what do they need to know from you about how you're spending money in AWS, and security who wants to better understand are we patched for the upcoming audit? Are we compliant based on these terms? It's really important to start thinking about how you optimize in AWS based on those ideas, those conversations with other teams. So, that's kind of ultimately what I'm thinking about, specifically, today, specifically about the conversation between finance and engineering and talking about cost attribution.

Pete: But Jesse, aren't tags supposed to solve all of my problems when it comes to cost allocation?

Jesse: [laugh]. Oh, I wish. They are supposed to. There's that whole idea of ‘set it and forget it,’ there's a big movement of ‘tag it and forget it,’ and as much as I want to believe in that, it’s unfortunately just not true. Like, tagging is definitely a first step, but it goes so much further than tagging and I think that's one of the big things that a lot of folks miss or don't think about when they're talking about tagging and cost attribution.

Pete: If you loved it, you would have put a tag on it.

Jesse: [laugh].

Pete: But really, while tagging is an important thing to do, and we've seen some of our clients, their tagging percentages can be upwards of 90 percent, which is herculean in ability and effort to reach that level of coverage, but even then, getting that last 5 to 10 percent in many cases could be actually impossible to do because there can be a series of spend within Amazon which is just untaggable, or at least untaggable in a realistic way. And that's where multiple accounts can really help your business help you break out those costs in a really, really clean way.

Jesse: Yeah, I think that's one thing that's really important to think about because there's always going to be unattributable or untaggable spend in any AWS account, especially in a shared account where you have multiple teams deploying resources, multiple products sharing the same resources or sharing the same space. So, this is where we start talking about things that finance might want to know about in terms of how much is it actually costing us to run your different products? How much is it actually costing us to run these various different microservices of a given product?

Pete: Yeah. Like, services within a broader product, like service profitability within maybe a large SaaS product, knowing that level of insight, right?

Jesse: Absolutely. And then you can start thinking about not only what is it costing me, but then maybe you can start thinking about charging that back to the actual teams that are spending the money on those resources. And I get that a lot of people probably cringe when they hear me say that, so I don't want to say that that's an immediate first step that you should take, but start thinking about the budget of each component of your larger microservice or each component of your product if you've got a large SaaS product. Think about, how are you budgeting for future spend to better understand where can you spend more money? Where can you spend less money? Do you have the funds to hire another employee? 

Maybe there's some optimization work that you want to focus on that will free up funds to hire another employee, or to look at other things. Start having that conversation with your finance team to better understand what are the things that are really important to them, and how can you help them answer their questions and reach their goals? And that's ultimately where tagging resources is a very, very important first step to look at all this spend from the perspective of the team or the product. But again, this is not the only step; you should also be looking at splitting up the spend into separate linked accounts.

Pete: Yeah, so most folks getting started on Amazon take their resources and put them into, maybe, a single account with multiple VPCs—those VPCs could be broken out by Dev and Prod, or maybe they're broken out by availability zone—or maybe you create an account for Prod and an account for Dev, then that's your separation. Some clients of ours and some companies out there, they're collecting accounts based on acquisition. So, maybe it's different global business units have various accounts related to it. But if you are starting from scratch, if you are implementing something brand new, we actually recommend a much more intentional planned approach. But there's three key things to remember when you do this: within all of AWS, the account itself is really the only hard security boundary, all the way down to the root level access that can be controlled with hard tokens. 

That individual Amazon account is that hard security boundary. The next important thing is that accounts are free. It's free software, they're just giving it away. And you can create these very easily with AWS Organizations. And finally, tooling exists and is mature, both, again, native and third-party tooling to make multi-account management very easy, very straightforward in a way that just did not exist, God, even what, four to five years ago. I mean, it's truly amazing.

Jesse: Yeah, that's when I especially want to call out because AWS Organizations has come a very long way in terms of helping an organization manage multiple AWS accounts, not just in terms of creating them, but also applying standardized policies and security policies across these accounts so that you can literally create cookiecutter AWS accounts that are already adhering to the best practices that you as an organization need to follow.

Pete: Yeah. So, when you are creating your accounts, there's a series of accounts that, largely, everyone's going to need, how much you use each of these accounts is really dependent on your architecture or organization. But at the very least, you got to create one account; you got to get started. And this is your master—or your, what we would call ‘root’ account, moving away from an antiquated terminology of the master/slave—we would call it the root account. It’s the core account you create. This is your first one; this contains the AWS Organization’s configuration. 

The Master Payer Account lives here as well—that's a Amazon wording there. And this is, again, all org-wide control policies, and ideally, all of your savings plan and reserved instances would be managed within this account as well. And that allows for those savings, those commitments to be spread across all accounts underneath it. And that allows you to really ensure there's as little waste as possible when making those accounts. Go ahead, Jesse. Yeah.

Jesse: This is also a really great opportunity to give finance access to this root account in a very limited way so that they can look at billing information. So, they can look at Cost Explorer, for example, and see spend trends across all of your sub-accounts; they can see what spend looks like across all of your products, across all of your resources, and get a better understanding of the total AWS spend, broken down by product, or by microservice, or by team, or by some other business unit or entity that you decide.

Pete: Exactly. The next account that you want to create is essentially your security account. This would be all your security and logging tooling, like your SIEM or your SIM—whichever way you want to pronounce it—your vulnerability audit scanners. But most importantly, it's your CloudTrail logs. So, CloudTrail is something that you should absolutely enable in every single account, but you want to have those go to this account; if there was only one thing your “Security”—air-quotes—account was used for, its storage of your CloudTrail logs in a safe, immutable place in that location with a restricted group of people who have access to it. That's really important. 

Over time, as your usage grows, maybe you're using Transit Gateway, for example, to connect all of these accounts so that they egress their network traffic through security. Maybe you're running various proxies within the security account. Again, it's a very restricted centralized place that, again, at the very least, no matter what you're doing, if you are deploying anything to Amazon, should contain just your CloudTrail logs.

Jesse: And I think this also gets back to the point that I made earlier about AWS Organizations and applying sort of a cookiecutter template to all of the accounts that you create, after you've created this security account, you can apply this cookiecutter template to all of your other sub-accounts in such a way that they all automatically point and send all of this information back to this security account. And again, as Pete mentioned, only the security team will have access to this account, or only your specific SIM group, or your specific security group will have access to this account. And so when you talk about things like Platform as a Service, the security team doesn't need to have access to every single individual AWS account, but as long as they have access to this central hub, they can see what's happening in all the other accounts and they can have conversations with the teams who are deploying resources in those other accounts by knowing that maybe there's a potential vulnerability in a specific business unit account, or a specific team account, or a specific environment account. And if they look at the unit that that account is for, or if they look at the user-defined cost allocation tags that are associated with the resources that are part of this vulnerability, they have a way to talk to somebody and actually get this issue resolved.

Corey: This episode is sponsored by our friends at New Relic. If you’re like most environments, you probably have an incredibly complicated architecture, which means that monitoring it is going to take a dozen different tools. And then we get into the advanced stuff. We all have been there and know that pain, or will learn it shortly, and New Relic wants to change that. They’ve designed everything you need in one platform with pricing that’s simple and straightforward, and that means no more counting hosts. You also can get one user and a hundred gigabytes a month, totally free. To learn more, visit newrelic.com. Observability made simple.

Pete: Yeah, the next account, this could have a few different names: maybe it's your Tools account, maybe it's your shared services account, but this is your org-wide tooling, shared tooling, CI and CD systems, your version control systems, source control systems, config management systems, things like that. And these would be the things that, maybe, are required to support all these other services. This could even be your monitoring infrastructure might live here. And these would be the things that maybe you take this spend—talking about cost allocation, just like the security account—you take the spend that's within here and as one client of mine referred to the best analogy that you give the peanut butter schmear across all of the other—

Jesse: [laugh].

Jesse: —from a cost perspective, you just spread that peanut butter, that you spread that spend across all the other accounts. So, if you had five other specific service accounts, you could divide this spend by five. And that's how you essentially show back or charge back that cost to those accounts. But most importantly, this is going to improve your security posture by having, again, a central place with all of the company assets. And again, AMI repositories, container repositories, source control, everything like that. 

You can restrict this down, you can give the level access, you can peer with these other accounts, a lot of interesting stuff you can do. But largely creating a place where you're capturing, you’re bucketing spend—even if it's not tagged within this shared services kind of group—to be allocated at a later time.

Jesse: This is where I feel like I need a ‘plus-one emoji’ audio bite so that I could just add that. I don't have anything specific to add to here. And—

Pete: Oh yeah.

Jesse: Yeah, [laugh].

Pete: Like the old—is that from the 90s or whatever? Yeah. [laugh].

Jesse: Yeah, yeah, yeah, absolutely. And so, like, that—just a huge plus-one to making this account and put all of your tools together in a shared resource space.

Pete: Yeah, the last account that we'll talk about is essentially an identity account: user access control. This is really terminating your access control in this account, whether it's an SSO solution or Federation solution, or you're just creating individual user accounts or IAM roles that people are authenticating against. Again, centralize it all in one place. You should not have user accounts existing in all of your accounts. Amazon, IAM, Federation has largely solved that issue. 

You can use the AWS SSO services, there's third-party services that are great, like Okta, and OneLogin. Again, if there's one thing you should not be doing, it's creating individual user accounts in your individual Amazon accounts. That will become, essentially, impossible to manage, so don't even start there. Start off with a some type of Federation, or some type of Single Sign On, it will make your lives so much better.

Jesse: And again, going back to the conversation about AWS Organizations and creating these cookiecutter accounts, one of the things that you can add to these templated AWS accounts that you later create for your different brands, or teams, or products, you can add all of this federated access built-in, baked in when you create the account so automatically you have all of the specific leverage points that you need for federated access without going into every account individually and creating a bunch of roles and enabling a bunch of feature flags or services.

Pete: So, those are generally the core accounts you're going to create, not all-inclusive. There might be things that are specific to your business or organization. But, Jesse, what do we do now? Okay, we've got our core accounts. I've got my SaaS app here; where do I go with this?

Jesse: This is always the age-old question where, “What do I do now?” And the answer is my favorite answer of, “Well, it depends.” It really depends on what you ultimately want to do with your different business units. It depends on what your business units are. And those business units could be as broad as different products, or particularly different actual business units or parts of the organization that are using AWS in different ways, or it could be as small as different components of a single product, like different microservices or different teams that are leveraging AWS in different ways. 

Start thinking about how does your organization look at the different components of your product or products, and start breaking that down. Is it a single brand with a single product? Is it multiple brands with multiple products? Start breaking down that tree, start breaking down that list into all the smaller components, and look at that as the list of accounts that you ultimately want to create. So, for example, you may have two products and you'll want to create different accounts for those two products, number one, but number two, you might also want to create different accounts for different deployment environments. 

So, for example, you may want Product One to have a development account and a separate production account. You may want Product Two to have a separate development account and production account. So, start thinking about that kind of binary tree, almost, which I shudder to even bring up in this day and age. But start thinking about that kind of tree of different business units, and identities, and ways that you slice and dice the components of your engineering organization, and start thinking about that for your linked accounts. 

Pete: Yeah, I mean, I think you could also, too, take those—maybe their brand accounts, their product accounts—definitely don't get too granular here because you could really, kind of, shoot yourself in the foot going out in the future. But you want to have some level of separation here to make it easy to not only attribute those costs but to, kind of, separate those workloads. And maybe you say to yourself, “Well, I just really need to know that Prod spend is different than Dev spend.” Then great. Then that's the solution for your organization. 

Again, none of these really preclude you from moving and being more segmented in the future; you could create, let's say, a new product initiative, deploy that into a brand new account. Now that one brand new product is in its own Amazon account for cost attribution reasons. I can't tell you how many times I've been asked at certain stages of a startup’s life that I've worked at, “What is the cost of certain services to us?” Or, “How much is a certain client costing us?” And when you get these questions, then you need to start diving into, kind of, per product spend. 

But one thing that I think more businesses should be doing is to answer that question is, what is the cost of these features? Being able to go to your product teams, or being able to arm your product teams with this information for products and features within an application, for example, to help them drive their decisions on, “Well, if this one feature is used by 1 percent of our clients but represents 30 percent of our spend,” that should be a decision point, a decision process to, “Do we keep it? Do we not keep it? Do we refactor this?” Do something with it, right? But if that spend is just locked up, what do you do? You don't even know that that's happening.

All right. Well, Jesse, I think we gave a good intro into how to create your accounts. What if I'm on Amazon? I mean, is it worthwhile for me to start pursuing some of these strategies, even if I have a ton of technical debt in my infrastructure?

Jesse: Yeah, I absolutely think it is. I think that there are very small steps that you can make over time that will absolutely help along this route, even if you have existing technical debt, even if you have other things that you are concerned about, even if you are in a quote-unquote, ‘brownfield space,’ there is still absolutely opportunities to start moving the needle on this work with cost attribution and cost allocation.

Pete: Yeah. So, don't think just because you've been on Amazon Web Services for a decade and you're kind of locked in your ways, that you can't improve what you have. You can create new accounts, you can start creating new accounts, and start migrating services into those accounts. Again, it's not going to happen overnight, but a concerted investment over a period of time will absolutely net benefits in the future. 

So, do you have a question for Jesse and myself to try to answer or at least stumble our way through a podcast episode? You can now ask us via a website at lastweekinaws.com/qa. And QA is not short for ‘quality assurance,’ as you might think, that is short for ‘question and answer.’ But because we are all former sysadmins and technical operators, we're pretty lazy and we don't like to type out very long things. So, we've just shortened it to ‘QA.’ So, go to lastweekinaws.com/qa enter in your question, but more specifically, what kind of questions do we want to focus on, Jesse?

Jesse: Yeah, so obviously, this is a podcast about AWS, so the question should be scoped to AWS. Really past that, we try to focus on questions related to cost optimization and optimization of AWS services in general. I particularly have a very soft spot in my heart for a lot of the more qualitative conversations, but I think a lot of the content that we cover on this podcast is more quantitative, so I think any questions related to AWS architecture, things that you might want us to cover in the future, specific questions around? “How do I…” with cost allocation or cost attribution or anything related to cost optimization, we are happy to answer.

Pete: Absolutely. So, head over there, fill out the form, ask us a question, and yeah, we'll do our best to answer that on one of our future podcasts. If you enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review, give it a five-star rating on your podcast platform of choice, and tell us what your most impressive account structure at your business. Do you have just one account for everything? There's nothing wrong with that. Thanks again. Bye.

Announcer: This has been a HumblePod production. Stay humble.