To say AWS has “a lot of services” is like saying the Pacific Ocean is “slightly damp.”
In case you’re somehow reading this but have been asleep under a tree for the last 15 years of cloud computing, look no further than my second list of 17 ways to run containers on AWS as an example.
The array of services is dizzying and complex. Perhaps worst of all, AWS displays no meaningful sign of slowing down any time soon. Why is this?
The clear-cut answer, at least from my perspective: The people who launch services accrue fame, glory, and promotions.
How a company’s culture can drive its output
As a general rule, companies hire smart people who make good decisions based upon the context they operate within. It follows that internal structures at Amazon must incentivize this sprawl of services that seems to never end.
Let me be clear: Google also clearly incentivizes launching things. A recent tweet describes how launching services at Google is necessary to be promoted. How individuals are rewarded has a whole lot to do with how they behave in the workplace. One outcome of running an organization the way that Amazon does it is that individual managers have tremendous sway over their teams, from how they operate to what they work on. Ultimately, each manager is only formally accountable upward. The higher a manager climbs, the bigger their potential blast radius becomes.
The difference between Google’s and Amazon’s launch practices is that, over in Mountain View, maintaining or improving an existing service apparently earns nothing more than a pat on the head. That can explain Google’s particularly ruthless form of product garbage collection.
Google clearly sees little, if any, issue forcing its customers to migrate to avoid losing fingers and toes to a service deprecation. Meanwhile, AWS has demonstrated over 15 years that APIs are promises; outright deprecations of APIs are very rare. As for AWS service deprecations, there have been two to my knowledge:
Amazon Sumerian. The page for Amazon Sumerian states that it’s no longer accepting new customers and is working to transition customers off of it.
RDS on VMware. The feature release update for RDS on VMware links to both a product page and a blog post that simply redirect to the generic RDS landing page. That one’s been pretty clearly memory-holed.
While clearly incentivizing new service launches, AWS also has some kind of structure in place that all but guarantees that services will also see continued ongoing investment.
The challenge here is that without a periodic culling process, the explosion of services becomes a problem in its own right. New service launches were great back in the days when there were a couple of dozen AWS services. However, the current state of things means that we’ve long since crossed a Rubicon wherein I can talk convincingly about services that don’t really exist and not be called out on them by AWS employees.
The seams are showing: Repetitive AWS products
The lack of internal coordination is easily observable from the outside by the number of services that clearly duplicate functionality; they simply exist within different parts of the organization.
First AWS Systems Manager Parameter Store launched, then AWS Secrets Manager did nearly a year and a half later. They’re mighty close to one another in terms of functionality — but Parameter Store is free, whereas Secrets Manager costs 40¢ per month per secret.
Had there been an overarching, cohesive view of the roadmap and who was shipping what, it seems it would have been remarkably hard to greenlight both of these.
A more recent example is the launch of Amazon CloudWatch Evidently for feature flagging, when “feature flags” was already a launch use case for AWS AppConfig. I’m even more confused by the subsequent launch of AWS AppConfig Feature Flags, because now it’s not even slightly clear to me what the delineation is between these offerings.
A third, even more recent example stems from the existence of both Amazon WorkSpaces and the AWS Cloud9 IDE. Despite those offerings, the Amazon SageMaker team recently shipped a blog post talking about how to run code-server within SageMaker.
This all leads to a sense of haunting familiarity, a ghostly echo of, “Haven’t I seen this capability somewhere before?”
“Let a thousand flowers bloom” may have been a fantastic perspective when AWS had only a few dozen services. It now features hundreds of options in its management console, and the opposite problem has come around.
When there are plausibly a half-dozen AWS services that can be used to achieve an outcome, what’s the best path forward? Guidance in that respect is sorely lacking. I’m left with the sense that AWS product managers are just as confused as we are.
I don’t know what future lies in store for today’s DBAs, but I don’t think any of them want a world where their primary job function is deciding which of AWS’s 40 managed database services is the right fit for a given workload.
The value of free services
One of the problems with having every service rolling up to managers looking for glory is that it invariably pushes every service to turn a profit.
To my mind, there’s no earthly reason that AWS Config should charge per rule evaluation. Tiny yet highly dynamic environments get charged a lot in Surprise Bills (which they can’t opt out of if they’re using Control Tower to manage their AWS Organization), while large, stagnant environments spending huge money on EC2 instances that they treat like an extension of their data centers get charged nothing by Config.
There’s a strong argument that an awful lot of services cost money when their core value should be built into the offering itself. For example, GuardDuty, Detective, and Security Hub all watch your environment for signs of bad actors. Isn’t that the sort of thing that cloud providers should be offering as a core part of the offering to differentiate themselves from data centers? Apparently not.
Particularly in recent years, every higher level service seems to come out of the gate with an eye toward “being run at a direct profit itself.” If they recognized that everyone at AWS is (presumably) on the same team, then maybe they could see that a “free” service can drive adoption of more EC2 or S3 … which means the whole company comes out ahead.
The corollary to this is that if you make a great service, the price becomes almost irrelevant to customers. Be forewarned, AWS product owners using this paragraph to make an internal point: “Great” is defined by the customer using it, not by you. And from what I’ve seen, we customers have higher standards than many of you do.
Brace yourself for re:Invent 2022
No time is a better reminder of this launch dysfunction than re:Invent season. AWS’s never-ending stream of product launches is nothing more than the product of its company culture and reward systems for managers. But until it realizes that its spread of offerings has stretched beyond what’s helpful, even into repetitive territory, customers won’t be clear on the best tools for their workloads. The internal cracks are showing. Speaking as a customer myself, someone needs to bring order to these warring factions.
