In case you haven’t run across it yet, the Federal Trade Commission has asked the public to weigh in on cloud computing providers’ business practices. Given what I do both as the Chief Cloud Economist at my consulting firm, The Duckbill Group, and author of the delightfully sarcastic Last Week in AWS, I have Some Thoughts™. While none of this is likely to be particularly new to my longtime readers, it occurred to me that I’d never consolidated a lot of these points in one place before.
From the FTC’s Request for Information:
The staff of the Federal Trade Commission is inviting public comments about the practices of Cloud Computing Providers and their impact on end users, customers, companies, and other businesses across the economy. FTC staff is studying a wide array of issues related to market power, business practices affecting competition, and potential security risks. Staff is also interested in cloud computing with respect to specific industries, including but not limited to healthcare, finance, transportation, eCommerce, and defense.
The RFI has 20 questions addressing security risks, market power, and business practices. I’m only going to address the ones that I have the most knowledge and experience with, and also are the most interesting to me; that’s what you get when I’m volunteering!
Before we get into things, as a reminder, Duckbill is not a partner of any cloud providers or other vendors in this space. While I do receive compensation for voice-of-the-customer feedback from AWS and Google (see my disclosures page for more details), my perspective is my own, and this article was not shared with any cloud providers in advance.
Question 2: Insufficient resourcing
Do cloud providers continue to invest sufficient resources in research and development? In which areas are cloud providers investing in most heavily, and why? Are there areas in which cloud providers have not invested sufficient resources, and if so, why?
Google Cloud and Azure both have what’s termed a “global control plane,” while AWS has a hard isolation boundary between its various regions. This global control plane has led to a number of notable global outages for both Google Cloud and Azure, but as of this writing, there has never been a global AWS outage that spanned multiple regions. The corollary to this is, of course, that it’s a lot harder to work coherently with multiple regions. For virtually every AWS service, developers are effectively using entirely different AWS accounts when working between regions.
It’s worth pointing out that AWS at least has publicly committed to having different availability zones (AZs) located in different physical facilities, separated by a minimum distance to limit the likelihood that an entire cloud region would go offline. The marketing pages for Google Cloud and Azure make no such explicit assurances — leading to what are periodically embarrassing revelations.
It’s not clear why Google Cloud and Azure chose to build on top of a global control plane (though I strongly suspect the answer here is “convenience,” both for the people building it as well as its customers), but it is known that the practice of building per-region control planes is substantially more expensive in engineering time. Personally, if a cloud provider is going to invest anywhere that’ll lead to meaningful differentiation along the axis of durability, I think it’s got to be in service of fault isolation and resilience.
Questions 5 and 6: Cloud provider contracts and negotiations
To what extent are cloud customers able to negotiate their contracts with cloud providers? To what extent are customers experiencing take-it-or-leave-it standard contracts? How does the ability to negotiate depend on the size of the customer, the sensitivity of the data, the purpose for which the data will be used, or other factors?
What incentives are cloud providers offering to customers to obtain more of the cloud services they need from a single provider? Are cloud providers linking, tying, or bundling their cloud services with other services?
Let me begin with an aside: What I’m about to say here is widely known in the community and may well strike some of you as being overly basic. That’s very intentional; I don’t break confidentiality, and part of doing what I do as a consultant is internalizing a lot of what I see across the industry without “showing my work” as to how I got there.
Now then, let’s start with some background context on AWS contracts!
AWS customers generally become eligible for contractual discounts (dubbed “Private Pricing” by AWS) when they hit a $1 million/year run-rate on their spend. As it’s run-rate and not a hard monthly threshold, some customers growing organically will talk to their AWS account manager before they’re spending $83,000/month.
The initial entry point for an AWS contract is the AWS account manager. This is a fancy word for “your account rep who also works on commission.” That said, they don’t own the contract process or terms. (Holy moly, can you imagine ANY company where sales was able to unilaterally set contract terms?! I’ve worked in sales and let me tell you: It would be an unmitigated disaster.) That part of the process is handled by a specialized group inside AWS. In many ways, the account manager’s hands are tied, and they have no control over the terms a customer receives.
So, here’s what you can expect as a cloud customer. Note that this is the common case; there are going to be exceptions to everything when you get to AWS’s scale. If this doesn’t align with what you’ve seen, congratulations: You’re an outlier.
When your annual spend with AWS hits about $1 million/year run-rate, you receive outreach from your account manager, who suddenly has far more time for you than they once did. They offer what’s varyingly termed an Enterprise Discount Program (EDP) or a Private Pricing Agreement (PPA), depending upon the phases of the moon. In return for committing to spend certain amounts of money that start at $1 million a year and tend to grow by a usual 20% per year (though there are a variety of ways to negotiate this) for a fixed number of years, you receive a percentage discount. You are almost always required to upgrade to “Enterprise Support,” which starts at $15,000 a month and scales upward as a tiered percentage of your AWS bill, and to switch to invoice payments, which means you are no longer able to save up your American Express credit card reward points for a trip to Mars.
Your options to negotiate the EDP/PPA terms are extremely limited at this scale. You’re still a very small fish in a very big ocean, and AWS has been bursting at the seams for years as it tries to scale these functions.
Your spend grows and grows and grows. If it grows faster than your contract accounts for, you may get a mid-cycle renegotiation offer, or you can ask for one. If there’s a shortfall (read as: you’ve committed to spend more than you’ve actually spent), you’re contractually obligated to pay the shortfall to AWS — though there are publicly citable examples of AWS slightly increasing the overall commitment amount and extending the term of the contract to give you more time to reach the agreed-upon level of spend. That said, I would absolutely not count on this. “We’re depending on Amazon to be kind-hearted human beings” is an adorable and not particularly defensible position to take.
Historically, 50¢ of every dollar you spent on the AWS Marketplace (where third-party companies sell offerings to AWS customers, and AWS gets a percentage) counted toward your contractual commitments. At the start of 2022, this was expanded to 100% of your spend on new contracts, up to a maximum of 25% of your annual commitment figure. This has the result of incentivizing customers to transition existing vendors to the AWS Marketplace, very often to the third party’s annoyance.
In short, AWS controls increasingly more of the contracting process with your other vendors, which, of course, happens on Amazon’s terms.
As an aside, let me be crystal clear on this point: AWS absolutely does not want you talking positively about your experiences with its other large service provider competitors. Don’t believe me? Find a large reference AWS customer and try to get them to do a case study for Google, Microsoft, or Oracle’s cloud offering. It’s basically impossible. You’ll note that those companies’ offerings aren’t on the AWS Marketplace.
Question 10: Porting data between cloud providers
What barriers (e.g., contractual, technological, or other), if any, exist to offering services that compete with individual services offered by cloud infrastructure providers (e.g., hosted databases, Content Delivery Networks, etc.)? What costs do cloud customers face in:
a) switching software services?
b) using multiple cloud providers?
c) porting their data from one cloud provider to another? How important is data portability to competition and to entry?
The biggest obstacle to porting data is the pricing issue among the three major cloud providers around data transfer. Generally, data ingress (data transferred from the internet to the cloud provider) is free, while data egress (sending data outward) is billed at what frequently amounts to exorbitant rates.
While AWS has had multiple executives tell stories about how this is “a reflection of their own cost structure for data transit,” it’s worth pointing out that this data transfer pricing paradigm persists even when transferring data from AWS to other providers via mailing the data on storage devices. Using those devices to send data to AWS incurs no per-GB fee, whereas sending data from AWS outward on those devices adds on a per-GB surcharge.
Twitch, a video-streaming service that Amazon acquired for nearly $1 billion about a decade ago that has recently pivoted to antagonizing the creators who make that platform successful, serves as a stark example of just how egregious these egress fees are. Back when AWS announced its Interactive Video Service, I conducted an analysis using Twitch’s public statistics. It demonstrated that if Twitch were using Amazon’s IVS to serve its own platform at retail rates for data egress, within one year, AWS’s cost for those services would have been roughly 10 times as much as Amazon paid to acquire all of Twitch.
Moving data, particularly out of the cloud providers, is absolutely an economic nonstarter for many customers. As a result, migrating between cloud providers with significant amounts of data is rarely done.
Lastly, it’s not just the cost that’s the problem — it’s the complete lack of clarity around what sending data between any two points is actually going to cost. I had to run a bunch of experiments to come up with something that even starts to work as a visual model for this, and it’s not for the faint of heart:
In practice, people run their application at small scale, see what it costs, determine whether it’s acceptable, and proceed accordingly. Everyone takes their AWS data transfer bill on faith. For what it’s worth, despite how labyrinthine and confusing the pricing is, I’ve yet to discover AWS making an error on customer data transfer bills in their own favor. It’s not malicious, it’s just confusing.
Final thoughts on cloud provider business practices
Cloud providers once focused on customer retention via innovation and customer satisfaction. Today, they have enough deterrent business practices in place to make switching providers an expensive Herculean undertaking. Fundamentally, there’s enough differentiation at higher levels of service to create an effective form of customer lock-in. At the lower levels of service, there are other barriers such as idiosyncratic security and identity abstractions, expensive data egress fees, and contractual disincentives.
It seems that there’s been a steady shift toward making migrations painful enough along various axes that customers choose to continue on with their existing providers. Is this intentional, or an organic shift? I can’t say. I can say that the various cloud providers start to look a lot more anti-competitive with every passing year, and I worry about what this portends for the state of the internet in another generation or so.
