AWS has been very publicly insecure about the perception that it’s lagging behind in the Generative AI space for the past year. Unfortunately, rather than setting those perceptions to rest, AWS’s GenAI extravaganza at re:Invent 2023 seemed to prove them true.
Of the 22 GenAI-related announcements, half of them are still in preview. Many were seemingly developed in crash programs launched since ChatGPT was released a year ago. All distracted AWS from resolving more relevant challenges that customers are facing.
Amazon Q: Quartermaster or omnipotent being?
Nowhere is this more evident than in Amazon Q (currently, you guessed it, in preview).
In case you’re blissfully unaware of the re:Invent announcement, Amazon Q is a catchall brand (similar to GitHub Copilot or AWS’s own SageMaker) that encompasses coding assistance, a chatbot in the AWS console, a chatbot on AWS’s marketing website, a natural language processing interface for QuickSight, a version for AWS Supply Chain, and several more. It’s hard to pin down just which incarnation of “Amazon Q” someone might be talking about (itself a significant problem!), but here I’m focused on the chatbot available on the website and inside the AWS Console.
The Amazon Q chatbot is the equivalent of Microsoft Clippy without the personality, but add a traumatic brain injury. It now shows up on every AWS page load. It periodically opens a giant chat window that obscures content and requires multiple mouse clicks to minimize. It’s an omnipresent icon in the lower right of the page that cannot be removed short of using a browser extension.
However bad you might guess Amazon Q is, the hour I spent plying it with questions demonstrates that it’s oh-so-very much worse than that.
Amazon Q’s … challenges
In one short session playing with Amazon Q, it has given me false information about:
historical AWS price hikes:
how to configure AWS services:
unannounced upcoming region launches:
how AWS contractual discounts work (which, by the way, AWS doesn’t even publicly acknowledge exist):
which services see dismal customer adoption:
as it releases wolves onto the streets of Seattle
… and so many other things that AWS employees would never ever tell me, even under the strictest of NDAs.
In fact, if an AWS employee were to answer my questions the way Amazon Q does, I imagine Andy Jassy would parachute into the room to personally fire them on the spot. Instead, AWS has given Amazon Q a place of prominence on every webpage and promoted it heavily.
Lest you think I’ve somehow subverted the service to grind an ax, let me give you a poignant example. This image shows a very reasonable request that a customer might ask and a very questionable response from Amazon Q:
that went into crafting this question.
A few things to note about Amazon Q’s answer:
1. The specific instance recommendation is incorrect. If cost is your primary concern, r7iz instances are not where you should be looking.
2. The general sweep of instance recommendations is incorrect. If you want to save money per node, that’s one of the strongest selling points for AWS’s Graviton processors, which AWS has been heavily promoting for several years.
3. This image is a screenshot that was posted to the official Amazon Q launch blog post.
After I pointed these flaws out to AWS, it silently updated the blog post image to this version:
updated to “Max 1000 characters” as the team
frantically polishes the service post-release.
Note that the AWS team still explicitly has to tell Amazon Q that it wants Graviton instances — something that customers are not likely to know that they have to ask for. To its credit, Amazon Q correctly notes that r6g instances are less expensive than the newer r7g instances, which amusingly is an observation that AWS has been adamant about challenging on the grounds of price for performance.
My own business partner, thinking I was simply exaggerating for effect, made some legitimate attempts to communicate with Amazon Q himself, with now-predictable results. And he’s not the only one, either. Other people have gotten incorrect answers about SES, the history of AWS services launches, and Amazon Q refused to compare AWS to Azure compute. There’s also this gem, where Amazon Q makes up all sorts of BS about my company, The Duckbill Group!
piece of information.
Perhaps even more worrisome is that re-running the query yielded completely different results!
I know there’s a small army of people who worked on these releases. So before you think I’m punching down: I believe AWS’s headlong rush to GenAI, dropping the ball in the process, is a violation of its own leadership principles, not the fault of individual employees on the team. The company failed them; they didn’t fail the company. In the process, AWS has failed its customers. We’re all worse off for this.
On the plus side, given its prominent placement and positioning, it sure does seem like I should be able to take everything Amazon Q says as an official statement on behalf of AWS, which is actually an upgrade over Amazon’s own public relations. It’s truly an official AWS spokesmodel (hat tip to Forrest Brazeal for coining that amazing turn of phrase).
ChatGPT still reigns supreme
Keith Mattix has observed that customer expectations for chatbots such as these are now sky-high thanks to just how great ChatGPT is. Three years ago, we might have all dismissed Amazon Q as another silly, overly-simple chatbot attempt. But ChatGPT changed the world, and Amazon Q makes it look like Amazon didn’t notice, didn’t care, or shipped it anyway.
In fact, that’s what makes all this so galling: For each of these examples, ChatGPT answers the same prompt correctly. It’s clearly possible to solve the types of problems that Amazon Q is struggling with — but those solutions take significant time and investment, plus an incubation period that may not align with Amazon’s annual conference.
AWS made a choice here. In so doing, it has cemented its reputation for now as a company that cannot deliver on Generative AI at anything above an infrastructure level (but happily tries to mislead its customers to believe otherwise).
GenAI lies outside AWS’s current competency
I’m somewhat sympathetic to the position that AWS found itself in that led to this point. There’s been pressure from market analysts to demonstrate that they along with everyone else are in fact, generative AI companies.
The problem is that from where I sit, AWS simply isn’t good at these sorts of large-scale, cross-functional, higher-order problems. The things AWS is excellent at all revolve around innovation of infrastructure.
Run the GPUs required for other companies’ GenAI plays, improve the data storage services and the network to make that better … but as soon as AWS attempts to move up the stack into the application space, the wheels fall off in major ways. It requires a competency that AWS does not have and has not built up since its inception.
It leaves AWS stuck in the unenviable position of being the rails atop which the actually exciting companies in this space ride. Amazon seemingly either cannot or will not accept this, and I do get that; the ultimate fate of “plumbing” companies is to become like today’s Tier 1 backbone providers: incredibly important to making everything else work but largely invisible to most people. More than attention, the problem for these companies is that it’s the folks building on top of them that capture the lion’s share of the value delivered.
As a result, Amazon apparently torpedoed what could have been a great 2023 product roadmap to focus on things that, frankly, are not good and don’t advance AWS’s reputation. Amazon can talk as much as it wants about how it’s been doing AI and ML for the past 20 years, but it’s obvious that everything AWS announced in the GenAI space last week was less than a year in the making.
An AWS spokesmodel declined to comment for this article
Amazon’s prepared statement
Shortly before this article was published, presumably in response to Platformer’s reporting of Amazon Q leaking confidential information, AWS sent us this prepared statement.
“Some employees are sharing feedback through internal channels and ticketing systems, which is standard practice at Amazon. No security issue was identified as a result of that feedback. We appreciate all of the feedback we’ve already received and will continue to tune Q as it transitions from being a product in preview to being generally available. Amazon Q has not leaked confidential information.”
