How This Chief Cloud Economist Got Nailed With an AWS Surprise Bill

It’s been a long while since this happened to me, but I recently had the ~pleasure~ of being surprised by an AWS bill I wasn’t expecting.

For those who may not be fully aware of my day job, I’m the Chief Cloud Economist at The Duckbill Group. For the last five years, my world has revolved around AWS billing intricacies and sorting them out for a wide variety of fascinating clients. In other words, you can dispense with the traditional Children-of-Reddit-style admonishments to set up billing alarms and budgets, and the accusations that I have no idea what I’m talking about.

You can safely assume that for me to be surprised by an AWS bill, something very strange must have happened. And frankly, that’s something I find very concerning.

The surprise bill culprit: Amazon SageMaker Canvas

At re:Invent last year, AWS announced Amazon SageMaker Canvas. If you look at that blog post, it says absolutely nothing about a pricing model, but that’s OK.

I spun up a SageMaker Canvas job in mid-December, played with it for roughly 45 minutes, and then turned it off — or so I thought.

Delving into my inbox, I found a notification that I skimmed and promptly disregarded from AWS back in April that said, in part:

Until March 2022, session charges were not enabled. As a result your use of Canvas only incurred model training charges. Starting April 2022, you will see session charges for SageMaker Canvas usage. For more information see the SageMaker Canvas Pricing page.

I had ignored this, as I obviously wasn’t still running a SageMaker Canvas application; I’d deleted it. This was obviously intended to let me know that going forward, I’d be charged in a different way should I reuse the service at some point.

Time passed.

In late June, I started getting a warning that my SageMaker usage was approaching the limits of the free tier. THAT was something that I didn’t ignore! I logged into the relevant account, played hide-and-seek with the bill, and eventually found a “SageMaker Notebook Run Instance” in Ohio. A thorough study of the notebook section of the SageMaker console revealed no notebook, so I did the responsible thing and opened a ticket with AWS support

To support’s credit, they escalated to the SageMaker team and responded relatively rapidly with the correct answer, but I never saw it. A mail filter rule of mine ate their response. That one’s entirely on me.

The next notice I got was from my own internal billing monitors six days into July, as the bill for this thing churned past $260.

To say I was perplexed would be an understatement. After some digging on my part and some explaining on AWS’s, I now know what went wrong.

At launch, SageMaker Canvas only charged for actively running training jobs. This is what I killed after 45 minutes in December. However, I didn’t log out of my Canvas “session.”

Starting at the end of April, AWS added charges of $1.90 an hour for every “session” in Canvas. I got an additional two-month grace period, since that’s the two-month free tier the first time you spin up SageMaker resources. That kicked the can into July, when my open “session” began accruing charges.

As a result of these six days of sneaky session charges, AWS managed to reduce my stash of AWS credits by roughly $260.

What makes SageMaker Canvas’ billing different

This “session” billing dimension is, to the best of my knowledge, unique among AWS services. I’m used to starting an instance or container and being billed until I terminate it. Never before have I been billed for a “session” that did not either:

• Auto-terminate, as in Amazon WorkSpaces’ ability to shut down after half an hour of inactivity by default.
• Conclude once my AWS console session had timed out or been logged out of.

Further, unlike virtually every other AWS service, there is no way to see the resource for which I was being charged in the “normal” AWS console. I had to explicitly log into the SageMaker Canvas separate console to see any of this. From within that console, it was also not at all obvious that there was a running job — because there wasn’t. There was simply a session that had not been logged out of, which was being billed as a running notebook. The notebook list was empty.

Time to burn them to the ground

Mistakes that eat AWS credits aren’t able to be granted concessions (read as: refunds), so my $260 in credits are now forever lost to me.

This is fine. I’m obviously not salty. But in return for having fallen into the SageMaker team’s cunning trap, I come instead with $260 worth of uncomfortable questions for them about this entire experience.

1. My IAM user session times out after an hour. This is annoying, but I’ve learned to live with it. Why does a Canvas “session” not time out apparently ever? Did someone not flag the overloading of the term “session” during the pricing discussions?
2. It costs $2,000 to leave a Canvas “session” running for a month. Were I a student or individual learner on the hook for this charge, I would’ve been better off had I left the oven on instead of SageMaker. I mean, at least I can get insurance for house fires, and the deductible would easily be less than two grand. Do you have any plans to offer AWS Bill Overage Insurance?
3. Do customers using this service routinely have “sessions” that last the better part of a year? To my mind, seeing a user session last that long means I should probably call that user either an ambulance or a coroner.
4. Why was a service launched to General Availability at the end of November, but a significant pricing dimension not billed until the following April?
5. At any point when deciding to start charging the $1.90 per hour session fee, did someone suggest, perhaps, stopping any existing multimonth sessions as of the onset of that fee, unless a customer explicitly opted out of that? Presumably, you must’ve placed that Amazonian on a PIP instead of promoting them for their customer obsession and empathy if so.
6. Is tricking customers into paying fees that even a Cloud Economist didn’t see coming a new AWS operating practice?
7. At $2,000 a month per “session,” does this mean that AWS was being incredibly generous in letting me run that Valuable Computing Resource to the tune of $14,000 before kindly asking me to start paying my own way, or is the service instead astronomically priced to the point where it bears little if any relation to AWS’s underlying cost structure? If the latter, is this an attempt to price based upon value? If so, don’t you think you should try delivering actual value before pricing it?

To be clear, none of those questions is rhetorical. I expect answers.

SageMaker Canvas’ billing is next-level predatory for AWS

I always empathize with the usual billing panics and have sympathy for customers who get caught in strange billing edge cases. But this particular pattern takes AWS billing to an entirely new level of predatory. When I’m falling victim to it, I worry about who else out there is too.

If this is going to be AWS’s path going forward, I’d like to know so that I may plan accordingly.