Whiteboard Confessional: Google’s Deprecation Policy

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Corey: Normally, I like to snark about the various sponsors that sponsor these episodes, but I'm faced with a bit of a challenge because this episode is sponsored in part by A Cloud Guru. They're the company that's sort of famous for teaching the world to cloud, and it's very, very hard to come up with anything meaningfully insulting about them. So, I'm not really going to try. They've recently improved their platform significantly, and it brings both the benefits of A Cloud Guru that we all know and love as well as the recently acquired Linux Academy together. That means that there's now an effective, hands-on, and comprehensive skills development platform for AWS, Azure, Google Cloud, and beyond. Yes, ‘and beyond’ is doing a lot of heavy lifting right there in that sentence. They have a bunch of new courses and labs that are available. For my purposes, they have a terrific learn by doing experience that you absolutely want to take a look at and they also have business offerings as well under ACG for Business. Check them out. Visit acloudguru.com to learn more. Tell them Corey sent you and wait for them to instinctively flinch. That's acloudguru.com.

Corey: Welcome to the AWS Morning Brief. In lieu of the Whiteboard Confessional’s traditional approach today, I want to talk about a different kind of whiteboard issue. Specifically the whiteboard interview you wind up taking at Google, which is just a giant red herring because the real question is “How well did you erase the whiteboard afterwards?” so it aligns with their turning stuff off that people love policy. I'm joined this week by my colleague, Pete Cheslock from The Duckbill Group. Welcome, Pete.

Pete: Hello.

Corey: So, we're talking today about Steve Yegge’s article that went around the internet three times over the weekend, and it was titled “Dear Google Cloud, Your Deprecation Policy is Killing You.” Normally, you would think that that would be some form of clickbait headline, but it's not. It was a massive 23-minute long read, as per Medium. And we will, of course, throw a link to this in the [show notes]. But, Pete, what was your take on this thing?

Pete: Well, I missed it on the first go around, but when you sent me over the link, and the first thing I saw was Medium saying, a 23 minute read, and you had told me how this post had blown up. I think that really speaks for how incredibly well written this post is about this particular issue, that people in this world are willing to invest 23 minutes to read it. I was locked into it the whole time. It held my attention the whole time because of just how deep it went into Google and just how they operate.

Corey: Steve Yegge is famous for doing the platforms rant back in 2012 or so. He's a former Amazon employee who I think spent something like seven years at Amazon, about an equal time at Google, left to go run architecture at Grab a couple of years back, and then, due to these unprecedented times, is now independent/doing his own thing right now. So, that is an absolutely fascinating trail because when he writes about this stuff, he knows what he's talking about. This isn't one of those, “Eh, I’m just going to go ahead and pen something that's poorly articulated, and see what happens.” What's more amazing is I haven't seen much in the way of pushback on this. The points that he hits in this article are pretty damning, and even people from Google are chiming in with, “Yeah, that tracks.”

Pete: Yeah, and for all those listening that maybe haven't read this yet, maybe going to go read it after listening to this. What the real, I guess, crux of this post is about is how Google aggressively deprecates things and the kind of culture within Google that really drives that world to happen, and how just opposite it is to a company like Amazon. I think my biggest takeaway from this was this light bulb, “Oh my goodness, it all makes sense now,” idea of how aggressively Google deprecates things has to do with code cleanliness. They don't like five different APIs to do the same thing, so they'll deprecate four of them and keep things clean and whatever. And what I think is really interesting, too, that I read in here is how internally, this works great for Google Because they have all these tools that can automatically update code, and update APIs, and let people know if a deprecation is happening. But he compares this to, like, Java and Emacs, which historically, take decades—if ever—fully removing APIs. It was a really fascinating read.

Corey: It really was and one thing that stuck with me was, it makes perfect sense in hindsight. If you are Google and can dictate how all of your employees write software that makes it into production and have automated tooling to go back and handle deprecations for you, then great, that does work. The problem is, is that the rest of the world is not like your internal engineers. The problem I see behind Google Cloud, by and large, is that it assumes that everyone tends to write software the way that Google engineers would. That's not a valid assumption, I assure you. I write software that is nothing like anyone who calls themself a software engineer would ever write, but your cloud offering has to support my nonsense.

Pete: Right. Compare this to Amazon. So, that's one of the biggest other takeaways that really hit me. And this came up when I think I was looking at a cloud bill and seeing SimpleDB still on it. SimpleDB, which they don't really market it, they don't tell you to use it, it's not part of design things. Although, Corey, you can correct me if I'm wrong there, if it's included, if it's really talked about much anymore, I don't think it is—

Corey: No, they've tried to bury it, but they are still hiring for that team from time to time.

Pete: That's—yeah, I remember you had mentioned that. And so think about that. Think about the m1.medium, right. I think m1.medium was the first instance type back in 2006—

Could you imagine if Amazon deprecated something related to Lambda or DynamoDB? I mean, it would be catastrophic for organizations to have to make these changes to their codebase.Corey: It was either m1.medium or m1.large. I don't recall.

Pete: It was that m1 class, both of which you could run today. I've definitely seen instances of recent launches of that class of instance, and, to your point, SimpleDB they're hiring for, and you can provision that in places. They just—Amazon does not turn anything off in their cloud, and that's huge. And to Steve's point in his post, he doesn't have time to go through and migrate all his code to the next new great thing. 

This episode is sponsored in part by our good friends over at ChaosSearch, which is a fully managed log analytics platform that leverages your S3 buckets as a data store with no further data movement required. If you're looking to either process multiple terabytes in a petabyte-scale of data a day or a few hundred gigabytes, this is still economical and worth looking into. You don't have to manage Elasticsearch yourself. If your ELK stack is falling over, take a look at using ChaosSearch for log analytics. Now, if you do a direct cost comparison, you're going to say, “Yeah, 70 to 80 percent on the infrastructure costs,” which does not include the actual expense of paying infrastructure people to mess around with running Elasticsearch themselves. You can take it from me or you can take it from many of their happy customers, but visit chaossearch.io today to learn more.

Corey: Exactly. He starts off with an anecdote when he was a Google employee, where he'd spun up some big table instance and got an email from an internal team saying you're running a very, very old binary in this data center, and it was years after he'd spun this thing up. And it was, we'd like to help you upgrade to the latest version, or migrate your workloads. Let us know what we can do. That's the kind of outreach you get when Amazon eventually decides to deprecate an AWS service.

So, what I like about this entire rant, as it were, was that he's speaking from the perspective of a current GCP customer, and this stuff is driving him up a wall. It doesn't make the ‘Killed by Google’ lists. But the old version of a service API is being deprecated in favor of a new one, so you have to go and patch your code is apparently a very common GCP problem. And yeah, I look at some of the stuff that I spun up in my account back when I was first learning Lambda in 2016, and sure, it's running in deprecated runtimes, but it's also still running. If I want to update anything, then I've got to change the runtime, and in almost every case, I'm not using advanced language features, so it's fine.

But these things are still sitting there, just working. I have heard stories from friends at AWS accounts that have been there since, basically, Bezos was in diapers. And when they're using things that wind up getting deprecated, it's an increased series of very polite, very solicitous emails, that if they continue to fail to respond to results in a personal outreach from an engineer to, “Hi, we really need to turn that thing off. Can you please work with us to help migrate on to something else? We'll help you do it.”

Pete: Yeah, I think the other point that he really touched on about, you know, it really comes down to backwards compatibility. When he talked about Java and its historical support for old APIs, deprecated API notices when you launch an application, and even to the point I think he talked about the video game, the game that he has been working on for a couple of decades now still runs. He has not had to modify things; he's using specific deprecated APIs. But he called out specifically the great, kind of, Python 2 to Python 3 mess, and really talking about the fact that that was such a hard shift—if you have a bunch of code in Python 2, and there's no backwards compatibility for whatever you're using in Python 3, are you going to spend the time to rewrite that code into Python 3, or would you adopt a new programming language?

And he even specifically says this. He says, “How much Python software—” I'm just quoting from his blog right now, “How much Python software was rewritten in GO (or Ruby, or some other alternative) because of that backward incompatibility?” And so, he even went on to just be like, “Listen, what if Apple did the same thing, and just breaking all this backwards compatibility? And you do this and maybe you lose 10 or 20 percent of your user base. You do that a few times, and your user base is going to just erode before you.”

Corey: Yeah, two or three times of doing that, and you've lost half your users, which is not a good metric that any platform wants to see.

Pete: It's just incredible, yeah. And I think he had a great line, which was, like, my upcoming blog post, like, “How to Aim for Number Two, and Miss” or something like that. But it's true because if you're in enterprise, and you're using Google Cloud because let's say you sell something competitive to Amazon, or I think the common trope is you want to sell something to Walmart and you're offering runs on Amazon. If you're a big enterprise, and you're committing these resources—enterprises, they don't upgrade things. They just normally build new things. Old things stay around forever. It's the great adage of, “Broken gets fixed, but shitty lives forever.” How are these enterprises going to respond when the rug gets pulled out from underneath them at all these intervals? My guess is, is that places like Azure and Oracle look a lot more attractive.

Corey: That's what Google's missing. For all its faults, Microsoft is very good at providing obnoxiously extended support. I've heard whispers that there are still Windows XP customers in some manufacturing floors that are getting long term support even now because the enormously expensive piece of equipment requires something that only runs on XP to function.

Pete: Right. And if you imagine, even, like, XP Embedded, I had a car navigation system from 2004 that ran XP Embedded. That car is still on the road somewhere and may need some sort of support. And again, that's just a car. What about all the other systems that are out there?

It's definitely something that I would keep in mind, I would think about when I'm building an application in the cloud world, I'm not going to want to have to spend my very limited and very expensive engineering time to constantly change these APIs that are getting deprecated, which I think most importantly, these APIs getting deprecated, these may not actually bring value to your product or your business. It might just be toil; it might just be busy work that your company has to do because your cloud provider keeps making you. But all that is going to be an opportunity cost wasted away, and you're not focusing on the thing that hopefully brings your company money.

Corey: What just kills me about this entire piece is how there's a chorus of violent agreement around this. If you want people to use your cloud, on some level, you can't expect them to update things that are already working correctly, from whatever it is that they've built on this, that, “Oh, there's going to be a new library version that if you don't update this within the next year, it's going to start breaking.” People don't generally want to do that. If you're going to update something that you've built and shipped into production, you want it to be on your timetable, not because your upstream vendor decided that it was time for you to upgrade now that they come up with something better/different.

And a lot of these are the same services that are getting deprecated and updated. It feels like you're on a treadmill, at which point one of the more contentious things that he said was, “It is less DevOps effort to run these open-source versions of whatever it is that you're using instead, and just manage the VMs because at least that stuff's going to work.” Now, people have argued about that, but I disagree that it's necessarily the wrong approach. Yes, it's more work on some level, but you also get to determine where that work takes place and how it’s functioned. You aren't beholden to someone else's timelines in quite the same way.

And as we look at a lot of cloud migrations, people are still struggling with how to migrate their freaking mainframes from the 1970s. That's a long time. How much PHP written in the early 2000s is still an active production? You've got to understand that customers do terrible things, and if you're not willing to support those things, maybe you shouldn't be a cloud provider.

Pete: Yeah, I mean, towards the end of this article, this post, he’s like, “Listen, they want you to use this. They want you to use their cloud. They just don't want to support anything because support is not part of that Google DNA.” And he even says, “Listen, the engineers support each other,” because he called out that big table anecdote where this engineering team reached out to help support him.

But you got to remember, too, is that they reached out because he was running something old. They don't like that. It's not clean. They need to go and upgrade this thing sitting around, and they're willing to help you to do that. What's interesting, though, and I'd love to hear from any larger customers that have maybe a more of an economic advantage with Google—although does any really exist for GCP—but, who have been able to push back on some of these deprecations? Can they push back and use money as a way for a company that truly doesn't need money? I mean, they just—it's so much ad money, that it's almost like, what's the point of Google revenues? I mean, it's really fascinating to see.

Corey: It really is. Pete, thank you for taking the time to rant about this with me. Me ranting in a monologue about this would be far less coherent. It's challenging to wind up and start talking smack about GCP, just because I'm perceived—rightly or wrongly—as an AWS fan. Which I am, but I want Google to get better at this. I want there to be a viable Google option. I don't want the default answer to be AWS for everything. So, we'll see what comes out of it.

Pete: Yeah, thanks a lot for letting me join. I think just reading this and being kind of blown away by it was a really a great way to start the day for me, so I appreciate it.

Corey: Thank you for joining. Pete Cheslock, cloud economist here at The Duckbill Group. Along with me, Corey Quinn, chief cloud economist at same, here on the AWS Morning Brief: Discussions.

Announcer: This has been a HumblePod production. Stay humble.