Tim is a general-purpose Internet-software geek. He specializes in Web, search, writing, speaking, business.  He founded Textuality in 1996. He is available for consulting on issues of technology leadership, software construction, and distributed systems. You can follow Tim's musings on his blog ongoing and on Twitter.

Corey’s Fake PR FAQ: https://www.lastweekinaws.com/blog/introducing-aws-elastic-beanstalker/

Announcer: Hello, and welcome to Screaming in the Cloud with your host, Cloud Economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: It's, at least as of this recording, morning on the West coast, which means there's no better time than to inflict a homework assignment upon you in the form of a 42 page ebook from StackRox. Learn about the dancing flames of EKS cluster security, evade the toxic dumpster of the standard  controls, and tame the wild beast of best practices for  minimizing the risk around cluster workloads. Become renowned for your feats of daring, as you learn the specific requirements for securing an EKS cluster and its associated infrastructure. To learn more, visit snark.cloud/stackrox. That's  snark.cloud/stacROX.

Corey: This episode is brought to you by Trend Micro Cloud One, a security services platform for organizations building in the cloud. I know you're thinking that that's a mouthful because it is, but what's easier to say? "I'm glad we have Trend Micro Cloud One, a security services platform for organizations building in the cloud" or "Hey, bad news. It's gonna be a few more weeks. I kind of forgot about that security thing." I thought so. 

Trend Micro Cloud One is an automated, flexible, all-in-one solution that protects your workflows and containers with cloud native security. Identify and resolve security issues earlier in the pipeline and access your cloud environment sooner, with full visibility, so you can get back to what you do best, which is generally building great applications. 

Discover Trend Micro Cloud One, a security services platform for organizations building in the cloud. Whew. At trendmicro.com/screaming.

Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Tim Bray, formerly a VP and distinguished engineer at AWS, and now an unemployed bum. Welcome to the show, Tim.

Tim: Delighted to be here.

Corey: So, no matter what side of the world you live on and what your perspective is, it's pretty clear, there's one thing everyone can agree on, and that is that you are basically just the side of a war criminal. You were one of the original voices behind the XML spec, and for those rare few of us who really like XML, you also participated in the JSON atrocity.

Tim: Well. So, what's plan C if neither XML or JSON are going to work for you?

Corey: Uh, YAML. It's how we build our safest, whitest spaces.

Tim: [laughs]. Yeah, you know, YAML’s so convenient and pretty looking. Most senior engineers hate it for the simple reason that in YAML when you have a message, there's nothing that marks the end of the message, so if you screw up and drop some off the end, or the network breaks at the wrong time, well, you just go ahead and process YAML as if you got it all. But you didn’t, and that can cause some really horrible things. It's worth talking about XML and JSON a little bit though. 

So, we did XML in the late 90s; it was completed in ’98. So, the really astonishing thing was before that, there just weren't any data formats that were vendor-independent, whether it's computer vendor or database vendor, or programming language independent, which is shocking because we'd been doing computing and networking for 20 years at that point. And so since XML was the first one, it got picked up and used for everything, which didn't work out that well. XML was dreamed up by a cabal of publishing technology bigots, who really cared about deeply nested document structures and tables of contents and index cross-referencing and things like that, which turns out not to be the greatest choice for using in REST APIs. Having said that, there's still a thriving XML community doing things like legislation, large-scale technical documentation—like for airplanes—humanities computing, so on and so forth. So, if you're in the publishing space, hey, that's what you want. 

As for JSON, you know, JSON is irritating in some respects, but we've made it work. It does most of what you need. We all know what the next few things we would change in it, if we could, to make it better are, but we're not going to because it's too late. You know, all the software is written and you can't be changed anymore. The question of what the best way to package up data is to ship it back and forth between heterogeneous systems is interesting, and you can have a lot of fun. And these days, there's things like Thrift, and Avro, and protobuf, and people, for very poorly worked out and verified reasons, seem to think those are the universal choice in the future. Eh, I’m unconvinced.

Corey: [unintelligible] you can go schemaless with it, and have all kinds of fun ways of packaging. I mean, SaltStack back in the early days, with Python Pickles on top of that.

Tim: Yeah, that's absolutely true. And actually, that was one of the most interesting things I was working on there at AWS, is we did a schema repository last year. And that means that, from the AWS point of view, you now have data types with ARNs. And I think that there's way, way too many AWS services that interchange—

Corey: You can stop the sentence there if you'd like. Way too many AWS services. Full stop.

Tim: [laughs]. That's another interesting discussion you can have. You know, Andy loves to get on stage and talk about how many new things they shipped every year. And—

Corey: And everyone else gets this sinking sense of dread in the pit of their stomach when they see it, of, oh, dear Lord, at least 20 of those would have helped solve problems that I have, but I'm too busy doing my job that isn't keeping up with Amazon services.

Tim: Yeah, you know, having said that, are you actually going to say it's wrong because, you know, it seems to be going pretty well. Yeah, yeah, there's more there than any one human being can stay on top of, and in a lot of cases, people pick the technology based on what they already know, as opposed to what might be optimal.

Corey: I suspect that's the case on almost every scenario. People talk about, on stage, about the exciting far-future stuff they're doing, but if you look at actual cloud bills and the environments that the serious, huge reference customers have, it's all a bunch of, to be frank, the boring stuff. And we pick that boring stuff for the simple reason of, we know how it breaks. If you don't know how something breaks, it's hard to trust it with critical workloads.

Tim: Sure, fair enough. But in 1982, you could have said, well COBOL’s what everything runs in, so why would we be poking around looking at other stuff? And you need to throw a bunch of new stuff at the wall to see what sticks. And some of it is sticking. I mean, if you look at the numbers for things like Lambda, and EventBridge, and so on, you know, a lot of people are using that stuff these days. 

And it is non-traditional, and the way things are done in 15 years is going to be quite different from the way things are done today, and the only way we're going to get there is by introducing new things, and finding out which ones actually meet a demand.

Corey: So, there's a lot of things we can talk about in the technical space, but first I want to get to the—well, I wouldn't even call it an elephant in the room, because everyone acknowledges it, talks to it, it's wearing a name tag, and often gets its own introduction, and slide deck on stage at the podium—you were a VP slash distinguished engineer at AWS. There are less than 20 of them at all of Amazon. It is the highest pinnacle of technical achievement. At other companies, they will be called ‘fellows’ in many cases. You resigned on the basis of ethical issues earlier this year. Talk to me about that, since that is virtually unheard of—at any company—for someone at that tier of technical achievement, and that level of seniority within an organization, to leave publicly citing anything.

Tim: Well, on the other hand, I didn't see, really, any other realistic choices. When you achieve VP rank, you're expected to be on the side of the company consistently, and be willing to get behind what the company is doing, and not be a [BLEEP]-disturber in public, and I respected that. And when I found the company was doing something that I just couldn't live with, I made that clear going through proper channels internally. And having done that, what other choices were there? It's not a thing that one can just get on stage and say, “Well, firing whistleblowers is okay, because—” because there's nothing that comes after the ‘because.’ so… resigning seemed like the only possible thing to do at that point. And it didn't make me happy. You know, AWS is a fantastic organization. I loved working there: the people are great, the customers are greater, the work is fantastic, but, you know… it just didn't seem like there were any other options.

Corey: I hear you. Specifically, for those who have not been following that particular saga, there were a number of high profile firings of warehouse workers who just so happened to be involved in speaking out specifically around aspects inextricably linked to labor organizing, which, heaven forbid, we wouldn't ever want the people who work in our warehouses to be able to bargain collectively; that could end badly for us.

Tim: Well, except for that's empirically false. I mean, there was a really great case study that just happened in May, I believe, where the Amazon warehouse workers in France were concerned about the handling of the COVID and didn't think the company was doing enough to protect them, and so they started raising their voices. But, you know, Amazon doesn't talk to unions. Only, in various jurisdictions in Europe, the law says you have to. So, the union took them to court and won a judgment that Amazon had to talk to the union, and ship only essentials. 

Amazon reacted by shutting down all the warehouses—which doesn't seem like the smartest thing to me—and appealing the court judgment, which they lost on appeal. And then, having been backed into a corner, they went and talked to the union, in a matter of a few weeks, worked out a deal and got all the warehouses reopened and working. And, you know, Amazon's doing okay in Europe, so it is empirically true that Amazon can, in fact, work in a unionized environment and still be successful.

Corey: To be clear, my statement earlier was dripping in sarcasm, which doesn't always come through, especially if you're reading the transcript of this episode, rather than listening to it so you can pick out the sarcastic overtones of my voice. But yeah, it absolutely makes sense to me that if you are working in virtually any employee role, that organizing makes an awful lot of sense. And I say this as a business owner. I don't view the fact that employees can come and talk to you on a collective basis to in any way be a negative direction for society to evolve in. Now, I understand that Amazon has its own position on this, and that's fine. To be very blunt. I don't know what it takes to run an 800,000 employee company. I don't ever expect or hope to find out. I feel that if I've gotten to that point, something has gone egregiously wrong.

Tim: Well, it sounds trite to say, “It's just too big,” but I think it's just too big. And it's not just an Amazon problem. Amazon is a symptom of the problem. I think that it's not controversial to say that there is an overly high concentration of wealth and power in the economy. And that's specifically egregious and visible in big tech where you have Google, Microsoft, Facebook, Apple—who am I missing—the Big Five that dominate their sectors and behave in classically monopolistic ways.

And I don't think it's controversial to say that that is dysfunctional regardless of your ideology, even if you are a firm believer in capitalism first, last, and always, capitalism doesn't work that well when you get overly high degrees of monopolization and concentration. And I'm a little bit even more fundamentalist than that. I think that it just doesn't work that well when the companies get, just, too big. And there's some line in the sand when you get bigger than that it just becomes hard to exhibit any aspects of humanity in dealing with either your employees or your customers. And I think objectively, you can look back the last couple of decades and say, “No, a lot of things about the economy just aren't working that well.” The experience of dealing with businesses, as a customer or as an employee is not getting better, it's getting worse. And we need to explore some, I think, fairly radical measures to sort that out.

Corey: Again, I run a 10 person company and believe it or not, we do have an ethics policy of who we will and will not do business with. And to be direct, the way that I've structured that has been that in 15 years when my now three-year-old winds up going to college, when I tell her where her college fund came from I don't want to be ashamed by the answer. And that means different things to different people, and it's super squishy, but because I'm small, I have the privilege of being able to say that. I don't want there to be a logo on The Duckbill Group's website for customer wall that is the same logo you're going to see on bomb parts scattered around a village somewhere. It's just a baseline level of who I will and will not do business with. I think you don't get to be a certain size of company and still hang on to that. I don't think there's any large scale cloud provider in the trillion-dollar range of valuations where they've kept their souls. I don't think you can and still get to that level. Do you agree with that? Do you think that there's a way around that, where there was a better path forward, or is this just the trade-off you have to make for growing to that scale?

Tim: Well, the world is complicated, and nothing is white or black. I will say that in my view, AWS as an organization is run pretty ethically. There were very few things we did they gave me, personally, heartburn. And I noticed that things that are stinky do get addressed, like the recent change of policy around recognition. I would be much happier if the company walked away from a whole bunch of policing and security sector businesses. But would just making it smaller address that? I don't know. 

I think that, in general, if you dislike the way that larger organizations are behaving, the only effective solution to that is to change the rules because Amazon and its competitors—AWS and its competitors—by and large, sort of, kind of, do play by the rules. And as John Adams says, we still are mostly an empire of laws rather than of men, although if he’d said that these days, it would be people. And so change the laws. I mean, politics is boring, and icky, and slow, and tedious, but it's really I think, the only hammer we have to drive this nail. So, if you don't like the fact that the company is selling to businesses that you think are awful, well, maybe we should change the rules to make those businesses not possible anymore. 

If you don't like the way that ICE behaves, well, change the way that—the legislative framework so that ICE is simply not allowed to do all the egregiously brutal things they're doing these days. I hate to say it; I mean, I would like to be able to talk to, say, Facebook, the way I talk to my kids and say, “Now you play nice, or no dessert.” But that doesn't work. It has to be done, I think, through traditional legislative frameworks.

Corey: Now, and it comes down to a lot of power imbalance, too. We saw recently, AWS filed yet another non-compete lawsuit against a former employee, in this case, Brian Hall, who went from AWS to GCP, and on some level, it’s, yeah, okay. It's a contract that you signed, and you should live by that contract. I'm sympathetic to that argument. The counterpoint is, though, is the way it is scoped is so hilariously overbroad. It’s scoped to all of Amazon—which, first, can you name a single industry that you could safely guarantee by the time you leave that company, Amazon will not be doing anything with? 

And it covers anything you may have had access to, by a strict reading, which is really the only way to read something like that before you sign it. And it's the only company I've seen that enforces these things this aggressively and at this scale. And I've got to say more than almost anything else, things like that, where Amazon is, I guess, unkind and punching down are the few points where I lose respect for them. Sure, they can ship a bad technical product, and I can make fun of it, but we're all still friends at the end of the day. You can't start punching down at your own employees, which is really what ties this all together, and still be someone I look at with an, “Oh you,” look on my face.

Tim: Well, yeah, and obviously the thing that drove me from the company—the firing of the activists—is you can describe it using similar terminology: ‘Punching down.’ Having said that, it isn't just Amazon. Microsoft has a history of doing this, too, bringing up the non-compete cannon. And what Amazon and Microsoft have in common? Well, they're both Washington State companies. 

Non-competes are non-enforceable in California, and that's one of the big reasons why a large part of the technology industry is still in California. I mean, the birth of Silicon Valley was Intel, and Intel was an act of treachery when a bunch of people from Fairchild Semiconductor went off and decided to take what they learned and build a new company around it. And that's fine. I've got no problem with that. And I think Washington state would do its citizens and its employers and its employees a service by striking down the ability to enforce non-compete clauses.

Corey: I would absolutely hope so. I think that there is so much that goes wrong, and could be better than it is. It's frustrating. I think that non-competes are one of the most aggravating things in the world because when it comes time to go somewhere else, there is such an imbalance of power. 

I mean, I'm a small business. If I'm trying to hire someone and Amazon reaches out with, “So, just so you're aware there's a non-compete issue here,” It's very easy for me to say, “Oh, I don't want to go into a legal battle with Amazon. So, I'm just going to go with my second choice candidate instead.” It provides an incredible chilling effect. Now, personally, because I am who I am, my response is, “Bring it; I will burn this place to the ground fighting you if I need to,” just because I don't know when to pick and choose my battles. But it is the common case that it is incredibly chilling. And the cavalier attitude, and the leaked memos that come out of Amazon about these firings has just been bizarre to me. It’s, what are they so afraid of?

Tim: Well, yeah. At the end of the day, it was the ethical dimension of the whistleblower firings that drove me out, but you also have to point out that that was, like, really stupid, egregiously stupid. I thought, “Maybe I'm missing something.” As you say, I don't run an 800,000 person company, either, but firing whistleblowers feels like, “I never heard of the Streisand effect. I'm putting up letters of fire 50 meters high in the sky saying ‘hey, there's something here we really don't want you to look at.’” 

I don't see how anybody with any maturity could think that doing that would have a good result, creating a climate of fear in response to activism. And the people who are activists had no thought of gain for themselves. They weren't trying to make money, or advance their careers, or anything like that. They were doing something that I thought was wholly admirable. I mean, they could have done something like say, “Okay, you guys are upset about that. Great, you got a new job serving on the task force to make the warehouses safe, and prove we've done it.” There was a certain amount of loss of respect in my mind for leadership around that.

Corey: I would agree. I think that there's so much that Amazon could do in this space, and demonstrate real leadership and, for whatever reason, it's choosing not to. I mean, I consider myself an Amazon fan. I think that they do a lot of good things, and I think a lot of what they do is admirable, but I'm believed when I say those things, because I also call them out when they do things that are awful, and this is one of those awful things.

Tim: In terms of all the places I've worked in—I'm old guy: I’ve been doing this for 40 plus years—AWS was by far the best managed place I ever worked, and also the asshole density was really low compared to anywhere I've worked. And, you know, there would be whole weeks at a time when I never got mad at anybody. Which, as you know, in most jobs is not the common state of affairs. So, yeah, they're doing a lot of things right, and that's why this kind of stuff stands out in such stark contrast.

Corey: One thing that—especially in the wake of the non-compete lawsuit, which I've been getting more traffic on than I have the warehouse firings—I've gotten a bunch of outreach, both from people considering working in Amazon, and people who do work at Amazon expressing concerns about their non-competes, and what should they do? And I have my own thoughts on it, but where do you stand?

Tim: So, I got the same thing. When I quit, I got a lot of outreach from Amazon employees saying, “Oh, gosh, now you quitting makes me feel bad. Can I go on working here?” And to be fair, I had to point out that I am elderly, and nearing retirement age, and financially secure, so this was a lot easier for me than it would be for a lot of other people.

Corey: Oh, we are both dripping in privilege in this context.

Tim: Absolutely. And I think that at the end of the day, that the problem isn't really Amazon. The problem is the Reagan/Thatcher consensus, 21st-century capitalism, and the imbalance of power and wealth in modern society. And if you're going to boycott anything remotely related to that, you're going to have a hard time making a living. And there are lots worse places to work. Amazon is—by a wide margin—not the worst operator out there. So, I've actually been advising people, “No, don't do what I did, unless you're absolutely unable to sleep at night, and got a financial plan.” But that's all I could say.

Corey: In what you might be forgiven for mistaking for a blast from the past, today, I want to talk  about New Relic. They seem to be a relatively legacy monitoring company, and I would have agreed with that assessment up until relatively recently, but they did something a little out there: they reworked everything. They went open source, they made it so you can monitor your whole stack in one place. And most notably from my perspective, they simplified their pricing into something that is much more affordable for almost everyone. There's even a free tier with one user and a hundred gigs per month, totally free.

Check it out at newrelic.com.

Corey: So, let's pivot a little bit towards the thing that, again, you were one of the best in the world at, which is sort of funny, because originally, you were one of less than 20 people who had reached the pinnacle of the individual contributor engineering ladder at AWS, then you resigned in protest and suddenly, “Oh, that guy had terrible judgment. We never liked him anyway. It was probably an aberration,” was the entire thrust of the messaging around you. That's some impressive level backpedaling, but let's not kid ourselves here, you can be as modest as you want, but I'm not going to be. 

You’re one of the best engineers in the world. That is why you were in the role you were in; it's why you've done the things that you've done, and I want to talk to you a little bit about technology. So, from a high level, let's talk about cloud stuff, for example. I think most notably in a combination technical/political venue to transition us, let's talk about fear of lock-in. a lot of people are scared of being all-in on a provider that reduces their choices in the future. Where do you stand on it?

Tim: I think it's a really important issue, and not one that simple or straightforward. You know, AWS is a $40 billion business these days, which is to say bigger than a lot of the customers. And in the IT world historically, there has been a trend to platform lock-in. For example, if you are a large company, you no longer get to decide what your budget for desktop technology is; you pay whatever Microsoft says you're going to pay; you have no bargaining power. Similarly, on your database spend, you pay whatever Oracle says you're going to pay; once again, no bargaining power. 

And you just know that in a lot of industries that have some decades of experience in IT, the leadership, the CEO, the board, even, is going to the CIO and saying, “Now, don't you come back here in a couple of years, and tell me that you've lost control of our cloud spend.” And, okay, that's a very reasonable concern to have, and I can't see how you could argue against that point of view. Well, you can because in practice it turns out to be really difficult to be cloud-agnostic. I mean, if all you're going to do is rent CPUs, and storage, and operate databases, fine, you can do that. But even at that level, you find that there's a bunch of really annoying semantic gaps between the way the major cloud vendors think about what you mean by ‘instance’ and what you mean by ‘object’ and so on and so forth.

So, you can in fact build out your technology stack in a cloud independent way, but you're going to pay a lot more, and you're going to go a lot slower. So, I think the choice for a startup is a no brainer. A startup should pick one of the cloud vendors, jump into bed, aggressively make use of all the hot stuff they're shipping in that particular year, and that way, they'll get the most velocity; they'll spend less; they'll move faster; they'll have a better experience. You know, startups are short of time and short of money, so that's what they should do. Now, if you're a bank, or a pump manufacturer, or a book distributor, or something like that, can I honestly say that, “Ah, blow off that stupid lock-in concern, just go jump into bed with a cloud vendor?” I don't know. 

I think that is a more nuanced discussion and one that you have to have. How much are you willing to pay? Now, a lot of people say, “Well, the cloud lock-in isn't really at the level of the APIs. It's at the level of data gravity.” Once you've got several petabytes of data sitting in Google Cloud or AWS, well, you're just not going to move out because it's too hard. I’m not sure that's true. I think you can in fact do that. And in fact, these days, you can go on a case-by-case basis. I mean, are you willing to use a proprietary database like Dynamo, or BigTable, or something like that—

Corey: Or Route 53?

Tim:—[laughs] in the knowledge that, then you really pretty well are going to have a hard time getting off that vendor. Maybe not. On the other hand, if you are going to decide, “I’m going to standardize on Postgres. Am I willing to use hosted Postgres, and use somebody else's control plane to create databases?” Well, that's a much less severe degree of lock-in. 

And this is a very complicated discussion, and I would be willing to have an opinion after a discussion with any individual business that is getting into the cloud, but would I lay down a dictum saying, “Oh, fear of lock-in is dumb, or always resist lock-in?” I would not say either of those things. It really is a case-by-case situation.

Corey: Very much so. My railing about multi-cloud is always from a best-practices position. I think that as a general best practice, pick a provider—I don't care which one—go all-in. Now, there's a litany of exceptions to that, where it does not make sense, where you cannot do that specific thing, and that's fine. My argument has been against hearing this from a conference stage from some second or third rate vendor saying, “Oh, multi-cloud is absolutely what you want to be doing,” and accepting it blindly. That has been my issue.

Tim: Mm-hm. Fair enough. And I think this is a super hot area of technology, and I think there's lots of opportunity for vendors to facilitate the process of being multi-cloud where necessary. I mean, HashiCorp is now a unicorn, right? And we should all be paying pretty close attention to what they're doing because I think their existence shows that there's a hot button out there that they've pressed, and a lot of people care about a lot.

 

You know, at the moment, I can only speak to AWS because that's where I worked, but AWS is a really pretty safe place to jump into bed with at the moment because people like Andy Jassy, and Charlie Bell, and so on, are just really customer-obsessed. They spend a huge proportion of their time trying to figure out what's making customers unhappy and just fixing it. And they're super aggressive about cutting prices, and things like that. And that's great, so that's the kind of vendor I want to have as a customer. But 10 years from now, when I'm retired, and Charlie's retired, and so on and so forth, is it still going to be the same, or is it going to be a bunch of ex-private equity MBAs running the thing and trying to figure out how they can extract the maximum rent? It's a thing that's reasonable to worry about.

Corey: Well, always a disturbing concept. One thing you said a minute ago about HashiCorp being aimed at the multi-cloud story. When I had Mitchell Hashimoto on the show previously, we talked about this. His idea was never that Terraform should be used in this multi-cloud way. It's not about workload portability, it's about workflow portability. 

And having seen that play out at multiple customers in the past, it's always seemed to align in that particular direction, where, okay, you're always going to have to do a lot of rework to get something that works on one provider in Terraform to work somewhere else, but the workflow, how you interact with your codebase, how you get things from your head into production, that's what you can wind up porting between providers. So, I want to be very clear on that, or oh, my stars, will I get letters. But let's talk about something else that I think has areas of lock-in commonality. Let's talk a bit about, for example, the idea of event-driven architectures. It feels to me like the event model is one of the least portable things you could have, depending upon what it is you're doing. Am I wrong in that?

Tim: It depends. I mean, each of the cloud vendors has their own event pub/sub, routing, invocation frameworks, and it's shocking how similar what they do is. If you care about all the different semantics that you can have around messaging and eventing services, there's not that many. And you can make up a laundry list of things that you care about. And you say, “My requirements are A, B, and C. And yeah, I can do that at Google Cloud, or AWS, or Azure.” 

And given that, it's kind of annoying, that things are, relatively speaking, incompatible. There's a lot of people I talked to who were using things like Apache Camel, which is this big, hairy, complicated API, but what it does is provides an abstracted send message, receive message thing. And I'd see people using that, and then wiring it up to SQS, or RabbitMQ, or something like that, and then the person who's running the Java production code didn't have to care about their messaging framework. So, there are some tools for making it less proprietary. But having said that, that's what you kind of expect, because the event-driven stuff is sort of the new shiny at the moment, what with Lambda, and EventBridge, and that kind of stuff. 

That is where we're pushing back the boundaries and figuring out new ways to build applications. So, you go explore the new territory first, then after you figured out what you use, you write the rules for it. You know, after you figured out what works, you write the rules for it. And we're still in the stage of figuring out what works, and the initial signs are that, by and large, event-driven software works pretty well, and is highly applicable in a lot of situations. But do we have a set of commonly agreed on best practices and standards yet? No.

Corey: That's part of the issue is that as standards fail to emerge, and they start turning into these, I guess, de facto standards rather then ones that are imposed, there's an awful lot of companies jockeying, for whatever it's worth, to start being the voice that determines what those standards are going to look like. And increasingly, it seems that those standards have been driven less by the community and more by whoever has market share and is the first mover in the space. Does that align with your experience?

Tim: Well, this is the classic thing is you get a market leader who becomes the de facto incumbent, and then all the market trailers get together and form a standards organization. [laughs]. And, whereas they would never come out and say that, one of the goals of the standards organization is to slow down the incumbent to give the up-and-coming smaller parties a chance to compete on a slightly different playing field. And that's okay. That's fine. 

I've been in both roles now, as the market incumbent and as the upstart trying to get a standard put in place. It's part of the organic flow of how we do things in this technology. In the space at one point, SQL was seen as a highly proprietary technology, and these days, it's regarded as a pretty level playing field. I'm not smart enough to predict how this is going to shake out, but I think I’d put my stake in what I said a couple minutes ago, which is that, you can either use the cutting edge, sharp new stuff that does magic, or you can play on a level, well-understood, standards-driven playing field, but you really can't do both.

Corey: Yeah. It definitely seems that for better or worse, there's going to be a shakedown at some point. It feels to me--this is aligned in a lot of ways, with the idea of complexity gets built on top of complexity, and it almost like a sawtooth graph because eventually people look at it and say, “This is nuts,” And it collapses down into something that a human being has a hope of understanding. Kubernetes, right now, feels like it is incredibly complicated and overwrought. In a few years, it won't be because something's going to come along with, huh, okay. Maybe a team of engineers who cost a quarter-million bucks apiece is not something every company is going to want to run its application architecture, and there's going to be some evolution there. I guess cynically, I tend to view things like that let companies cosplay as cloud service providers themselves when they don't work for one.

Tim: Yeah, I don't want to diss Kubernetes too much because it does some—

Corey: Oh, I do.

Tim: [laughs]. It—I don't actually use it, either, but I don't want to diss it too much. It does some remarkably clever things that do make operator's lives easier once you've got everything set up. But it fails one important test in my mind. I think that technology should make easy things easy, and difficult things possible, and Kubernetes clearly fails the first half of that. It's too hard to understand, and bootstrap, and it costs too much, as you just said, in terms of engineering to keep it going. 

We've seen this movie lots of times before. There was ISO networking, and seven-level stack, and so on. And then the internet geeks came along with TCP/IP, which only did 20 percent as much, but it turned out to be the right 20 percent, and [unintelligible] rest of that stuff away. Same thing happened with the web. There were increasingly elaborate, complicated application framework, starting with Visual Basic and, you know, and Adobe's offering, and Novell’s offering, and so on and so forth. And then the web came along with a vastly simpler and impoverished user interface paradigm, and swept all that crap away because the 15 things it did really well turns out to be the ones that matter. 

And I would be deeply unsurprised if that didn't happen to Kubernetes. Kubernetes does a whole bunch of stuff, and all of it is not equally important. What I frankly expect to see is something come along that does the things you actually really care about, and you can read about it Monday morning and have an app up Tuesday afternoon, which is incredibly important. And if you look at the adoption of each real game-changing technology, it's almost always had the characteristic that it's easier than what came before. And this profession is hard enough as it is, without making things unnecessarily complicated.

Corey: So, at a high level, now we spent some time looking back, as you said, you've been doing this for 40 years. What technology trends are we seeing that interest you the most? What's the future look like?

Tim: At the moment, well, it's one that we already talked about a bit, which is, are there interesting tools that are going to be truly useful, and enabling you to avoid vendor lock-in without grossly increasing the cost, and decreasing the velocity? We can all be a little bit cynical about that now, but these things have happened, and the most classic example of that was the World Wide Web when it came along. Before the web came along, there were all these competing technology stacks, and you had to figure out whether you were going to be on the Microsoft stack, or the Sun stack, or the IBM stack. And then the web came along, and it was a platform without a vendor. That was a profoundly important thing; it was the first time we'd had a platform without a vendor, and is there going to be a public cloud platform without a vendor out there? Any steps towards that would be super interesting in my mind. So, that's one.

Another one moving in a completely, totally different direction is, I'm all excited about Augmented Reality, AR. How many years has it been, since a new mobile device changed anybody's life? Come on, they’re just not interesting anymore? Or, how long has it been since a new programming framework has really rocked the industry? Well, serverless in 2014, maybe, but they're few and far between. 

So, AR I think has the ability to really dramatically enrich the life of humanity at large, and right at the moment, the technology isn't good enough. What you would like to have is technology so you can just hold your mobile phone up and look through it and see a decorated version of the world. You know, you're walking in a park at night, and you can see fiery dragons climbing up the trees. You walk into a superstore and say, “Where's the deodorant?” And a big arrow appears in front of you pointing towards that. There's so many applications. 

Technology isn't good enough yet, but when the technology does become good enough, it's going to be a huge new area of human creativity, and innovation, and new business sectors that we can't even begin to think about. There's little birds here and there telling me that Apple's got some great stuff coming along. I’ll believe it when I see it, but the problems we're trying to solve are basically computer vision problems, and ML problems, and so on, and they'll get solved all right. So, that one's got me all excited.

Another area that I'm super interested in these days is public sector procurement. In my career, I got, twice, mixed up in large government procurements of technology, and oh my God, what a cluster-[BLEEP]. It's done very, very badly, and I think this is, sort of, generally universally true across the governments of the world. And the blue suit consulting companies that specialize in government technology procurements are just not working properly, and we have endless disasters. So, I'm aware of some efforts to try and bring sanity to that process, and that's super interesting to me as well. So, we could dive down any of those rabbit holes.

Corey: I think that the problem with rabbit holes is that in some cases, there's a rabbit at the bottom and, be careful, that rabbit’s dynamite; pointy teeth, and whatnot. One of the problems that I see is, as I look across the landscape, everyone's talking about the future, about the trends of what is coming. But if I look at my customer base, and I look at what is driving the actual spend—where the money's going—it seems that a tremendous constituency is treating the Cloud as someone else's data center, where they run VMs steady state, there's no auto-scaling, and it seems like the primary problem that they're trying to solve is that they suck at running data centers themselves. Some people give up halfway, and call it hybrid, and it feels on some level to me like they are improving their data center environment at the cost of leveraging the Cloud for what it's really capable of doing. And I find that depressing on the one hand, but on the other, I understand that they have what we'd like to disparagingly call legacy workloads, which means they make money. How do you feel about that?

Tim: Well, to certain extent, that's simply an inevitable result of arithmetic. So, AWS is, what, $40 billion a year now, and maybe the whole rest of the cloud industry together is that again? I don't know, something like that. And so $80 billion sounds like a lot of money. Last time I checked, the enterprise IT spend globally is like $2.7 trillion per annum. So, the whole public cloud infrastructure is a small sliver of the business.

Corey: Oh yeah, everyone talks about AWS being a $40 billion annual run rate, but Dell’s revenue was $60 billion in 2012. There's a tremendous—and that's one company that basically was selling hardware and some services back then. We’ve only seen the tip of the iceberg.

Tim: And for that very reason—as you say, running your own data center sucks. The cost, admin, velocity, security advantages of moving to the public cloud, are—I think everybody agrees now—are pretty high. And so I think that whether we like it or not, the revenue growth and core business of the public cloud is basically going to be doing exactly what you said: getting the people the hell out of there data centers, and improving their security posture, including their availability posture, that kind of stuff, just by moving existing stuff to the public cloud, your classic lift and shift. So, okay, that's fine. There's nothing wrong at all with that. It's a little bit boring, but it's an important big money, big deal. 

So, let's look at what comes next. If you look at your typical large enterprise, they'll have an inventory of apps—a couple of hundred is typical—apps that their business depends on, in any given year, they maybe do work on a single-digit number of apps. You know five or ten kind of thing, and the rest of them just tick tick along. And so what's actually interesting is the new stuff that gets built. And so, for new apps that are getting built, are they being just done on classic monolithic vert architectures, you know, web server in front and database engine behind? Well, yeah, some, and there's nothing wrong with that, especially for smaller-scale departmental ops. 

But anything that's being built that's new, and ambitious in scale, I think the technology uptake of the new stuff that I was working on is pretty good, actually. And incrementally as we do these five apps this year, another five apps next year, eventually the present and future start outweigh the past, but at the moment, they don't. The past is winning, we're moving the past from data centers into the cloud, and realistically that's where the revenue is coming from right now in its largest bulk, and that's okay. I don't see that there’s anything wrong with that, but obviously, those of us with an eye to the future are really, really worried about making event-driven, and serverless, and container, and all that stuff, technology, irresistibly attractive for the construction of the new apps.

Corey: The idea of throwing away the old in favor of the new is one that feels pervasive, and is something that everyone loves to talk about, but it feeds into a common deception that I'm seeing where everyone seems to believe that just after this next sprint is over, then, then we're suddenly going to start making good decisions instead of the bad ones we've made to date. And, oh, I'm as guilty of this as anyone; this is not me casting shade. I wish that oh, as soon as I get a little chance to breathe, I'm going to go back and fix all of my crappy broken code. Spoiler: he would not. But it's the hope that springs eternal. And you see that companies never seem to quite outgrow this.

Tim: Well, as a former principal engineer and distinction engineer at AWS, one of the things that principal engineers spend a lot of their time doing is stopping people from doing that. You know, respect the past is a core engineering principle. And you may hate the existing code base that's running your business-critical applications, or your business-critical AWS service that was launched prior to 2010, but it works. And part of the problem is that a lot of developers hate reading other people's code, and don't want to learn how it actually works, they just want to rewrite it themselves. And once you get to be in a position where you've done this for 20 or 30 years, you realize that, you know, that isn't as easy as you think it is. 

And embodied in that crufty old codebase is a huge inventory of decisions that were made to meet particular weird situations and corner cases, and achieve non-obvious behaviors to turned out to be correct, and there's no way to know that by looking at it. Now, things are getting better. There's this great book called Dealing With Legacy Code. And it defines legacy code, interestingly—nothing to do with age or anything like that—as code that lacks unit tests. 

And since unit tests became pervasive sometime between 2010 and 2020, things have gotten better because in many cases, the unit tests realistically represent the contract between the codebase and the outside world. And they make it much more thinkable to replace the codebase with something that's more modern, runs faster, runs cheaper, runs cleaner, emits less carbon, and if it still passes the unit test, hey, it's probably going to work. So, yeah, respect the past. Don't flippantly decide that you're just going to rewrite the system because you're smarter than the people who wrote it, because you're not. But on the other hand, we should be super dogmatic and make sure that we never do anything that isn't fully and completely unit tested to enable our successors to be a little bit more courageous in replacing the past where that's the right thing to do.

Corey: I think that's a really good takeaway here. If people want to learn more about what you're up to these days, how you're thinking about things, where can they find you? Now that you’re an unemployed bum, there's no corporate blog, you can drive people to. Where do they find Tim Bray?

Tim: Just type my name into Google. There's lots of stuff that comes up. I talk to the world on Twitter and on my blog, and I enjoy doing that, and I'm not going to stop doing that as long as I can lift my hands up to the keyboard. I published a PR FAQ for how to spin off AWS from Amazon, which is something that should obviously be done, but I also published an article about how Google should be broken up, which is another thing that should obviously be done. And I'm not as funny as you, Corey, but I do try to be kind of entertaining.

Corey: Oh, I was a big fan. I took inspiration from that to publish my own fake PR FAQ about Elastic Beanstalker. I love the format; I think it's an underappreciated comedic medium.

Tim: [laughs]. It's definitely an underappreciated medium. I've had a few advisory consulting gigs since I left, and a lot of people want to learn about what's this six-pager thing that Amazon talks about, and how does that really work? And it does really work, and I think I've given them value by talking to them about that. But I think the comedic six-pager is an underappreciated opportunity, so keep running with that.

Corey: That's a good way, I think, of framing it, and a decent enough place to leave it. Tim, thank you so much for taking the time to speak with me. I know that your days are full of unemployment these days—

Tim: That's right.

Corey:—but again, your time is always appreciated.

Tim: Oh, well, thanks. It's been delightful and entertaining, and I love talking about cloud stuff, so give me a call anytime.

Corey: Don't offer if you're not serious. Tim Bray, former VP and distinguished engineer at AWS, now unemployed bum, I am Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on Apple Podcasts, whereas if you've hated this podcast, please leave a five-star review on Apple Podcasts, and a comment telling me what's going to get you to leave your employer on ethical grounds.

Announcer: This has been this week’s episode of Screaming in the Cloud. You can also find more Corey at ScreamingintheCloud.com, or wherever fine snark is sold.

This has been a HumblePod production. Stay humble.