Episode Summary
Episode Video
Episode Show Notes & Transcript
- (0:00) Intro
- (1:18) Duckbill Group sponsor read
- (1:52) Corey's Puppet experience
- (3:49) What is Puppet?
- (5:04) Puppet’s role in DevOps
- (8:12) Challenges in technology adoption
- (12:36) Issues with legacy in tech
- (18:26) The misconception of “limited” skilled workers
- (23:16) Duckbill Group sponsor read
- (24:00) Corporate communication breakdowns
- (25:22) State of DevOps Report
- (32:02) Cloud adoption and missteps
- (37:46) More from the report and Nigel
- Puppet: https://puppet.com
- 2020 State of DevOps Report: https://puppet.com/resources/report/2020-state-of-devops-report/
Transcript
Nigel Kersten: [00:00:00] So legacy is really just a certain slice of rapid growth in applications and infrastructure that's sort of an unmanageable mess now.
Corey Quinn: Welcome to Screaming in the Cloud. I'm Corey Quinn. This promoted episode is sponsored by a long time, I wouldn't even say friend, so much as antagonist slash protagonist slash symbiotic company. With things I have done as I have staggered through the ecosystem. There's a lot of fingers of blame that I can point throughout the course of my career at different instances, different companies, different clients, etc, etc.
That have shaped me into the monstrosity that I am today. Far and away, the company that had the most impact on the way that I speak publicly is Puppet. Here to accept the recommendation for what I've become and how it's played out is Nigel Kirsten, a field CTO at Puppet, or THE field [00:01:00] CTO. I don't know how many of them they have.
Nigel, welcome to the show and how unique are you?
Nigel Kersten: Thank you, Corey. Well, I'm, you know, reasonably unique. I think you get used to being, you know, one of the few Australians living in Portland who's decided to move away from the sunny beaches and live in the grey wilderness of the Pacific Northwest.
Corey Quinn: This episode is sponsored in part by my day job, the Duck Bill Group.
Do you have a horrifying AWS bill? That can mean a lot of things. Predicting what it's going to be. Determining what it should be. Negotiating your next long term contract with AWS. Or just figuring out why it increasingly resembles a phone number, but nobody seems to quite know why that is. To learn more, visit duckbillgroup.
com. Remember, you can't duck the duck bill, Bill. And my CEO informs me that is absolutely not our slogan.
So to give a little context into that ridiculous intro, I was a traveling contract trainer for the Puppet Fundamentals [00:02:00] course for an entire summer back in, I want to say 2014, but don't hold me to it.
And it turns out that when you're teaching a whole bunch of students who have paid, in many cases, a couple thousand dollars out of pocket for, to learn a new software, where in some cases they feel like it's their taking their job away because they viewed their job rightly or wrongly as running the same script again and again.
And then the demo breaks and people are angry. And if you don't get a good enough rating, you're not invited to continue. And then the company you're contracting through hits you with a stick. It teaches you to improvise super quickly. Okay. So, I wasn't kidding when I said that Puppet was in many ways responsible for the way that I give talks now.
So, what do you have to say for yourself?
Nigel Kersten: Well, I have to say, you know, congratulations for surviving opinionated defensive nerds who think not only you, but your entire product, your demo could be replaced by a shell script. It's a tough crowd.
Corey Quinn: It was an experience, and some of these were community based, and some of them were internal to a specific company, and if people have heard more than one episode [00:03:00] of this show, I'm sure they can imagine how that went.
I gave a training at Comcast once, and set a personal challenge for myself of how many times could I use the word Comcastic in a three day training? And I would work it in and talk about things like the schedule parameter in Puppet, where it doesn't guarantee something's going to execute in a time window, it's the only time it may happen, if it doesn't fire off and then it isn't going to happen.
It's like a Comcast service appointment, and then they just all kind of stared at me for a while, and credit where due, that was the best user rating I ever got from people sitting through one of my trainings. So thanks for teaching me how to improv at basically what could have been a very expensive mistake on Puppet's part.
It accidentally worked out for everyone.
Nigel Kersten: Brilliant. Brilliant. Yes. You would have survived teaching the spaceship operator to that sort of a crowd.
Corey Quinn: Oh, I mostly avoided that thing. That was an advanced Puppetism, and this was Puppet Fundamentals because I just need to be topically good at things, not deep dive good at things.
But let's dig into that a little bit. For [00:04:00] those who are, who have not had the pleasure of working with Puppet, what is it?
Nigel Kersten: Sure. So Puppet is a pretty simple DSL. You know, DSLs aren't necessarily in favor these days. Domain specific language for those who have not caught up on that acronym. Yes. So, a programming language designed for a specific task.
And, you know, instead we've decided that, you know, the world will rest on YAML, and we've absorbed a fair bit of YAML into our ecosystem, but there are things that I will still stand by are just better to do in a programming language. If X, then Y, for example, is just easier to express when you have actual syntax around you, and you're not sort of forcing everything to be in a data specification language.
So Puppet's pretty simple in that it's a language that lets you describe the state that infrastructure should be in. And you can do this in a modular and composable way. So I can build a little chunk of automation code, hand it to Corey, Corey can build something slightly bigger with it, hand it to someone else.
And really this sort of collaboration is one of the reasons why Puppet's sort of been at the center of the DevOps movement, which at its core [00:05:00] is not really about tools, it's about reducing friction between different groups.
Corey Quinn: Back when I was doing my traveling training shtick, I found that I had to figure out a way to describe what Puppet did to folks who were not deep in the space.
And the analogy that I came up with, that I was particularly partial to, was, imagine you get a brand new laptop. Well, What do you do with it? You install your user account and go through the setup. You install the programs that you use, some which have licenses on it. You copy your data onto it. You make sure that certain programs always run on startup, because that's the way that you work with these things.
You install Firefox, because that's the browser of choice that you go with, etc, etc. Now imagine having to do that for, instead of one computer, a thousand of them. And instead of a laptop, their servers. And that is directionally what Puppet does.
Nigel Kersten: Absolutely. This is the one I used for my mother as well.
Like, I was working around Puppet for years before, and I used, the way I explained it was You know, when you get a new iPad, you've got to set up your Facebook account and your [00:06:00] email. Imagine you had 10, 000 of these. And she was like, and I was like, you know, companies like Google, company like big banks, they all have lots and lots and lots of computers.
And she was like, they run all those things on iPads. And I was like, this is not really where my analogy was going. But
Corey Quinn: right. And increasingly though, that seems like the world has shifted in some direction where when you explain that to your mother and she comes back with, Well, wouldn't they just put the application into Docker and be done with it?
Oh dear. But that seems to be in many ways that the direction that the zeitgeist has moved in, whether or not that is the reality in many environments, where when you're just deploying containers everywhere through the miracle of Kubernetes, if you'll pardon the dismissive scorn there, that. You just package up your application, shove it into a container, and then hurl it from the application team over to the operations team like a dead dog cast into your neighbor's yard for him to worry about.
And then it just sort of takes a, it sort of takes up the space of you don't have to manage state anymore because everything is mostly stateless in [00:07:00] theory. How have you seen it play out in practice in the last five years?
Nigel Kersten: I mean, that's a real trend. And you know, like, the size of a container should be smaller than an operating system.
And the reality is, you know, like, I'm a sysadmin. I love operating systems. I nerded out on operating systems. They're a necessary evil. They're terrible, terrible things. Registry keys, config files, you know, like, they're a pain in the neck to deal with. And if you look at I think what a lot of operations folks missed about Docker when it started was that it didn't make their life better.
It was worse. It was like this actual sort of terrible tool chain where you sort of tied together all these different things. But really importantly what it did is it put Control into the hands of the developers. And it was the developers who were trying to do stuff, who were trying to shift applications.
And I think Docko is a really great technology. In the sense of, you know, developers could ship value on their own. And that was the huge, huge leveling up. You know, it wasn't the interface. It wasn't the user experience. Wasn't all these things. It was just that the [00:08:00] control got taken away from the IT trolls in their basement going, No, don't touch my servers.
And instead given straight to the developers. And that's huge because it let us ship things faster. And that's ultimately the whole goal of things.
Corey Quinn: The thing that really struck me the most from conducting the trainings that I did was meeting a whole bunch of people across the country in different technological areas of specialty, in different states of their evolution as technologists.
And something that struck me was just how much people wound up identifying the with the technology that they worked on. When someone is the AIX admin and the AIX machines are getting replaced with Linux boxes, there's this tendency to fight against that and rebel rather than learning Linux. And I get it.
I'm as subject to this as anyone is. And in many cases, that was the actual pushback that I saw against adopting something like Puppet. If I identify my job as being the person that runs all of these [00:09:00] But carefully curated scripts that I've spent five years building. And now that all gets replaced with something that is more of a global solution to my local problem, then it feels like the thing that made me special is eroding.
And we see that with the migration to cloud as well, when you're the storage admin and it just becomes an API call to S3, that's kind of a scary thing. And when you're one of the server hugger types, again, as guilty as anyone of this, and you start to see cloud coming in as like a rising tide that eats up what it was that you became known for, it's scary.
And it becomes a foundational shift in How you view yourself. What I really had a lot of sympathy for was the folks who'd been doing this for 20 years. They were, in some cases, a few years away from retirement. And they'd been doing basically the same set of tasks every year for 25 years. It's one year of experience repeated 25 times.
And they don't have that much time. time left in their career, intentionally, so they want to retire, but they also don't really want to learn a whole bunch of [00:10:00] new technologies just to get through those last few years. I feel for them, but at the same time Me too,
Nigel Kersten: totally. But what are you going to do? But without sounding sort of too dismissive there, like, I think it's a natural tendency for us to identify with the technology when, if that's what you're around all the time, you know, mechanics do this, truck drivers with brands of trucks, people like build attachments to the technology they work with, because we fit them into this bigger Techno social system.
But I have a lot of empathy for the people in enterprise jobs who are being asked to change radically because the cycle of progress is speeding up faster and faster. And as you say, like they might be a few years away from retirement. I think I used to feel a lot more differently about this when I was really hotheaded and, you know, sort of much more of a tech enthusiast.
And that's what I identified with in terms of, you know, it's okay for a job to just be a job for people. It's okay for someone to be doing a job because They get good health care and good benefits and it's feeding their family. Like, that's an important thing. You can't expect everyone to always [00:11:00] be incredibly passionate about technology choices in the same way that I think, you know, many of us who live, live on Twitter and hang out in this space are.
Corey Quinn: Oh, I have no problem whatsoever with people who want to show up for 40 hours a week ish, work on their job, and then go home and have lives and not think about computers at all. There's this dark mass of developers out there that basically never show up on Twitter, they aren't on IRC, they don't go to conferences.
And that's fine. I have no problem with that. And I hope I don't come across as being overly dismissive of those folks. I honestly wish I could be content like that. I just don't hold still very well.
Nigel Kersten: But yeah, so I think you touched on a few interesting things there. And some of those we sort of cover in the state of DevOps report, which is coming out in the next few weeks.
Indeed. And the state of DevOps report
Corey Quinn: is started off at Puppet. And they've now done it for, what,
Nigel Kersten: 10 years? This is the 10th year, which is completely crazy. So I was looking at the stats [00:12:00] as I was writing it and it's 10 years of stated DevOps reports. I think it's 11 years of DevOps Weekly, Gareth Roshgrove's newsletter.
It's 12 or 13 years of DevOps days that have been going on. You know, this is longer than I spent in primary and high school put together. It's kind of crazy that the DevOps movement's still kind of chugging along, even if it's not necessarily the coolest kid on the block. Now that, you know, GitOps, SRE, Flavor of the Month, various kinds of permutations of how we work with technology have perhaps got a little bit cooler, but it's still very, very relevant to a lot of enterprises out there.
Corey Quinn: Yeah, as I frequently say, legacy is a condescending engineering term for it makes money, and there's an awful lot of that out there. Forget cloud, there are still companies wrestling with, do we explore this virtualization thing? And that was something I was very against back in 2006, let's be very honest.
I am very bad at predicting the future of technology. [00:13:00] And I can see it for small niche edge workload cases, where you have a bunch of idle servers. But for the most part, who's really going to use this in production? Well, basically everyone, because that in turn is what the cloud runs on. Yeah, I think we can safely say I got that one hilariously wrong.
But hey, if you aren't going to make
Nigel Kersten: predictions, then what's the matter? But the industry pushes you in these directions. So there was this massive bank in Asia who I've been working with for a long time, and they were always resistant to adopting virtualization. And then it was only four or five years ago that I visited them, they were like, right, we're Okay, it's time, we're rolling out VMware.
And I was like, so I'm really curious, what exactly changed in the last year or two, in like 2014, 2015, that you decided virtualization was the key? And I'm like, Oh, there was
Corey Quinn: this jackwagon who conducted this training. Yeah, no, no, sorry. I can't fend credit for that one.
Nigel Kersten: They couldn't order one rack unit servers with CD drives anymore because their whole process was actually provisioning with CDs before that point.
Welcome to the brave new world
Corey Quinn: [00:14:00] of pixie booting, which is kind of hard. So yeah, virtualization is easier. I, you know, sometimes people have to be dragged into various ways of technological advancement, which gets to the real thing I want to cover. Since this is a promoted episode where you're talking about the state of DevOps report, I'm almost less interested in what this year's has to say specifically than what you've seen over the last decade.
What's changed? What was true 10 years ago that is very much not true now? Bonus points, you can answer that without using the word Kubernetes more than twice.
Nigel Kersten: So I think one of the big things was that we've definitely passed peak DevOps team. Like if you may remember, there was a lot of arguments and there's still regular, is DevOps a job title?
Is it a team title? Oh, I was very much on the
Corey Quinn: no side until I saw how much more I would get paid as a DevOps engineer instead of a systems administrator for the exact same job. So, you know, I shut up and I took the money. I figured that the semantic arguments are great, but. Yeah,
Nigel Kersten: and that's exactly what we've, we've written in the [00:15:00] report and I think it's great.
The sysadmins, you know, we were unloved, you know, we were in the basement, we weren't paid as much as programmers. The running, you know, the joke used to be for developers DevOps meant I don't need ops anymore, but for ops people it was I can get paid like a developer.
Corey Quinn: In many cases, oh, well, systems administrators don't want to learn how to code.
It's Yeah, you're remembering a relatively narrow slice of time between the modern era where systems administrator types need to be able to write in the lingua franca of everything, which is of course YAML, as far as programming languages go. And before that, to be a competent systems administrator, you needed to have a functional grasp of C.
And there is only a limited window in which a bunch of bash scripts and maybe a smidgen of Perl would have carried you through. But the deeper understanding is absolutely necessary, and I would argue always has been.
Nigel Kersten: And this is great because you've just linked up with one of the things we found really interesting about the report is that, you know, when we talk about legacy.
We don't actually [00:16:00] mean the oldest shit, because the oldest shit is the mainframes, it's a lot of bare metal applications, a lot of that in big enterprises. We're still waiting for an AWS 400 to replace some of that. Well, it's administered by real systems engineers, you know, like the people who wrote C, who wrote kernel extensions, who could debug things.
What we actually mean by legacy, is we mean late 90s to late 2000s, early 2010s, stuff that was put together by kids who, like me, happened to get a job because you grew up with a computer and then the dot com explosion happened. You weren't necessarily particularly skilled and a lot of people were, they didn't go through the apprenticeships that mainframe folks and systems engineers actually went through and everyone just held this stuff together with, you know, duct tape and dental floss and then now we're paying the price of it all.
Like way back down the track. So legacy is really just a certain slice of rapid growth in applications and infrastructure. That's sort of an unmanageable mess now.
Corey Quinn: Oh, here in San [00:17:00] Francisco, legacy is anything prior to last night's nightly bill. It's turned into something a little ridiculous. I feel like the real power move as a developer now is to get a job, go in on day one, rebase everything in the Git repository to a single commit with a message, legacy code, and then follow that up.
And that's the power move and that's how it works. And that's also the attitude we wind up encountering in a lot of places. And I don't think it serves anyone particularly well to tie themselves so tightly to that particular vision.
Nigel Kersten: Yep, absolutely. But this is a real problem, this space. And one of the things we found in the Stead Devils report is that Let me back up a little and give a little bit of methodology of what we actually do.
We survey people about their sort of performance metrics, you know, like how quickly can you do deploys? What's your sort of mean time to recovery? Those sorts of things. And what practices do you actually employ? And we essentially go through and do statistical analysis on this and everyone tends to end up in three cohorts.
They separate pretty easily of low, medium and high evolution. And so one of the things we found is that [00:18:00] everyone at the low level has all sorts of problems. They have issues with what does my team do? What does the team next to me do? How do I talk to the team next to me? How do I actually share anything?
How do I even know what my goals are? Like fundamental company problems. But everyone at all levels of evolution is stuck on two big things, not being able to find enough people with the right skills for what they need and their legacy infrastructure holding them back.
Corey Quinn: The thing that I find the most compelling is the idea of not being able to find enough people with the skills that they need.
And I'm going to break my own rule and mention Kubernetes as a prime example of this. If you are effective at managing Kubernetes in production, you will make. A very comfortable living in any geographical location on the planet, because it is incredibly complex. And every time we've seen this In previous trends, where you need to get more and more complexity and more and more expertise just to run something, it looks like a sawtooth curve, [00:19:00] where at some point that complexity gets abstracted away and compressed down into something that is basically a single line somewhere, or it happens below the surface level of awareness.
My argument has been that Kubernetes is something no one's going to care about in roughly three years from now. Not because we're not using it anymore, but because it's below the level of awareness that we have to think about. In the same way that there aren't a whole lot of people on the planet these days who have to think about the Linux virtual memory management subsystem.
It's there and a few people really care about it, but for the rest of us, we don't have to think about that. That is the infrastructure underneath our infrastructure.
Nigel Kersten: Absolutely. I used to make a living, and it's ridiculous looking back at this for a year or two. Doing high performance custom compiled apaches.
I was really, really good at this.
Corey Quinn: Well, yeah, Apache is a great example of this. Where, back in the 90s, to get a web server up and running, you needed to have three days to spare, and in depth knowledge of GCC compiler flags, and hope for the best. And then RPM came out. And then, [00:20:00] okay, then YUM, or other things like that on top of it.
And then things like Puppet started showing up. And we saw, oh great, now Insure installed. Great. And then we took a step beyond that, and it was, oh, now it's just a Docker run, whatever it is, and these days, yeah, it's a checkbox in S3.
Nigel Kersten: So, so let me make, get your Kubernetes prediction down, right. So you're not, you're predicting Kubernetes is going to go away like, you know, Apache and highly successful things.
It's not an open stack failure state. It's a Apache invisibility state.
Corey Quinn: Absolutely. My timeline is a bit questionable. Let's be fair, but it's on the aggressive side, but yeah, I think that Kubernetes is inherently too complex for most people to have to wind up thinking about it in that way. And. We're not talking small copies, we're talking big ones, where you're not in a position, if you're a giant blue chip fortune 50, to hire 2, 000 people who all know Kubernetes super well, and you shouldn't have to.
There needs to be some [00:21:00] flattening of all of that High level complexity. Without the management tools though, with things like Puppet and the things that came before in a bunch of different ways, we would all not be able to get anything done because we'd be too busy writing and assembly. There's always going to be those abstractions abstractions, and very few people understand how it works all the way down, but that's in many cases okay.
Nigel Kersten: That's civilization, you know? Like, do you understand what happens when you plug in something to your electricity socket? I don't want to know. Like, I just want light.
Corey Quinn: And more to the point, whenever you flip the switch, you don't have that doubt in your mind that the light is going to come on. So if it doesn't, that's notable.
And your first thought is, oh, I The light bulb is out, not the utility company is down. And when we talk about the cloud being utility computing
Nigel Kersten: Has someone put a Kubernetes operator in this light switch that may break this process?
Corey Quinn: Well, okay, IoT does throw a little bit of a crimp into those words, but yeah.
Let's talk more about the state of DevOps report. What notable findings were there this year?
Nigel Kersten: So one of the big things that we've [00:22:00] seen for the last couple of years has been that most companies are stuck in the middle of the evolutionary progress. And, you know, anyone who deals with large enterprises knows this is true.
Like whatever they've adopted in terms of technology, in terms of working methods, you know, agile, various different things. Most companies don't tend to advance to the higher levels. Most places stay mired in mediocrity. So we wanted to dive into that and try and work out why are most companies actually stuck like this when they hit a certain size.
And it turns out the problems aren't technology or DevOps. They're like really fundamental problems like We don't have clear goals. I don't understand what the teams next to me do. We did a bunch of qualitative interviews as well as the quantitative work in the survey with this report and we talked to one group of folks at a pretty large financial services company who were like, Our teams have all been renamed so many times, if I need to go and ask someone for something, I literally page up and down through ServiceNow trying to find [00:23:00] out where to put the change request and they're like, how do I know where to put a network port opening request for this particular service when there are 20 different teams that might be named the right thing and some are obsolete and I get no feedback whether I've sent it off to the right thing or to a black hole of enterprise despair.
Corey Quinn: Here at the Duck Bill Group, one of the things we do with, you know, my day job is we help negotiate AWS contracts. We just recently crossed 5 billion of contract value negotiated. It solves for fun problems such as how do you know that your contract that you have with AWS is the best deal you can get?
How do you know you're not leaving money on the table? How do you know that you're not doing what I do on this podcast and on Twitter constantly and sticking your foot in your mouth? To learn more, come chat at duckbillgroup. com. Optionally, I will also do podcast voice when we talk about it. Again, that's duck bill group.com.[00:24:00]
That doesn't get better with a lot of modernization. I mean, I feel like half of my job, and I'm not exaggerating, is introducing Amazonians to one another. Corporate communication between departments and different groups is very far from a solve problem. I think the tooling can help, but I've never been a big believer in solving political problems with technology.
It doesn't work. People don't work that way.
Nigel Kersten: Absolutely, like one of my earliest times working at Puppet doing sort of higher level sales and services and support. Huge national telco walk in there, we've got the development team, the QA team, the infrastructure team. In the course of this conversation, one of them makes a comment about using apt get.
And the others were like, what do you mean? We're on RHEL. And it turned out production was running on RHEL, the QA team running on CentOS, and the developers were all building everything on Ubuntu. And because it was Java apps, they almost didn't have to care, but you know, write once, debug everywhere.
History doesn't [00:25:00] repeat,
Corey Quinn: but it rhymes. Before Docker, so much of development and startup land was, how do I make my MacBook Pro look a lot more like an EC2 Linux instance? And it turns out that there's an awful lot of work that goes into that, that maybe isn't the best use of people's time. And we start to see these breakthroughs and these revelations in a bunch of different ways.
I have to ask, this is the 10th year that you've done the State of DevOps report. At this point, why keep doing it? Is it inertia? Are you still discovering new insights every year on top of it? Or is it one of those things where, well, someone in marketing says we have to do it, so here we are?
Nigel Kersten: No, actually, it's not that at all.
So definitely we're going to take stock after this year because 10 years feels like a really good point to sort of, it's a nice round number and, you know, a certain kind of number system. Mainly the reason is, you know, a lot of my job is going and helping big enterprises just get better at using technology.
And it's funny how often, like, I just get folks going, Oh, I read this thing. Like, [00:26:00] people who aren't on, you know, the bleeding edge, constantly discussing these things on Twitter or whatever, but the Stead DevOps support makes its way to them. And they're like, Oh, I read a thing there about how, how much better it is if we standardized on one operating system.
And that made a really huge difference to what we were actually doing, because you had all this data in there showing that that is better. And honestly, that's the biggest reason why I end up doing it. You know, it's the fact that, It seems to be a tool that has made its way through to very hard to penetrate enterprise folks.
And they'll read it and managers will read things like that are like if you set clear goals for your team and Get them to focus on optimizing the legacy environment You will see returns on it and like I'm being a little bit facetious in the tone that I'm saying because a lot of this Stuff does feel obvious if you're you know You know, constantly swimming in this stuff day to day.
But it's not just the practitioners who it's just a job for in a lot of big companies. It's true of a lot of the management chain as well. They're not necessarily going out and reading up on, you know, modern agile [00:27:00] IT management practices day to day for fun. You know, they go home and do something else.
Corey Quinn: One
Nigel Kersten: of
Corey Quinn: my favorite conferences is Gene Kim's DevOps Enterprise Summit. And the specific reason behind that is these are very large companies that go beyond companies in some cases to institutions where you have the US Air Force as a presenter one year, and very large banks that are 200 years old.
Every other conference, it seems, more or less involves people getting on stage to deliver conferenceware and tell stories that make people at those companies feel bad about themselves. Where it's, we're Twitter for pets and this is how we deploy software. Or the ever popular, this is how Netflix does stuff.
Yeah, Netflix has basically no budget constraints as far as hiring engineering folks go, and lest we forget, their failure mode is someone can't watch a movie right now. It's not exactly the same thing as the ATM starts spitting out the wrong balance in the streets. [00:28:00] And I think that there's an awful lot of Discussion where people look at the stories people tell on conference stages and come away feeling bad from it.
Very often I'll see someone from a notable tech company talk about how they do things and, wow, I wish my group did things like that. And the person next to me says, yeah, me too. And I check and they work at the same company. And it's, Like, the stories we tell are not necessarily the stories that we live.
And it's very easy to come away discouraged from these things. And that goes triply so for large enterprises that are regulated, that have significant downside risk if the technology fails them. And I love watching people getting a chance to tell those stories.
Nigel Kersten: Let me jump on that really quickly. Please, by all means.
One is, you know, having done four years at Google, things are a shit show internally there too. You talk about it like it's prison. I like it. Like People get horrified when they turn up and they're like, Oh, what? It's not all gleaming, perfect software artifacts, you know, delivered from the hand of Errs.
But I think what Gene has done with DevOps Enterprise Summit is fantastic and how people share [00:29:00] more openly their failure states. But even there, and this is an interesting result we found from a few years ago, State Dev Support, even those executives are being more optimistic because it's sort of beaten into you as a senior executive.
You're putting on a public face. And even when they're trying to, Share the warts and all story. They can't help but put a little bit of a positive spin on it, because I've had exactly the same experience there, where someone's up there telling a war story, and then I look, turn to the person next to me, and they work at that same 300 year old bank.
And they're like, actually it's much much worse than this and we didn't fix it quite as well as that. So I think, you know, the big tech companies are terrible inside, unless they're Netflix. And the big enterprises are also terrible, but they're also No, no, I've
Corey Quinn: talked to Netflix people too. They do terrible things internally there too.
No one talks about the fact that their internal environments are always tire fires. And there are two stories, the stories we tell publicly and the reality. And if you don't believe me on that, look at any company in the world's billing system. As much as we all talk [00:30:00] about Agile and various implementations thereof, when it comes to things that charge customers money, we're all doing Waterfall.
Nigel Kersten: Absolutely.
Corey Quinn: Because mistakes show when you triple charge someone's credit card for the cost of a small country's GDP. It's a problem. I want to normalize those sorts of things more. I'm looking forward to reading this year's report just because it's, it's interesting to see how folks who are in environments that differ from the ones that I get to see experience this stuff and how they talk about it.
Nigel Kersten: Yeah. And so one of the big results I think there for big companies that's really interesting is that. One of the sort of anti patterns is having lots of different types of teams. And I kind of touched on this before about, you know, having confusing team titles being a real problem and not being able to cross organizational boundaries quickly is really, really, you know, it's a huge inhibitor and cause source of friction.
But it turns out the pattern that is actually really great is one that the team topologies guys have discovered. If you've been following what Matthew Skelton and Manuel Paez [00:31:00] have been doing for a while, they've basically been documenting a pattern in software organizations of a small number of team types of a platform team, value stream teams, complicated subtest system teams, and enabling teams.
And so we worked with Manuel and Matt on this year's report and asked a whole bunch of questions to try and validate the team topologies model. And the results came back and they were just incredibly strong. Because I think this speaks to some of the stuff you mentioned before that no one can afford to hire an army of Kubernetes developers.
And whatever the hottest technology is in five years, most big companies can't hire an army of those people either. And so the way you get scale internally before those things become commoditized is you build a small team and create the situation where they can have outsized leverage inside their organization.
Like, get rid of all the blockers to fast flow and make their focus self service to other people. Because if you're making all of your developers learn distributed systems operations arcane knowledge, [00:32:00] like, that's not a good use of their time either.
Corey Quinn: It's really not. And I think that that's something that gets lost a lot is, I've never yet seen a company beyond the very early startup stage where the AWS bill exceeded the cost of the people working on the AWS bill.
Payroll is always a larger expense than infrastructure unless you're doing something incredibly strange. And, uh, Oh, I want to save some money on the cloud bill is very often offset by the sheer amount of time that you're going to have to pay people to work on that because contrary to what we believe is engineering hobbyists, people's time is very far from free.
And it's also the opportunity cost of if you're going to work on this thing instead of something else, well, is that really the best choice? It comes down to contextualizing what technology is doing as well as with what's happening over in the world of business strategy. And without having a bridge between those, it doesn't seem to work very well.
Nigel Kersten: Absolutely. It's insane. Like, it's literally insane that as an industry we will optimize 5%, [00:33:00] 3 percent of our, you know, infrastructure build or application workload and yet not actually re examine business processes that are causing your people to spend 10 percent of their time in synchronous meetings.
You know, like you can save so much more money and achieve so much more by actually optimizing for fast flow and getting out of the way of the people who cost lots of money. So one last topic
Corey Quinn: that I want to cover before we call it an episode. You talk to an awful lot of folks and it's easy to point at the aspirational stories of folks doing things the right way.
But let's dish for a minute. What are you seeing in terms of people not using the cloud properly? I feel like you might have a story or two on that one.
Nigel Kersten: I do have a few stories. So in this year's report, one of the things we wanted to find out of, like, are people using the cloud sort of, you know, in the way we think of cloud, you know, elastic, consumption based, all of these sorts of things.
We use the NIST metrics, which I recognize can be a little controversial, but I think you've got to start somewhere as a certain foundation. It turns out just about everyone is [00:34:00] using the public cloud. And when I say cloud, like, I'm not really talking about people's internal VMware that they've rebadged as cloud.
Like I'm talking about. the public cloud providers. Everyone's using it, but almost no one is taking advantage of the functionality of the cloud. They're instead treating it like an on premise VMware installation from the mid 2000s. They're taking six weeks to provision instances, they're importing all of their existing processes, they keep these things running for a long time.
If they fall over, they're going to have to re install themselves. One person is like tasked with, hey, do you know how pet number 45 is actually doing here? They're not really treating any of these things in the way that they're actually meant to. And I think we forget about this a lot of the time when we talk about cloud because we jump straight to cloud native, you know, the sort of bleeding edge of folks and serverless, highly orchestrated containers.
I think if you look at the actual numbers, the vast majority of cloud usage, it's still things like EC2 instances on AWS. And there's a reason because it's a familiar paradigm for people. [00:35:00] We're definitely going to progress past there, but I think it's easy to leave the sort of You know, the people in the middle behind when we're talking about cloud and how to improve the ecosystem that they all operate in.
Corey Quinn: Part of the problem too is that whenever we look at how folks are misusing cloud, it's easy to lose sight of context. People don't generally wake up and decide I'm going to do a terrible job today unless they work in, you know, Facebook's ethics department or something. Instead, it's very much a people are shaped by the constraints they're laboring under from a bunch of different angles and they're trying to do the best with what they have.
Very often, the reason that a practice or a policy exists is because once upon a time there was a constraint that may or may not still be there and going forward the way that they have seemed like the best option at the time. I found that the default assumption that people are generally smart and doing the right thing with the information they have carries you a lot further.
in many respects, than what I did as a terrible junior consultant, which is, oh, what moron built this, uh, invariably to said moron, [00:36:00] and then the rest of the engagement rapidly goes downhill from there. Try and assume good faith, and instead when you see something that makes no sense, ask, why is it like this?
Rather than, why is it like this?
Nigel Kersten: Counts for a lot. It's the fundamental attribution bias. You know, it's why we think all other drivers on the road are terrible, but we actually had a good reason for swerving into that lane. This isn't how I would
Corey Quinn: have built it, so it's awful. Yeah, exactly. And in some cases though, there are choices that are objectively bad, but I try to understand where they came from there.
Company policy historically around things like data centers trying to map one to one to cloud often miss some nuances. But hey, there's a reason it's called a digital transformation, not a project that we did.
Nigel Kersten: And I think you've got to always have empathy for the people on the ground. I quite often have talked to folks who've got like a terrible cloud architecture with the deployment.
I'm like, well, what happened here? And they went, well, we were prepared to deploy this whole thing on AWS. But then Microsoft salespeople got to the CTO and we got told at the last minute [00:37:00] we're redeploying everything on Azure. And so these people were often, you know, you're given a week or two to pivot around a decision that doesn't necessarily make any sense to them.
And there may have been a perfectly good reason for the CTO to do this. They got given really good kickbacks, you know, like in terms of bonuses for like how much they were spending on the infrastructure. I mean discounts. But people on the ground are generally doing the best with what they can do. They end up building crap.
It's because, you know, our system, society, capitalism, everything else is at fault.
Corey Quinn: I have to say, I'm really looking forward to seeing the observations that you wound up putting into this report as soon as it drops. I'm hoping that I get a chance to speak with you again about the findings and then I can belligerently tell you to justify yourself.
That was my favorite follow ups. Brilliant. If people want to get a copy of the report for themselves or learn more about you, where can they find you?
Nigel Kersten: Just head straight to Puppet. com and it will be on the banner on the front of the site.
Corey Quinn: Excellent. And we'll, of course, put a link to that in the show notes, if people can't [00:38:00] remember puppet.
com. Thank you so much for taking the time to speak with me. I really appreciate it. Awesome. No worries. It was good to catch up. Nigel Kirsten, Field CTO at Puppet. I'm cloud economist, Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five star review on your podcast platform of choice.
Whereas if you've hated this podcast, please leave a comment. Leave a 5 star review on your podcast platform of choice, as well as an insulting comment telling me that Comcastic isn't a funny word, and tell me where you work, though we already know.
