Chef Gets Gobbled Up (Whiteboard Confessional)

Corey: This episode is sponsored in part by Catchpoint look, 80% of performance and availability issues don't occur within your application code in your data center itself. It occurs well outside those boundaries. So it's difficult to understand what's actually happening. What Catchpoint does is makes it easier for enterprises to detect, identify, and of course validate how reachable their application is. And of course, how happy their users are. It helps you get visible and to reach a bit availability, performance, reliability, of course, absorbency. Cause we'll throw that one in too. And it's used by a bunch of interns and companies you may have heard of like, you know, Google, Verizon, Oracle, but don't hold that against them. And many more. To learn more, visit www.catchpoint.com and tell them Cory sent you, wait for the wince.

Welcome to the AWS Morning Brief: Whiteboard Confessional, now with recurring perpetual guest, Pete Cheslock. Pete, how are you?

Pete: I'm back again.

Corey: So, today I want to talk about something that really struck an awful lot of nerves across, well, the greater internet. You know, the mountains of thought leadership, otherwise known as Twitter. Specifically, Chef has gotten itself acquired.

Pete: Yeah, I saw some, I guess you would call them, sub-tweets from some Chef employees before it was announced, which is kind of common, where responses ranged from, “Oh, that's something new,” to, “Welp.” And I've thought it—I was like, “Wow, that's interesting.” Of course, then I start looking for news of what happened, of which we all found out not long after.

Corey: Before we go into it, let's set the stage here because it turns out not everyone went through the battles of configuration management circa 2012 to 2015 or so—at least in my experience. What did Chef do? What was the product that Chef offered? Who the heck are they?

Pete: So, Chef, they were kind of a fast follower in the configuration management space to another very popular tool that I'm sure people have used out there called Puppet. Actually, interestingly enough, the founders of Chef ran a consulting company that was doing Puppet consulting; they were helping companies use Puppet. And both of those tools really came from yet another tool called CFEngine, which in many ways—depending on who you ask—it's kind of considered the original configuration management, the one that had probably the earliest, largest usage. But it was very difficult to use. CFEngine was not something that was easy, it had a really high barrier to entry, and tools like Puppet and Chef, they came out around the, let's say 2007, 8, 9 10 timeframe, were written in Ruby which was a little bit easier of a programming language to get up and running with. And this solved a problem for a lot of companies who needed to configure and manage lots of servers easily.

Corey: And there are basically four companies in here that really nailed it for this era; you had Puppet, Chef, Salt, and Ansible. And in the interest of full disclosure, I was a very early developer behind SaltStack, and I was a traveling contract trainer for Puppet for a while. I never got deep into Chef myself for a variety of reasons. First and foremost was that its configuration language was fundamentally Ruby, and my approach back then—because I wasn't anything approaching a developer—was that if I need to learn a full-featured programming language at some point, well, why wouldn't I just pivot to becoming, instead, a developer in that language and not have to worry about infrastructure? Instead, go and build applications and then work nine to five and not get woken up in the middle of the night when something broke. That may have been the wrong direction, but that was where I sat at the time.

Pete: Yeah, I came at it from a different world. So, I had worked for a startup that no one has probably really ever heard of, unless you have met me before, like, know who I am, but a company called Sonian which was very early in the cloud space. It was email archiving, so it wasn't anything particularly mind-blowingly interesting because it's compliant email archiving, but what was interesting is that we were really early in the cloud space, and a lot of the tools that people use today just didn't exist for managing cloud servers. It was 2008, 2009, pretty early, EC2 timeframe. How would you provision your EC2 instance, back then? Maybe you use CFEngine, maybe use Puppet. 

And actually, interestingly enough, that company—Sonian—was originally a Puppet shop because Chef didn't exist yet. And there were a series of issues we ran into, technical capabilities that Puppet just couldn't do for us at the time. And again, that time being 2009, 2010, and a lot of the very early Chef team, founding team, early engineers, were really working with us very closely to bootstrap our business on Chef writing a lot of those original cookbooks that became community cookbooks. And so, my intro into Chef and the Chef community is a lot earlier than most, and I went a lot deeper with it just by nature of being so early into that space.

Corey: One of the things that struck me despite not being a Chef aficionado myself was, first, just how many people in the DevOps sphere were deeply tied into that entire ecosystem. And two, love or hate whatever the product, or company, et cetera, did, some of the most empathetic people I've ever met were closely tied to Chef’s orbit. So, I have not publicly commented until now on Chef getting acquired, just because I'm trying to give the people who are in that space, time to, I guess, I don't know if grieve is the right word, but it's important to me that I don't have a whole lot to say there, and it's very easy for me to say something that comes across as crass, or not well thought out, or unintentionally insulting to a lot of very good people. So, I'm sitting here on the sidelines watching it and more or less sitting this one out, but it's deeply affected enough people that I wanted to talk about it here.

Pete: Yeah. And I'm glad that we are taking this opportunity to talk about it a bit. I had a lot of thoughts and feels. I tried to write a blog post about this to try to get them down somewhere, and a couple of paragraphs into it, I just, I really couldn’t… it just seemed like a meandering random mess of words without any real destination. But a few people online have mentioned this, and I'll definitely call it out as well, which is that Chef was, it was a tool. It was a tool like any other. You either loved it or you hated it. If you hated it, you probably really loved Ansible, or you really loved Puppet. It was a really, kind of, Vim versus Emacs feel to it, where you either we're all in on it or not.

But the thing that I think Chef really brought for me is not only leveling up my career in a way that I would not be where I'm at today if it wasn't for that tool and that community, but just how genuine everyone was within that community, and the interactions that we had at conferences, at Chef conferences, DevOps conferences and things of that nature, and even continued the conversations online back before Slack, which it's hard to even remember that: when we all were on IRC, and we were in the Chef IRC channel, and it was a fantastic channel with a ton of people who would dive in and help you out on your Chef problems. But I think the biggest thing that tools like Puppet and Chef really helped a lot of struggling sysadmins like myself is it got you into programming in a really safe, easy way. When you wrote a Chef cookbook, you were actually writing Ruby. It was its own custom DSL for Ruby, but what was great about it is that you could translate what you were writing in Chef, and how Chef operated into, then, other tools. And I really attribute a lot of those early tools that came out—Sensu was written in Ruby, Vagrant was written in Ruby, imagine all of those early DevOps tools—again, before Golang—were written in Ruby, and I wonder if it's because of tools like Chef and Puppet, which were also in that language, and it was now very approachable for people to build in.

Corey: I would argue at this point that if I take a look now through the lens of, you know, the last 15 years or so I spent in this space, with the benefit of that hindsight, Chef was clearly head and shoulders above its competitors and a lot of different ways. I feel like their, more or less, acquisition if we view it as a loss—which it appears that a lot of people do, so I'm going to defer to the wisdom of crowds there—is fundamentally because of that the fact that the industry as a whole pivoted away from running these long-lived instances that need to have their configuration updated on a consistent, ongoing basis. Containers and stateless architecture, more or less, have eaten the world. The stateful things like databases are increasingly being handled as managed services, and all these companies are open-source projects then, with enterprise offerings. And okay, great. We have to manage our database servers for those exception cases with Chef. That's great. A lot of those shops find that the open-source version of these products is more than sufficient for their needs because, let's face it, the enterprise versions are not inexpensive.

Pete: Exactly, I think there’ll be a lot of armchair quarterbacking on why Chef didn't succeed like it should have. I will also armchair quarterback and tell you that my thoughts on it is that—and this is not an original thought in any way. I have very few original thoughts—but open source is not a business model. And one thing that I—I think I put this on Twitter last year at some point when Docker had a similar end—actually Chef had a more successful end than Docker—but Docker raised obscene amounts of money, hundreds of millions of dollars, and they were sold for like assets, right? They basically were just dissolved. 

What was most shocking about Docker essentially going away—I mean, obviously, the open-source will be around forever, but Docker was the thing that really ate Chef's lunch. If you didn't need to really care about the server anymore because you were just deploying your applications within these containers, then you don't actually need Chef anymore. So, Docker was really successful at destroying Chef—and arguably Puppet’s—business without ever actually figuring out a way to make money itself. And to me, that is one of the most spectacular things ever.

Corey: This episode is sponsored in part by our good friends over a ChaosSearch, which is a fully managed log analytics platform that leverages your S3 buckets as a data store with no further data movement required. If you're looking to either process multiple terabytes in a petabyte-scale of data a day or a few hundred gigabytes, this is still economical and worth looking into. You don't have to manage Elasticsearch yourself. If your ELK stack is falling over, take a look at using ChaosSearch for log analytics. Now, if you do a direct cost comparison, you're going to say, “Yeah, 70 to 80 percent on the infrastructure costs,” which does not include the actual expense of paying infrastructure people to mess around with running Elasticsearch themselves. You can take it from me or you can take it from many of their happy customers, but visit chaossearch.io today to learn more.

Corey: I would absolutely agree with that. I think that the entire industry shifted in a terrific way that makes things a lot more accessible. You don't need to be a sysadmin anymore to build something intelligent and workable, and for better or worse, in the way that modern architectural practices have evolved, there just isn't space for a configuration management company in the several billion-dollar valuation range.

Pete: Exactly. And you look at the history of those companies, Ansible sold to Red Hat pretty early in their life. To be honest, I'm not sure what happened to Salt. You might know better than me; I was never really that close in that community. Chef and Puppet, they just hung around, and they had real businesses. 

The documentation from Progress, who was the acquirer of Chef, had claimed that they were doing about $70 million of recurring revenue. That's no slouch. That is good revenue that a lot of companies would love to see. The interesting part of that acquisition is that they were only sold for $220-ish million dollars. That's not a big multiple on top of a $70 million run rate, and it's definitely only twice what their investment was in. 

This is something I've written about in the past around how startups are funded. There usually is a concept of when investors put money in for their preferred shares, there's a concept of participating preferred, which I'm going to give not any detail at all; I recommend, highly, reading smarter people than me online who talk about it. But at a really, really basic level, it allows those VCs to take out the money they put in off the top before splitting the rest of the pie. So, if you had 100 million dollars in and all of those VCs take out their portion first, now out of a $200 million valuation, you only have $100 million left to split. Of course, you'll be splitting with those same VCs who already took their money out first, as well as everyone else who has shares, so it's something that you'll see in pretty much every startup’s capitalization table, but it causes common shareholders, like the employees, many of which who’d been there five to ten, years to see a lot less for their shares. 

And likely if you were an employee of Chef in the last few years, my guess is—because just the nature of VC funding—you'll have a valuation of your last round, it's hard for me to believe that those investors who put money in the last round, the price per share was probably a lot higher than the exit price. But due to this participating preferred model, they're able to at least not lose their money. Maybe they got a 2x return back because they got their money off plus the split, but again, as outsiders it's hard for us to know exactly who won on this one in the grand scheme of things.

Corey: There are too many stories in the world. Of employees who wind up not getting any form of value out of the exit that they wind up going through. There are so many different things that can happen. Finance is complicated, to be direct. A number of—I would argue most engineers are not sophisticated in the realm of corporate finance. 

And that's okay; I'm not saying that they should be. There are a lot of problems in that space, and I've always viewed equity with a healthy grain of salt because I don't consider myself any different. When I was in my employment stage of my career, I was absolutely not possessed of a sophisticated understanding of these financial aspects. And even if I were, the investors on the other side of the table, well, I'm doing one of those deals, maybe once every two or three years, they're doing three or four a week. They're always going to be better at this than I am because that's what they do. And that's okay.

Pete: Exactly. For every Datadog, for every Elastic, for every one of those companies, those startups that raised some money and IPO’d for billions of dollars, there's hundreds of Chefs; there's hundreds of companies that make it to this danger level where they raise a lot of money, but their revenue is just not growing because the market has shifted so fast. You can either raise under worse terms, or you can try to take an air quote, “Exit,” to just be done with it; to be able to get something for it. And then for all of those Chefs and whatever, those companies in the middle, there's thousands of startups that don't even get to the level like a Puppet or a Chef did. 

And I know that. I’ve worked for pretty much all of them at this point, where you get to the danger zone, you raise a Series B, a Series C, now you've raised 50 million, a 100 million dollars. You need to get to half a billion or a billion dollars a year in revenue—the numbers, it all just get so high. It's just the downside of VC: as the multiples get higher and higher, and harder and harder to hit, and thus, maybe companies make desperate decisions or short-sighted decisions that causes them, maybe, long term pain. Again, this is not easy at all, and I don't claim to be an expert on it. These are hard, hard problems.

Corey: Yeah, there's a lot of, again, very good people who care a lot—they're incredibly intelligent—who have been working both at the company and of course—we can't sell this one short, the larger Chef community. I really think that there's a tremendous amount of value there. Now, the company that acquired them—which I don't believe we've talked about yet—is Progress, and their tagline may as well be, “Who are you again?” They apparently have a bunch of enterprise CI/CD style software, but they're certainly not a commonly known name in the space, at least to me.

Pete: They're in Boston, apparently. I'm in Boston. I have never heard of them before. I actually had to go on LinkedIn to see if I had any first-level connections, and was shocked to find I had zero.

Corey: “Oh, they’re just in Boston for tax purposes.” “Well, doesn't that sound incredibly expensive?” “Well, they're in Boston for tax purposes that make no sense.” “What up?” “We didn't say they were good at it.”

Pete: [laughs]. Exactly. I mean, that's probably not exactly where you're going to be going for those reasons, for sure. But my hope is that, obviously, for the employees that are there that they get taken care of. I know a lot of friends who were there for a while, have left and moved on to newer places, and still have a lot of close friends there. 

And with an acquisition like this, sure, you're probably not making any reasonable money from your options, but there is the great secret of an acquisition where they don't want to lose you right away. If you're an engineer, if you're an operator there, you're actually an asset to that acquiring business, and in many cases, they could put in front of you a pretty lucrative retention bonus to stick around for one year, two years, something like that. And then a lot of ways, that's where you're going to make the money on an acquisition, for good or bad, is that the new company really wants you to stay and they'll make it in your best financial interest to do so. But at the end of the day, I think it's sad how the ending happened. It was inevitable given how fast the market moved, just like it was inevitable that Docker was not going to go anywhere because this market moves so fast. 

But the one thing that really rings true—and I don't think another company has done it as well, yet—is built such a strong community around technical operators. The Chef community was very tight-knit, it was very big. I look at it as a huge part of my career in leveling up my career by being able to meet just such an amazing group of people, such a wide, diverse group of operators, and engineers, and people who cared about this. And these are lifelong friends for me. It's really amazing. It's really the best part of Chef from my perspective.

Corey: I would agree with it. It seemed at some level it was trying to pivot and find its way. InSpec was great: the idea of compliance as code, which is a slightly different thing from configuration management as code, though they obviously have clear antecedents in common. The value, though, is that suddenly you can prove to auditors things about your environment. I loved the idea, but my entire impression of the space is and remains that it's effectively a sysadmin/SRE dead end with a 40 year long tail of decline ahead of it, as a space. That may be unfair, but it does definitely match to my understanding of the world.

Pete: Yeah, and it's not like Chef is going away anytime soon. It's an incredibly useful tool. There's a lot of companies that honestly are not using it that should be using it. There's a lot of enterprises out there—remember, if you go to a devopsdays event, those people that are there are such a small fraction of just overall business. There's tons of people that still haven't done anything with configuration management, let alone Docker. Docker is so far away from a maturity level where they're at. 

So, Chef is definitely not going away, and it's not like people are going to go suddenly rip it out, and it's open-source still. Depending on what they do with it. There's a whole licensing change which, obviously, we only have so much time to get into all of the interesting changes over time, but it's an open-source project; there's still a community around it. There's still people who really like the project, and I kind of bask my retirement on the hope that 20 years from now, there's some company who has some antiquated Chef install—like today you have some antiquated COBOL mainframe—where I can just live out my retirement working 10 or 20 hours a week fixing someone's legacy Chef install. That's really my hope for the future.

Corey: I think that's probably the best place to leave it. I hope for the sake of people I know, trust, and like, that things go well. I would love nothing better than to be completely wrong in everything that I just said because I want to be.

Pete, thanks once again for joining me. This is the AWS Morning Brief. If you've enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review and give it a five-star rating on your podcast platform of choice, and tell me in the comments why I'm a complete idiot about Ruby.

Announcer: This has been a HumblePod production. Stay humble.