Summer Replay – Breaking Down Productivity Engineering with Micheal Benedict

• (0:00) Intro
• (0:55) Panoptica sponsor read
• (1:36) What is product engineering?
• (2:44) The effectiveness of internal platforms
• (7:46) Solving AMI problems
• (10:23) Building foundations and learning woes
• (13:06) Micheal’s day-to-day at Pinterest
• (15:31) When engineering productivity starts to make sense
• (18:58) Changes Micheal would've made at Pinterest
• (20:56) Panoptica sponsor read
• (21:19) Biggest mistakes at Pinterest
• (23:46) Navigating outages in the cloud
• (30:51) Corey’s personal experiences with Pinterest
• (36:20) The legacy of code
• (40:31) Where you can find more from Micheal

• Pinterest: https://www.pinterest.com
• Twitter: https://twitter.com/micheal
• LinkedIn: https://www.linkedin.com/in/michealb/

• Panoptica: https://www.panoptica.app/

Transcript

Michael Benedict: There's a probability that GitHub can go down and that means people will be not productive for that couple of hours. What do we do then? And we had to kind of put a plan together to kind of how we can mitigate that part. And really build that confidence with the engineering teams.

Corey Quinn: Welcome to Screaming in the Cloud. I'm Corey Quinn. Sometimes when I have conversations with guests here, we run long, really long. And then we wind up deciding it was such a good conversation. And there's still so much more to say that we schedule a follow up and that's what happened today. Please welcome back Michael Benedict, who is, as of the last time we spoke and presumably still now, the head of engineering productivity at Pinterest.

Michael, how are you?

Michael Benedict: I'm doing great. And thanks for that introduction, Corey. Thankfully, yes, I am still the head of engineering productivity. I'm really glad to kind of speak more about it today.

Corey Quinn: This episode has been sponsored by our friends at Panoptica, part of Cisco. This is one of those real rarities where it's a security product that you can get started with for free, but also scale to enterprise grade.

Take a look. In fact, if you sign up for an enterprise account, they'll even throw you one of the limited, heavily discounted AWS skill builder licenses they got. Because believe it or not, unlike so many companies out there, they do understand AWS. To learn more, please visit panoptica. app slash last week in AWS.

That's panoptica. app slash last week in AWS.

The last time that we spoke, we went up one side and down the other of large scale environments running on AWS and billing aspects thereof, et cetera, et cetera. I want to stay away from that this time and instead focus on the rest of engineering productivity, which is.

Always an interesting and possibly loaded term. So, what is productivity engineering? It sounds almost like it's an internal dev tools team, or is it something more?

Michael Benedict: Thanks for asking because I get this question asked a lot of times. So for one, our primary job is to enable every developer, at least at our company, to do their best work.

And we want to do this by providing them a fast, safe, and a reliable path to take any idea into production without ever worrying about the infrastructure. As you clearly know, learning anything about how AWS works or any public cloud provider works is a ton of investment. And we do want our product engineers, our mobile engineers and all of the other folks to be focused on delivering amazing experiences to our pinners.

So we could be doing some of the hard work in providing those abstractions for them in such a way in taking away the pain of managing infrastructure.

Corey Quinn: The challenge, of course, that I've seen is that a lot of companies take the approach of, Ah, we're going to make AWS available to all of our engineers in its raw, unfiltered form, and that lasts until the first bill shows up.

And then it's okay, we're going to start building some guardrails around that, which makes a lot of sense. There then tends to be a move towards internal platforms that effectively wrap cloud services. And for a while now, I've been generally down on the concept and publicly so. In the general sense. That said, what I say that applies as a best practice or something that most people should consider does tend to fall apart when we talk about specific use cases.

You folks are in an extremely large environment. How do you view it? First off, do you do internal platforms like that? And secondly, would you recommend that other companies do the same thing?

Michael Benedict: I think that's such a great question because every company evolves with its own pace of development. And I wouldn't say Pinterest by itself had a developer productivity or an engineering productivity organization from the get go.

I think this happens when you start realizing that your core engineers who are working on product are now spending a certain fraction of time, which starts ballooning pretty fast, in managing the underlying systems and the infrastructure. And at that point in time, it's probably a good question to ask, how can I reduce the friction in those people's lives such that they could be focused more on the product?

and kind of centralize or provide some sort of common abstractions through a central team, which can take away all that pain, right? So that is generally a good guiding principle to think about when your engineers are spending at least 30 percent of their time on operating the systems rather than building capabilities, that it's probably a good time to revisit and see whether a central team would make sense to take away some of that.

And just simple examples, right? This includes upgrading OS on your EC2 machines, or just trying to make sure you're patching sort of all the right versions on your next big Kubernetes cluster you're running for X, serving X number of users. The moment you start seeing that you want to start, start thinking about if there is a central team who could take away that pain, what are the things they could be investing on to kind of help up level every other engineer within your organization?

And I think that's one of the best ways to kind of be thinking about it. And. It was also a guiding principle for us, uh, within Pinterest to kind of view what investments we could make in these central teams, which can up level each and every different type of engineer in the company as well. And just an example on that could be, your mobile engineer would have very different expectations from your backend engineer who's working on certain aspects of code in your product.

And it is truly important to understand where you want to kind of centralize capabilities, which both these types of engineers could use, or you want to divest and have like unique capabilities where it's going to make them productive. There's no one size fits all solution for this, but I'm happy to talk about what we have at Pinterest.

Which has been, uh, reasonably working well, but I do think there's like a lot more improvements we could be doing.

Corey Quinn: Yeah, but let's also be clear that, as you've mentioned, you are heavily biased towards EC2 instances for a lot of what you do. If we look at the AWS console, and we see hundreds of different services now, and it's easy to sit here and say, oh, internal platforms are terrible because all of those services are going to be enhanced in various ways, and you're never going to be able to keep up with feature parity.

Yeah, but if you could wrap something like EC2 in an internal platform wrapper, that begins to be a different story. Because sure, if someone's going to go try something new with a different AWS service, they're going to need direct access. But the EC2 product across the board generally does not evolve in leaps and bounds with transformative changes overnight.

Let's also not forget that at a company with the scale that Pinterest operates at, hey, AWS just dusted off a new feature and docs are still rolling out. It's not in cloud formation yet, but we're going to roll it out to production. Probably seems like the wrong direction to go in, I would assume.

Michael Benedict: And yes, I think that brings sort of one of the key guardrails, I think, which these groups provide.

So when we start thinking about what teams, centralized teams like engineering productivity, developer tools, developer platforms actually do is they help with a couple of things. The top three are, they can help pave a path for the most common use cases. Like, to your point, provisioning EC2 does take a set of steps all the time.

If you're going to have a thousand people doing that every time they're building a new service or trying to expand capacity, playing with their, you know, launch templates, those are things you could start like streamlining and making it simple by some wrapper. Because you want to. Address those 80 percent use cases which are usually common.

And you can have a wrapper which can just automate that. And that's one of the key things, right? Like, can you provide a paved path for those use cases? The second thing is, can you do that by having the right guardrails in place? How often have you heard the story that I just clicked a button and that now spun up like a thousand plus instances?

And now you have to juggle between trying to, you know, stop the Mars and do something about it.

Corey Quinn: Back in 2013, you folks were still focusing on this a fair bit, I remember, because Jeremy Carroll, who I believe was your first SRE there once upon a time, wound up doing a whole series of talks around how, Pinterest approach doing an AMI factory.

And back in those days, the challenges were, okay, we have the baseline AMI, and that's great, but we also want to do deployments of things, and we don't really want to do a new deploy of an entire fleet of EC2 instances for a single line of config change. So how do we wind up weighing off of when you bake a new AMI, or versus when you just change something that is, and what is deployed to them?

And it was Really a complicated problem back then. I'm not convinced it's not still a complicated problem, but the answers are a lot more cohesive. And making sure that every team, when you're talking about a company as large as Pinterest with that many teams, is doing things in the same way, seems like it's critically important.

Otherwise, you wind up with a whole bunch of unique looking instances, each have to be managed by hand as opposed to something that can be reasoned around collectively.

Michael Benedict: And that last part you mentioned is extremely crucial as well, because like I said, our audience or our customers are just not the engineers.

We do work with our product managers and business partners as well, because at times, we have to tie or change our architecture based on certain cost optimizations, which would make sense. Like you just articulated, like we don't want to have all the instance types. It does not add much value to a developer unless they're explicitly seeking a high memory instance or a GPU based instance in a certain way.

So we can then work with our business partners to make sure that we're committing to only a certain type of instances and how we can abstract our tools to only give you that. For example, our deployment system, Teletran, which is an open source system, actually condenses down all these instance types to like a couple of categories, like high compute, high memory.

And you've probably seen that in many of the new cloud providers as well. So people don't have to learn or know the underlying instance type. When we move from C3 to C5, it was just called as a high compute system. So the next time someone provisioned a new service or deployed it using our system. They would just select High Compute as the de facto instance type, and we would just automatically provision a C5 for them.

So that just reduces the, you know, like, complexity of the cognitive overhead individuals would have to go through in learning. Each instance type, what is the base AMI that comes on it? What are the different configurations that need to go in in terms of setting up your AZ scaling properties? We give them like a good reasonable set of defaults to get started with and then they can then work on like kind of optimizing or making changes to it.

Corey Quinn: Ignoring entirely your mispronunciation of AMI, which is of course three syllables, and that is a petty hill upon which I will die, it occurs to me the more I work with AWS in various ways, the easier it gets. And I used to think in some respects it was because the platform was so, was improving so dramatically around me.

But no, in many cases, it's because the first time you write some CloudFormation by hand, it's a nightmare and you keep smacking into weird issues. But the second or third time, it's super easy because you just copy the thing you've already built and change the relevant bits around. And that was the learning curve that I went through playing around with a lot of these things.

When you start looking at this from a large scale environment where it's not just about upskilling the people that you have to understand how these things integrate in AWS land, but also the consistent onboarding of engineers at a fairly progressive clip is great. trainings on all these things, and there's a lot of knobs and dials that can blow up and hurt people.

At some point. Building the guardrails or building the environment in which you are getting all the stuff abstracted away from where the application engineers have to think about this at all, it eventually reaches a tipping point where it starts to feel like it's no longer optional if you want to continue growing as a company because you don't have the luxury of spending six months of onboarding before you let someone touch the thing they were hired to build.

Michael Benedict: And you will see that many companies very often have very similar programming practices like you just described. Even I learned it the same way. Like you have a base template, you just copy paste it and start from there on. And no one goes through the bootstrapping process manually anymore. You want to, I think we call it cargo culting, but in general, just get something to bootstrap and start from there.

One of the things we learned in, in sort of in a hard way is that can also lead to kind of you pushing, you know, not great practices because people don't know what is sort of a blessed version of a good template. Or what actually would make sense. So some of the things we have been like working on and this is where like centralized teams like engineering productivity are really helpful is we provide you with the blessed or the canonical way to do certain things.

Case in point example is a CICD pipeline or a delivery of software services. We have invested enough. in kind of experimenting on what works with some of the more nuanced use cases at Pinterest in helping generate sort of a canonical version, which would cover 80 percent of the use cases. Like someone can just go and try to build a service and they could just use the same canonical pipeline without learning much or making changes to it.

This also reduces sort of that cargo culting nature, which I called, rather than copying it from unknown sources and trying to like, again, it may cause havoc to our system, so we can avoid a lot of that because of these practices.

Corey Quinn: So, let's step a little bit beyond AWS. I know, I hate doing it too. But, I'm going to assume that your remit is broader than, oh, AWS Whisperer slash Wrangler.

So, tell me a little bit more about what it is that, uh, What your day to day looks like, if there is anything that could be said not to focus purely around AWS Whispering.

Michael Benedict: So, one of the challenges, and I want to talk about this a bit more, is our environments have become extremely complex in over time.

And it's the nature of, you know, like, rising entropy, like, we've just noticed that there's two things. We have a diverse set of customer base, and, uh, these include everyone trying to do different workloads or work, you know, service types. Microsoft Mechanics www. microsoft. com What that essentially translates into is that we've realized that our solution may not fit all of them.

For example, what works for a machine learning engineer in terms of iterating on sort of building a model and delivering a model is not the same as someone working on a long running service and, and trying to sort of deploy that. The same would apply for someone trying to operate a Kafka system. And that has made, I think, definitely our job a bit challenging in trying to assess where do you actually draw the line on the abstraction?

What is the right layer of abstraction across your local development experience, across when you move over to kind of like staging sort of your code in a PR model and getting feedback? And subsequently actually releasing it to production. Because this changes dramatically based on what is the workload type you're working on.

And we feel like that has been one of the biggest challenges where I know I spend my day to day and my team does too in trying to like help provide some of the right solutions for these individuals. There is very often we'll also get asks from individuals trying to do a very nuanced thing. Off late we have been talking about like thinking about how do you operate functions, like provide functions as a service within the company.

It does put us in a difficult spot at times because we have to ask the hard question, is this required? I know the industry is doing it. It's, it's definitely there. I, I personally believe, yes, it could be a future, but is that absolutely important? Is that going to benefit Pinterest in any formal way if we invest on some core abstractions?

And those are difficult conversations to have because we have exciting engineers coming in trying to do amazing things. It puts us in a hard spot as well, as to sometimes saying graciously no. I know many companies deal with it when they have these centralized teams, but I think it's part of that job.

Like when you say it's day to day, I would say I'm probably saying no a couple of times in that day.

Corey Quinn: Let's pretend for the sake of argument that I am tomorrow morning starting another company, Twitter for Pets, and over the next 10 years it grows to be larger than Pinterest in terms of infrastructure, probably not revenue because it turns out pets are not the lucrative source of ad revenue that I was hoping it would be, but you know, directionally the same thing.

It seems to me that building out this sort of function with this sort of approach to things is dramatically, uh, Early, as far as optimizations go, when it's just me puttering around on something. I'm always cognizant of the, the wrong people taking the wrong message when we're talking about things that happen like this at scale.

When does having an engineering productivity group begin to make sense?

Michael Benedict: I mentioned this earlier, like, yeah, there's definitely not a right answer, but we can start small. For example, this group actually started more as a delivery team. You know, when we started, like, we realized that we had like, different ways of deploying services or software at Pinterest.

So we first like gathered together to figure out, okay, what are the different ways and can we start like simplifying that part? And that's where it started expanding. Okay. We are doing button based deployments right now. We have thousand plus microservices. And we are seeing more incidents than we wanted to because anything where there's a human involved means there's a potential gap for error, right?

I myself was involved in a Ceph Zero incident, and I will be honest, like, we ended up deploying a Hello World application in one of our production fleet. Not the thing I wanted to be associated with my name, but,

Corey Quinn: you were suddenly saying hello to the world, in fact, and oops a doosie.

Michael Benedict: Yeah, so, And that really prompted us to rethink how we need to kind of enable guardrails to kind of do safe production rollouts.

And that's how, you know, those conversations start ballooning out. Yeah.

Corey Quinn: And in a healthy, correct way. We've all broken production in various ways, and it's, you correctly are identifying, I believe, the direction you're heading in, where this is a, process problem and a tooling problem, it is not that you are secretly crap and should never have been allowed near anything in production.

I mean, that's my excuse for me, but in your case, this is a common thing where it's, if someone can unintentionally cause issues like that, there need to be better processes and procedures as the organization matures.

Michael Benedict: And that's kind of like always the root or the starting point for these discussions and it starts like growing from there because Okay, you've kind of helped improve the deploy process, but now we're seeing insane amount of slowness, say, on the build processes, or even post deploy, there's like issues on how we monitor and look into data.

And that, I think, forces these conversations, okay, where do we have these bespoke tools available? What are people doing today? And you have to ask those hard questions like what can we actually remove from here? The goal is not to kind of introduce yet another new system. Many a times, to be honest, Bash just gets the job done.

Personally, I'm okay with that as long as it's consistent and people, you know, are able to contribute to it and you have good practices and kind of validating it. If it works we should go for it rather than introducing Get another YAML and, and some of that other aspects of doing that work. And that's what we encourage as well.

That's how I think a lot of this starts kind of like connecting together in terms of, okay, now this is sort of becoming a productivity group. Like they're focused on certain challenges where investing probably one person here may up level a few other engineers who don't have to do that on a day to day basis.

And I think that's one of the key items for especially folks who are running mid size companies to kind of realize and start investing in these type of teams to kind of like really up level sort of the rest of the engineering.

Corey Quinn: You've been doing this for a fair while. If you were to go back and start over again on day one, which is always a terrifying question on some level, what would you have done differently about building out this function as Pinterest continued to scale out?

Michael Benedict: Well, first I must acknowledge that this was just not me and there's like a ton of people involved in helping make this happen. No, it's fair. We'll blame them for the missteps. That is just fine with me. I kid. I kid. I think definitely the nuances, if I look back, all the decisions that were made, and at that point in time, there was a decision made to sort of move to, you know, Fabricator, which was back then a great open source code management system.

were with the current information at that point in time. And I'm not, I think it's very hard to always look back and say, Oh, we could have, you know, chosen X at one point in time. And I think reality, that's how engineering organizations always evolve, that you have to make do with the information you have right now to make a decision that works for you over a couple of years.

And I'll give you a small example of this. There was a time when Pinterest was actually on GitHub Enterprise. This was like circa 2013, I would say. And it really served us well for like five plus years. Only then, at a certain point, we realized that it's hard to kind of hire PHP engineers to support a tool like that.

And we had to rethink what is sort of the ROI and the investments we would make here. Can we ever map up or match back to sort of what are the offerings in the industry today? And that's when you sort of make decisions that, okay, at this point in time, it's clear that business continuity tops, you know, it's hard to kind of operate a system, which is at this moment not supported.

And then you make a call about, you know, making a shift or moving. And I think that's the key item. Like, I don't think there's anything dramatically that would have changed since the start. Perhaps definitely like investing a bit more individuals into the group instead of like going from there. But that said, I'm really sort of at least proud of the fact that usually these teams are extremely lean and small and they always have like an outsized impact, especially when they're working with like other engineers, other opinionated engineers, for what it's worth.

Corey Quinn: Few things are better for your career and your company than achieving more expertise in the cloud. Security improves, compensation goes up, employee retention skyrockets. Panoptica, a cloud security platform from Cisco, has created an academy of free courses just for you. Head on over to academy. panoptica.

app to get started.

Most folks show up and tend to do good today and you make the best decision at the time with the context and constraints that you have. My question I think is less around, well, what did you, what were the biggest mistakes you made, but more to do with the idea of based upon what you've learned and as you have shown.

As you've shined light on these dark areas, as you have been exploring it, has anything jumped out at you that is, oh yeah, now that I know, if I know then what I know now, I would definitely have made this other decision. Ideally something that applies a little more globally than specific within Pinterest, just because the whole idea aspirationally is that people might learn something from our conversation.

At least I will, if nothing else.

Michael Benedict: No, I think that's a great question. And I think there's three things that jump to me, top of mind. I think technology is means to an end, unless it gives you a competitive edge. And it's really hard to figure out at what point in time, what technology and why we adopted it's going to make the biggest difference.

Humans always tend to have a bias towards like aligning towards like where we want to go. So that's the first one in my mind. The second one is, and we spoke about this last time, embrace your cloud provider as much as possible. You would want to avoid taking on operational burden, which is not going to add value to the business.

If there's something you see you're operating, which can be offloaded because your provider can trust me, do a way better job than you or your team of few can ever do. Embrace that as soon as possible. It's better that way because then it frees up your time to focus on the most important thing which I realized over time is, I really think teams like ours are actually, we probably have the most value as a glue to all the different experiences a software engineer would go through as part of their SDLC lifecycle.

If we can simplify someone's life by giving them a clear view as to where their commit or their work is in this grand scheme of, you know, rolling out and giving them the right amount of data to kind of take action when something goes wrong. Trust me, they will love you for what you're doing because you're saving them a ton of time.

Many times we don't realize that when we publish 11 different UIs for you to go and check to just get your basic validation of work done. You know, we tend to so much focus on the technological aspect of what that tool does rather than the experience of it. And, and I've realized like, if you can like bridge The experience, especially for teams like ours, people really don't even need to know whether you're running Kubernetes or any of those solutions behind the scenes.

And I think that's one of the biggest takeaways that I have.

Corey Quinn: I want to double down on something you said about the fact that you are not going to be able to run these services as effectively as your provider can. And relatively recently, in fact, since the first time we spoke, AWS has released a new investment report in Virginia.

And from 2011 through 2020, they have invested in building AWS data centers there, 35 billion. I promise almost no company that employs people listening to this that is not, they're not themselves a cloud provider, is going to make that kind of investment in running these things themselves. Now, do cloud providers have sharp edges?

Yes, absolutely. Uh, that is what My entire career is about, unfortunately. You're not going to do a better job of running things more sustainably, more reliably, etc, etc. But there are other problems with this and that's what I want to start exploring here. Where in the olden days when I ran things in data centers and they went down a lot more as a result, sometimes when there were outages I would have the CEO of the company just standing there nervous, worrying over my shoulder as I frantically typed to fix things.

Spoiler, my typing accuracy did not improve. by having someone looming over me. Now, when there's an outage that your cloud provider takes, in many cases, the thing that you are doing to fix it is reloading the status page and waiting for an update because it is completely out of your hands. Is that something that you've had to encounter?

Because you can push buttons and turn dials when things are broken and you control it, but in an AWS or other cloud provider outage, all you can really do is wait unless you have a DR plan that is large scale and effective enough that you won't feel foolish or have wasted a huge amount of time and energy migrating off, because then it gets repaired in 10 minutes.

How do you approach that from your perspective? I guess the expectation management piece.

Michael Benedict: It's definitely, I know, something which keeps a lot of folks with an infrastructure up at night, because like you just said, at times we can feel extremely powerless when, you know, we obviously don't have direct control or visibility at times as well on what's happening.

One of the things we have realized over time as part of like running on our cloud provider for over like a decade now. It forces us to rethink a bit on our priority workflows, what we want our pinners to always have access to, what they need to see, what it's not, you know, like important or critical, because it puts into perspective, even for sort of the infrastructure teams, is to what is the most important thing we should always have it available and running, what is okay to be in a degraded state.

And to what time, right? So it actually forces us to define SLOs and availability criteria within the team, where we can broadcast that to the larger audience, including the executives. So none of this comes as a surprise at that point. I mean, it's not the answer probably you're looking for, because there's nothing we can do except set expectations clearly on what we can do and how we need to think about sort of the business when these things do happen.

So I know people may have a different view on this. I'm definitely curious to hear as well. But I know at Pinterest, at least, we have sort of like converged on our priority workflows. When something goes out, how do we kind of jump in to kind of provide a degraded experience? We have very clear runbooks to do that.

And especially when it's a set zero, we do have clear processes in place and how often we need to update our entire company and where things are. And especially, you know, this is where your partnership with the cloud provider is going to be a big, big boon because you really want to know or have visibility at the minimum Some predictability on, you know, when things can get resolved and how you want to work with them on some creative solutions.

This is outside the DR strategy, obviously, right? You should still be focused on a DR strategy, but these are just simple things we have learned over time. And how to just make it predictable for individuals within the company so not everyone is freaking out.

Corey Quinn: Yeah, from my perspective, I think the big things that I've found that have worked in my experience, mostly by getting them wrong the first time, is explain that someone else running the infrastructure, when they take advantage, there's not much we can do.

And no, it's not the sort of thing where picking up the phone and screaming at someone is going to help us. is the sort of thing that is best to communicate to executive stakeholders when things are running well, not in the middle of an incident. Then when things break, it's one of those, great, you're an exec, you know what your job is, literally anything other than standing in the middle of the engineering floor making everyone freak out even more.

We'll have a discussion later about What the contributing factors were when you demand that we fire someone because of an outage, then we're going to have a long and hard talk about what kind of culture you're trying to build here again. But there are no perfect answers here. It's easy to sit here in the sober light of day with things working correctly and say, Oh yeah, this is how outages should be handled.

But then when it goes down, we're all. Basically an inch away at best from running around with her hair on fire screaming fix it, fix it, fix it, fix it now. And I am empathetic to that. There's a reason that I fix AWS bills for a living and one of those big reasons is that it's a strictly business hours problem and I don't have to run production infrastructure that faces anything that people care about which is kind of amazing and freeing for someone who spent too many years on call.

Michael Benedict: Absolutely, and one of the things is this is not only with the cloud provider, I think in today's nature of how our businesses are set up, there's probably tons of other APIs you are using or you're working with, you may not be aware of, and like, we ended up finding that the hard way as well, like there were like a certain set of APIs or services we were using in the critical path, Which we were not aware of, like when these outages happen, that's when you find that out.

So, you're not only beholden to your, you know, provider at that point in time, you have to have those expectations set with your other SaaS providers as well. Other folks you're working with, because I don't think that's going to change. It's probably only going to get complicated with, you know, all the different types of tools you're using.

And then that's sort of like a trade off you need to kind of like, really think about. An example here is just like, you know, like I said, we moved in the past from, you know, GitHub to Fabricator. I didn't close the loop on that because we're moving back to GitHub right now. And that's one of the key projects I'm working with.

Yeah, it's a circle of life, right? But the thing is, we did a very strong evaluation here, like, because we felt like, okay, there's a probability that GitHub can go down, and that means people will be not productive for that couple of hours. What do we do then? And we had to kind of put a plan together to kind of how we can mitigate that part.

And really build that confidence with the engineering teams internally. And it's not the best solution out there, the other solution was just run our own. But how is that going to make any other difference because we do have libraries being pulled out of GitHub and so many other aspects of our systems which are unknowingly dependent on it anyways.

So you have to still mitigate those issues at some point in your entire SDLC process. That was just one example I shared, but you know, it's not always on the cloud provider. I think there's just many aspects of, at least today, how businesses are run. You're dependent, you have critical dependencies, probably on some SaaS provider you haven't really vetted or evaluated.

You'll find out when they go down.

Corey Quinn: So, I don't think I've told this story before, but before I started this place, I was doing a fair bit of consulting work for other companies, and I was doing a project at Pinterest years ago, and this was one of the best things I've ever experienced at a company site, let alone a client site, where I was there early in the morning, 8 o'clock or so, so you know, engineers love to show up at the crack of 1130, but so I was working a little early, and it was great, and suddenly my SSH session that I was using to remote into something or other hung, and it's tap up, tap enter a couple of times, tap it a couple more, It was hung hard.

What's the And then someone gently taps me on the shoulder, so I take the headphones off. It was someone from corporate IT was coming around saying, Hey, there's a slight problem with our corporate firewall that we're fixing. Here's a MiFi device just for you that you can tether to to get back online and get work done until the firewall gets back.

And it was incredible. Just the level of just being on top of things and the focus on keeping. The people who were building things and doing expensive engineering work that was awesome and also me, productive during that time frame was just something I hadn't really seen before. It really made me think about the value of where do you remove bottlenecks from people getting their jobs done?

It was, it remains one of the most impressive things I've seen.

Michael Benedict: That is great. And as you were telling me that, I did look up our internal system to see whether a user called Corey Quinn existed, and I should confirm this with you. I do see entries over here, a couple of commits, but this was 2015. Was that the time, like, you were around, or is this before that even?

Corey Quinn: That would have been around then, yes. I didn't start this place until late 2016.

Michael Benedict: I do see your commits, like, from 2015.

Corey Quinn: And they're probably terrible, I have no doubt. There's a reason I don't write code for a living anymore.

Michael Benedict: Okay, I do see a lot of GIFs, and I hope it's pronounced as GIF er. Okay, this is cool.

We should definitely have a chat about this separately, Corey.

Corey Quinn: Oh yeah, we can't explain this code? Absolutely not. I wrote it. Of course I have no idea what it does. That's the rule. That's the way code always works.

Michael Benedict: Well, you are an honorary Pinterest engineer at this point, and you have, yes, contributed to our API service and a couple of Puppet profiles I see over here.

Oh, yes,

Corey Quinn: you don't wind up thinking that that's a risk factor that should be disclosed. I kid, I kid. It's, I made a joke about this when VMware acquired SaltStack and I did some analytics and found there were 60 some odd lines of code I had written way back when that were still in the current version of what was being shipped.

And they thought, wait, is this actually a No, I am making a joke. My, the joke is, is my code is bad. Fortunately, there are smart people around me who review these things. This is why code review is so important. But there was a lot to admire when I was there doing various things at Pinterest. It was a fun environment to work in.

The level of professionalism was phenomenal. And I was just a big fan of a lot of the automation stuff. Fabricator was great. I loved working with it. Great, I'm going to use this to the next place I go. And I did and then it was, I looked at what it took to get it up and running and oh yeah, I can see why GitHub is so popular these days.

But it was neat. It was interesting seeing that type of environment up close.

Michael Benedict: That is great to hear. You know, this is what I enjoy, like hearing some of these war stories. Like I, I am, I'm surprised, like you seem to have committed way more than I've ever done in my duration here at Pinterest. I, I do do managing for a living, but then again, you know, Corey, the good news is your code is still running on production.

And, uh, we haven't, we haven't removed or made any changes to it. So that's pretty amazing. And thank you for all your contributions.

Corey Quinn: Oh, please, do you have to thank me? I was paid. It was fine. That's the value of work for hire. It's kind of amazing. The best part about consultants is when we're done with a project, we get the hell out and everyone's happy about it.

More happy when it's me that's leaving because of obvious personality related reasons. But it's It was just an interesting company from start to finish. I remember one other time I wound up opening a ticket about having a slight challenge with a flickering on my then Apple branded display that everyone was using before they discontinued those.

And I expected there to be, oh, okay, you're a consultant. Great. How did we not put you in the closet with a printer next to that thing breathing the toner? Most consulting clients tend to do. And sure enough, three minutes later, I'm getting that tap on the shoulder again. They have a whole replacement monitor.

Can you go grab a cup of coffee? We'll run the cable for you. It'll just be about five minutes. I started to feel actively bad about requesting things because I did a lot of consulting work for a lot of different companies and Not to be unkind, but treating consultants and contractors super well is not something that a lot of companies optimize for.

I can't necessarily blame them for that. It just really stood out.

Michael Benedict: Yeah, I do hope we are keeping up with that right now because I know our team definitely has a lot of consultants working with us as well. And it's always amazing to see like, you know, we do want to treat them as FTEs. Like it doesn't even matter at that point because we're all individuals and we're trying to work towards common goals.

Like you just said, like, I think I personally have learned like a few items as well from some of these folks, which is again, like I think speaks to sort of how we want to work and sort of create a culture of like, we're all engineers, we want to be solving problems together. And as you were doing it, we want to do it in such a way that it's still fun.

And you know, we're not having any restrictions of titles or roles and other pieces. But I think I digressed. It was really fun to see your commits though. I do want to track this at some point before we move completely over to GitHub. At least, you know, keep this as a record.

Corey Quinn: Yeah, basically look at this graffiti in the code base of a shit poster was here and here I am.

And that, that tends to be in some level the mark we leave in the universe. What's always terrifying is looking at things I did 15 years ago in my first Linux admin job. Can I still ping the thing that I built there? Yes, I can. How is that even possible? That should not have out, that should not have outlived me.

Honestly, it should never have seen the light of day in production. But here we are and. You never know how long that temporary clue you put together is going to last.

Michael Benedict: You know, like one of the things I was recalling, I was talking to someone in my team about this topic as well. You know, we always talk about 10X engineers.

I don't know what your thoughts are on that, but the fact that you just mentioned, you build something, it's still pings and there's like a bunch of things in my mind, like, you know, when you are writing code or you're working on some projects, the fact that it can outlast. You and sort of live on I think that's a big big contribution and secondly If your code can actually help up level like 10 other people I think you've really met the mark of 10x engineer at that point

Corey Quinn: Yeah the idea of the superhuman engineer is always been a strange and dangerous one if for nothing else from where I sit excellence is inherently situational like what we just talked about someone at pinterest is Potentially going to be able to have that kind of impact specifically because to my world view that there's enough You process and things around there that empower them to succeed.

Then if you were to take that engineer and drop them into a five person startup where none of those things exist, they might very well flounder. It's why I'm always a little suspicious of, this is a startup founded by engineers from Google or Facebook or wherever it is. It's Yeah, and what aspects of that culture do you think are one to one matches with the small scrappy startup in the garage?

Right. I'm predicting some challenges here. Excellence is always situational. An amazing employee at one company can get fired at a second one for lack of performance, and that does not mean that there's anything wrong with them, and it does not mean that they are a fraud. It means that what they needed to be successful was present in one of those shops, but not the other.

Michael Benedict: This is so true, and I really appreciate you bringing this up because, you know, whenever we discuss any form of, you know, performance management, that is a, in my view, personally, I think that's a incorrect term to be using, it is really at that point in time, either you have outlived sort of the environment you are Or the environment is going in a different direction where I think your current skill sets probably could be best used in the environment, you know, where it's going to work.

And I know it's, it's very fuzzy at that point, but like you said, yes, excellence really means you don't want to tie it to the number of commits you have pushed out or any specific aspect of sort of your deliverables or like how you work.

Corey Quinn: There are no easy answers to any of these things and it's always situational.

It's why I think people are sometimes surprised when I will make comments about the general case and how things should be, then I talk to a specific environment where they do the exact opposite and I don't yell at them for it. It's, it's really hard. In a general sense, I have some guidance, but there are usually reasons things are the way they are and I'm interested in hearing them out.

Everything's situational. The worst consultant in the world is the one that shows up, has no idea what's going on, and then asks, what moron set this up? Invariably, too said quote unquote moron and the engagement doesn't go super well from there. It's Okay, why is this the way that it is? What constraints shaped it?

What was the context behind the problem you were trying to solve? And well, why didn't you use this AWS service? Because it didn't exist for another three years when we were building that thing is a very common answer.

Michael Benedict: Yes, you should definitely appreciate sort of like all the decisions that were made in past.

Like people tend to always forget why they were made. You're absolutely right. Like what worked back then will probably not work now or vice versa. And it's always situational. So I think I can go on about this for hours, but I think you hit that to the point, Corey.

Corey Quinn: Yeah, I do my best. I want to thank you for taking another block of time out of your day to wind up talking with me about various aspects of what it takes to effectively achieve better levels of engineering productivity at large companies, with many teams working on shared code bases.

If people want to learn more about what you're up to, where can they find you?

Michael Benedict: I'm definitely on Twitter, so please note that I'm spelled M I C H E A L on Twitter, so you can definitely read on to my tweets there, but otherwise, you can always reach out to me on LinkedIn too.

Corey Quinn: Fantastic. And we will of course include a link to that in the show notes.

Thanks once again for your time. I appreciate it. Thanks a lot, Corey. Michael Benedict, Head of Engineering Productivity at Pinterest. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five star review on your podcast platform of choice.

Whereas if you've hated this podcast, please leave a five star review on your podcast platform of choice, along with a comment telling me that you work at Pinterest, have looked at the codebase, and would very much like a refund and an apology.