Optimizing for Happiness with Alfonso Cabrera

About Alfonso Cabrera
Alfonso Cabrera is the Director of Platform Engineering at Red Ventures, where he helps manage and optimize the extensive AWS footprint. He also spent time at AWS as a Solutions Architect and worked as a DevOps Engineer at a few startups. Alfonso enjoys fostering community and has organized DevOpsDays Charlotte for the past 5 years. Outside of work, he tries to stay in shape by playing sports of all kinds, and gets his adrenaline fix by riding motorcycles.


Links Referenced

Transcript
Announcer: Hello, and welcome to Screaming in the Cloud with your host, Cloud Economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: Gravitational is now Teleport because when way more people have heard of your product than your company, maybe that’s a sign it’s a time to change your branding. Teleport enables engineers to quickly access any computing resource, anywhere on the planet. You know, like VPNs were supposed to do before we all started working from home, and the VPNs melted like glaciers. Teleport provides a unified access plane for developers and security professionals seeking to simplify secure access to servers, applications, and data across all of your environments without the bottleneck and management overhead of traditional VPNs. This feels to me like it’s a lot like the early days of HashiCorp’s Terraform. My gut tells me this is the sort of thing that’s going to transform how people access their cloud services and environments. To learn more, visit goteleport.com.

Corey: This episode is sponsored by a personal favorite: Retool. Retool allows you to build fully functional tools for your business in hours, not days or weeks. No front end frameworks to figure out or access controls to manage, just ship the tools that will move your business forward fast. Okay, let's talk about what this really is. It's Visual Basic for interfaces. Say I needed a tool to, I don't know, assemble a whole bunch of links into a weekly sarcastic newsletter that I send to everyone. I can drag various components onto a canvas: buttons, checkboxes, tables, etc. Then I can wire all of those things up to queries with all kinds of different parameters: post, get, put, delete, et cetera. It all connects to virtually every database natively, or you can do what I did, and build a whole crap ton of Lambda functions, shove them behind some API’s gateway and use that instead. It speaks MySQL, Postgres, Dynamo—not Route 53 in a notable oversight, but nothing's perfect. Any given component then lets me tell it which query to run when I invoke it. Then it lets me wire up all of those disparate APIs into sensible interfaces. And I don't know front end. That's the most important part here: Retool is transformational for those of us who aren't front end types. It unlocks a capability I didn't have until I found this product. I honestly haven't been this enthusiastic about a tool for a long time. Sure they're sponsoring this, but I'm also a customer and a super happy one at that. Learn more and try it for free at retool.com/lastweekinaws. That's retool.com/lastweekinaws, and tell them Corey sent you because they are about to be hearing way more from me.

Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Alfonso Cabrera, who is currently the director of platform engineering at Red Ventures. Alfonso, welcome to the show.

Alfonso: Thanks, Corey. I'm excited to be on. I've listened to the show for quite a while now.

Corey: I would suggest it might be time to find a show with, you know, intelligent folks, but then again, that's why I have conversations with other people. Basically, we have this idea in storytelling where you have the hero's journey. I do something similar called the moron’s journey, and I'm the moron. So, it works out super well: every time I talk to someone on the show, I learn something. So, you run platform engineering at Red Ventures. Everyone has ventures; what is a Red Venture? And how did it come to become color-coded?

Alfonso: [laugh]. Great question. So, Red Ventures is a portfolio of influential brands and digital platforms and some partnerships as well. But essentially, we connect millions of people, through our audience with expert advice and information. 

We’re in a lot of different industries, so we have some businesses in home industry, in health, in travel, in finance. Some of the brands you may have heard of, things like Healthline, or The Points Guy if you travel a lot, but also things like Bankrate and MYMOVE and creditcards.com.

Corey: Yeah, I get nostalgic and sad about that now because again, I used to live on airline miles, more or less, and now it turns out that no one travels anymore. And I am… I guess I never thought I would actually miss traveling, but here we are.

Alfonso: I know I'm in the same boat. I miss it and ready to get back out there and go travel again.

Corey: [laugh]. So, what makes this fun is that I've met you a few years back when you were not a director at that point; you were in a different role at Red Ventures. I forget offhand what that was.

Alfonso: Yeah, I started pretty early on the cloud ops engineering team over at Ventures about four and a half years ago now. I was one of the first two folks that was helping the company move to the Cloud.

Corey: And then you quit. Because the best thing to do is go work somewhere else, given the opportunity. At least that's always been my philosophy, which is why my resume is a patchwork of tattered broken dreams. But you went to a small company called AWS and acted as a solutions architect for a while. And then you boomeranged back to Red Ventures. So, what was that like?

Alfonso: Yeah, for sure. It was a great opportunity to go to AWS and try out a different career path. I view solutions architecture there as customer-facing, almost similar to a sales engineer type of role. And I had a lot of fun. I learned a ton of stuff at AWS. 

Some of the things I really really enjoyed and took away from there that probably will stay with me forever is just the writing culture is crazy to see there. I was really fond of that. I think writing brings a lot of clarity of thought, instead of compared to a PowerPoint deck, for example, where you may have a few words in the slide, but generally, you're kind of just talking through things and when you write things down, whether it's a one-pager, a six-pager or the PR FAQs that they're well known for, it just brings a lot more clarity and gets everybody on the same page. Something that’s definitely stuck with me and I've continued to do as a habit over at Ventures.

Corey: I love your framing of that. Usually when you ask people, “Oh, so at this job, what did you take away from it?” The honest answer is generally, “A bunch of office supplies,” and that's about it. But every time you talk to someone who's left AWS, they talk about, on some level, the lessons that they learned by working there. How much of I guess AWS culture do you find is applicable or portable to other companies?

Alfonso: Yeah, actually, it's funny because one of the other things I really enjoyed there is the principles-based decision-making they do. Obviously, they're very vocal about their leadership principles and reference them in day-to-day decisions, or in meetings and in different projects they’re working on, and that's actually something that I've also seen at Red Ventures. One of the things I love about the culture at Red Ventures is it's very similar. We have belief statements that we discuss often. 

Those things are brought up when we're making decisions, or when we're starting new projects, we frame them against our belief statements. And it makes a big difference because it gives some consistency or table stakes to things, to decision-making, the projects you work on, and into discussions you have, and into just the company culture overall. I think it makes a big difference.

Corey: The challenge I always see is that you have these folks who go and look at, “Well, what did Amazon do that made them successful?” And their response is, “Oh, okay.” So, they read the leadership principles, pick their three favorites, and then try to, effectively, slap them into their own company culture without anything approaching context. It’s, “Okay. Frugality. Great. We're going to become super cheap. Two pizza teams, I bought a whole bunch of pizzas for everyone. Why is our code still broken?” 

And they don't realize that there's some secret sauce that only really works at Amazon, such as giving things ridiculous names, but that's a bit of a low blow. It's strange seeing how culture is shaped by a company and leadership principles—or their equivalent at other companies—seem like outgrowths of what those companies have become. But in your case, I think that, talking about what your takeaway is, I guess, more nuanced than that. It's not just about, well, we did X thing and Y result happened. Instead, you're talking about things like the writing culture. For folks who haven't been obsessively stalking a $2 trillion company for the past four years—ehem—explain a little bit more about what the writing culture is like, please.

Alfonso: Yeah, so for example, as a solutions architect, you typically are partnered with an account executive, depending on, kind of, what group you're in. I was part of a greenfield team, and there's a lot of writing. There are quarterly, essentially, business reviews; they call them something different internally. But you typically partner with your account exec, and write a six-pager or account territory plan and spend a lot of time and effort getting that right and being really thoughtful about it, and that gets reviewed with the entire area team—including leadership—every quarter. And that is the foundation for how you're going to go talk to your customers, how you're going to get different projects, how you're going to help them succeed. 

Obviously, it's a very customer-obsessed company, so it's definitely skewed towards, like, how do we help our customers succeed? What can we offer? What can we help with? What programs do we have? And all that is laid out, every quarter, in a really well-written document. 

They're really notorious for bringing red pens to a lot of these business reviews. And the way it works is we get into a room and everyone will read through the person's document with a red pen or some other way to mark up questions or thoughts they have about what's written there—

Corey: D minus you can do better than this. See me after class?

Alfonso: [laugh]. Yep, something like that. But really, it's to help that person think through what they're writing and what their plan is. We all want to see each other succeed, and that person puts a lot of thought into what they wrote, so we want to help them shape that from different people's experiences and diverse points of view. So, it's really awesome to see and something that I really, really enjoyed and will stay with me for the rest of my career, for sure.

Corey: So, what I always find interesting as well, and I'm not necessarily speaking to your specific situation, but the idea of a person leaving a company to go somewhere else, and then boomeranging, or coming back to the company that they left. Now, maybe that's because I can't envision doing that sort of thing myself, mostly because, again, I'm not a very good employee. My resignation letter was at one point carved into my boss's office door, and I'm generally not eligible for rehire. But what inspired you to leave, and what inspired you to return?

Alfonso: Yeah, I think honestly, in hindsight I enjoyed my experience at AWS, but in my mind, I was kind of optimizing for the wrong things. When I reflect back on it, I was optimizing for obviously just the prestige of working at AWS and what that entails. And really, as I reflected on when I was thinking about going back to Red Ventures, I'd rather just optimize for happiness. I think, ultimately, that's what's going to make me successful in my careers is if I do what I enjoy and really just focus on what will make me happy versus other ancillary things like money or prestige, or working at the cool startups in Silicon Valley or things like that. Everybody's going to have a different thing or a different view on what makes them happy and what they want to optimize for, but that's where I ended up.

Corey: There's something to be said as well for watching companies change. I mean, Red Ventures is, to my understanding, not exactly an ancient company. I mean, they've been around for a while, I've been to your office for a DevOpsDays that you folks hosted a couple years back, and, well ‘office’ doesn't really do justice. It's a campus. Especially for those of us who live in San Francisco. It's one of those, “Oh, wow, this would cost only several trillion dollars in real estate to make happen.” It's phenomenally huge and overwhelming, but to my understanding, it's not a company that's been around since the 1800s.

Alfonso: No. And I'll say that is definitely one of the things that also was exciting about coming back to our Ventures is our campus is beautiful, and it's kind of tucked in the suburbs of Charlotte. But yeah, we haven't been around that long. So, Red Ventures was founded in 2000 and we've grown significantly since then. Now we're up to, like, 3000 employees and in 10 cities across the US, and also we have offices in the UK and Brazil. 

But our campus, it’s—for me, it'd be hard to see another campus compete with what we have. It's pretty amazing. We have tennis courts, we have pickleball courts, bowling, basketball courts, obviously, we've hosted you for DevOps Day Charlotte in the arena auditorium that we have on the campus as well, so it's really awesome.

Corey: What I find interesting is despite the fact that they've only been around 20 years—my God, was 2000 really 20 years ago—you've become a large company, and obviously, Cloud wasn't really a thing when you were getting started. So, you've migrated into the Cloud—or are migrating into the Cloud. I'm sure you can give much more clarity on that than I can—but it also means that as a big company, it's not quite like a Twitter for Pets style migration, where, okay, I'm going to redeploy my single-tenant application—and even that's never simple—but okay, it took a few months, and then it was done. That's a lot of inertia, a lot of stuff to move, a lot of culture change that has to happen. What is that like? And you were gone for that long, did you see significant progression during your absence between the time you left and the time you returned?

Alfonso: Yeah. Honestly, we've had explosive growth in our usage of Cloud just in the past four and a half years since I first started there. But at a certain point, we took the stance of anything that we're going to develop that's a new project or a new application, we're just going to go straight to the Cloud and build it cloud-native and take advantage of the capabilities in AWS. We're not going to deal with hybrid-cloud. We don't want to spend our energy and focus on that; let's just start building the cloud-native muscles and learn more about the platform. 

And then over time, we'll transition to things that are still running in the data center and get those moved to the Cloud, which we're actively in the process of migrating our final apps that are running on-prem. So, it's been really exciting to see the growth, but what comes with that is a lot of waste and a lot of learning. Like, there are things that we didn't know about AWS, and how to optimize costs in the Cloud, and what the features and capabilities are. And even the things that we did learn, those things change. 

Now, serverless is a lot more mainstream than when we first started and changes how we approach building and designing applications. We want to take advantage of the ability to scale to zero and save costs, and shut things off when we don't need them, and all those things where we've been learning through over the past four years, and I think we're in a really good spot now.

Corey: People mistakenly believe the Cloud just means it runs on someone else's computer. In practice, it means it runs on money. And the capability story is fantastic, but there's no upper bound the way that there is at a data center where it's, “Well, we're completely out of capacity. We need to break ground on another one,” sort of limits what the upside cost it can potentially be in a short period of time, there is no upper bound with Cloud in the same way, it is not possible via conventional means to fill S3 in a given AWS region. 

Found out accidentally that it is not possible, and it reflected that in the bill. But there's definitely a story of discovering then that if everyone has the option to spin something up, that's great. There needs to be something that winds up as almost a following function of, “That's great. That experiment was good to know. We got results out of it. Now what? How do we enforce turning things off when people are done with them?” Because in practice, you spin something up? Great. You're going to retire before that instance does, left to its own devices.

Alfonso: Yeah, for sure. That's something my team has—has been a focus for my group over the past few months is, how do we tackle being more efficient in AWS and reducing our waste? We are a pretty large company with a lot of AWS accounts, and there are things that may not look like they're expensive in a single account, but when you scale out across 50, 100, 200, 300 accounts, that really adds up. It's reflected in the bill. So, for example, some of those smaller things we've been looking at is, do we need four NAT gateways in non-production accounts? Right? Like, obviously, in production—

Corey: No.

Alfonso: [laugh].

Corey: Sorry, did I say that out loud?

Alfonso: Right? So, we have account automation that will create accounts, but as part of that, we want to revisit that and say maybe we can just get by with a single NAT gateway and non-production accounts. We don't need the same level of redundancy there. What about CloudWatch log groups? 

Do we have retention policies by default with either our Terraform modules or just the ones that already exist? Are we just wasting money there? Duplicate CloudTrail trails is kind of a sneaky one. If you have an org-wide trail, that's free, but if you create a second trail, you're going to pay for all those events that you're already getting for free. And that adds up when you have hundreds of accounts. 

So, it's a lot of these really small things that to a single team that's operating in one or two accounts doesn't look crazy, but when you look at the big picture across the organization, you're spending a lot of money on those things.

Corey: The first, and challenging part, of course, is ensuring you have decent governance, ensuring that there's a relevant way of enforcing good practices, building guardrails in. But then the next question very quickly follows: how do you do that, organization-wide, in a way that doesn't absolutely suck and/or wreck the culture?

Alfonso: Yeah, I think I've definitely battled with that. We've gone through different iterations to figure out what works best, and I think we're still attacking that problem. One of the easy things that we've done is really leveraging what AWS already provides. So, in this case, like Trusted Advisor Cost Optimization Checks. It's a really easy place to start where you can go and get those across however many accounts you have, and just service that information to engineers. 

Typically, engineers want to do the right thing and optimize their costs, but they don't know where to start, or they're tied up building features, or providing business value. So, if we can reduce some of that friction there and just go through and aggregate all the findings that AWS is recommending, we don't have to go do that work to get the recommendations ourselves, we can leverage what they've already built, and we just service that to them. Maybe we create JIRA tickets for them; we track those month over month; we show what the total savings opportunity is since AWS already provides that information. So, we're just trying to find where we can plug in and glue things together and reduce some of the friction so we can service the savings opportunities to the right people.

Corey: The challenge, of course, is you want to be responsible, and, “Hey, who copied that extra petabyte of data that no one seems to need somewhere else?” And that's a painful experience that no one really wants to deal with unless they have no choice. But you also want to get out of the way and let people innovate because left to their own devices, first, you're going to have those same guardrails firing off, “Well hang on, you just spin up another instance, that's going to cost $7 a month. You better get a director-level approval to do that.” Which is ludicrous, and it also reinforces some of the bad behaviors that you'll see that make a lot of sense in a physical data center environment that makes zero sense at Cloud.

It shouldn't take you six weeks to provision an instance the way it would a physical server. And the heavier the process gets, the less likely people are to turn things off once they're done using them because it's so painful to get it back if they need to test it out again. I've talked to companies where when they're not running workloads on their test cluster, they'll have it do something like Folding@home, just so it doesn't show as idle, and then accounting yells at them. Which I think is just monstrous as far as having to subvert corporate process to do the right thing goes.

Alfonso: Yeah that's, I think, what we want to avoid. If you have too much process and really make it cumbersome for your engineers to do what they need to do for the projects and businesses they're working on, you're kind of just going to lose some of the trust there and honestly, probably have some more attrition because engineers want to be able to do what they need to do to move the business forward, and if there are roadblocks that don't make sense, or are kind of just arbitrary processes because that's the way we said it was best when we first started using the Cloud, you have to evolve those things. And I'm much more of a fan of the trust but verify model when it comes to both cloud security, but also just cost efficiency in the Cloud, and not necessarily blocking engineers upfront.

Corey: This episode is sponsored by our friends at New Relic. If you’re like most environments, you probably have an incredibly complicated architecture, which means that monitoring it is going to take a dozen different tools. And then we get into the advanced stuff. We all have been there and know that pain, or will learn it shortly, and New Relic wants to change that. They’ve designed everything you need in one platform with pricing that’s simple and straightforward, and that means no more counting hosts. You also can get one user and a hundred gigabytes a month, totally free. To learn more, visit newrelic.com.

Corey: One thing that falls directly under your wheelhouse and of course is near and dear to my heart is managing the AWS costs and managing the cost aspects of the AWS environment. And that's great. The problem that you have with that, start to finish, is that every time you talk to a company about, “How are you managing AWS costs?” They always give the same answer, specifically, “Terribly. We're sort of embarrassed about it, and therefore we're not going to talk about it.” 

I mean, I fix AWS bills for a living and I've never yet found any company that will stand up and say, “Oh, we've solved this problem. We're doing it's super well, and it's awesome.” The only folks who do are, “Great. What are you spending every month?” “Twenty whole dollars.” “Great.” That doesn't actually speak too much. 

And it gets worse when it becomes a purely engineering problem, specifically because you wind up having engineers who lack context. And I don't mean this as an insult to engineers, but you have this problem where an engineer, left to their own devices will gladly spend eight weeks trying to golf $200 a month off of their AWS developer environment. They cost more than that when they go for their first coffee break. There's no business value to optimizing at that scale. Then of course you have the $2 million a month thing running in AWS. Yeah, maybe spending a couple of weeks knocking a bill in half makes sense.

Alfonso: Yeah, absolutely right. You definitely don't want to optimize for things that, in the grand scheme of things, are not that important, are going to take up too much engineering time for little value in return. There's not a good ROI story there for the business, so you really want to just attack what are the biggest problems? What are the problems we have across all the accounts across the organization? What are the problems we have structurally? How can we improve the culture and have this flywheel effect of just building a culture of cost efficiency? Because I think that is what pays off and has a much better ROI. What are the inputs we can put in initially and just get this flywheel effect?

Corey: Part of the problem, I think, is building that flywheel and not getting lost. The challenge that I've seen about costing, by and large, is that it requires persistent sustained effort, but it's never a lasting priority for most companies. Usually, there's a freak-out whenever the AWS bill shows up that lasts a few days, and then something else takes precedence. But like death and taxes, the AWS bill shows up again at the following month, and restarts that process, and build more urgency. But it takes a lot of getting it wrong before companies start to see the value in getting it right. The painful part for me is that if you start off doing a few things that aren't that much additional work, you save yourself months of effort down the road.

Alfonso: Yeah. And the other thing is once you get to a certain size with your account footprint and your bill, it really, really takes dedicated people that are staying on top of the bill, reviewing the bill, and trying to optimize, managing RIs, which is just a nightmare. Thankfully, they have Compute Savings Plan now, but obviously, it only covers compute. I think my number one AWS wishlist item is for sure a data equivalent of the compute savings plan because that just makes things so much easier. Trying to optimize RI coverage, and then utilization, it’s just a moving target. When you're an organization of a certain size, you have so many engineers—

Corey: Oh, the chargeback or showback model becomes painful then, too, because you have an account that predicts their usage super well and says to buy exactly what they need, but those wind up applying to some other workload who decided to YOLO something into production. How do you attribute that back? Not for blame purposes, but for helping to optimize a footprint or understand a predictive model?

Alfonso: Exactly. And the entire process of how do you even validate that they need an RI for that workload? Is it something they're going to use for a year if you're looking at one-year RIs? There's so much legwork that's manual that has to go in there, it can become a nightmare when you get to a certain size, and staying on top of it's not really a game that you're going to win at with how they're structured today. And that's definitely one of our biggest pain points that we're still trying to get better at.

Corey: The hardest part for me is figuring out how do you fit whatever optimization approach you're taking into the company culture without breaking it. Because you don't want a bunch of engineers to generally have to think about this stuff, for the most part. I've been an advocate for a long time, of the idea that what most engineers need to know about AWS billing, fits on an index card. Sure you need in-house expertise past a certain scale; I'm not denying that at all. 

But that doesn't need to percolate out to every engineer working in the environment. There could be a centralized, almost SRE style model, where the cost optimization experts go around on a workload by workload basis and are available in a consult level. That seems to work better than teach every engineer to care about the bill. There's got to be a happy medium somewhere in there, but the hardest part is always culture.

Alfonso: Absolutely. You don't want your business engineers spending their time worrying about RIs or optimizing savings plan, or like, what do we need to buy for next month or some of those other minutiae details. That should be offloaded to a central team, or made easier through things like savings plan. You want them more focused on application optimization, in terms of costs. What are some of the things that we can tweak? Are we building things the right way from the get-go? Are we looking at surplus architectures, things that can scale to zero, in some of those optimizations, and not focused on some of the RIs and other workloads?

Corey: Part of the biggest thing that I think teams need to, I guess, wrap their heads around is it's important to focus on AWS cost control at some point. If it's not a problem for you right now, that's cool. We'll check back on you and a quarter or so because it's never a problem that goes away on its own. But at some point, conversely, it's time to stop focusing on it and stop caring because above some certain level of optimization or focus on attribution, learning to let go and have some wiggle room is always going to be the right answer. 

And you were mentioning earlier that serverless is an area of interest for you folks. This is also a different problem, particularly when communicating effectively with accounting. Because the idea behind serverless is that your usage is much more optimized, but it is also way more variable because it's completely beholden to customer workload.

Alfonso: Mm-hm. Yeah, absolutely. Although I still think in the end, you're definitely better off, if your use case fits it, designing almost an event-driven architecture or just working with serverless technologies because you still have the ability to scale to zero. You don't have to think about shutting things off when you're not using them, or when usage is slow or managing infrastructure in your non-production accounts. 

And I think we're getting better at that; we're moving more towards that direction, but we built a Terraform module to pause resources and non-prod and sandbox accounts because there was a need there, right? Because we still have a lot of applications that are running on EC2, or even just using things like RDS and Redshift that now you can pause; before you couldn't. So, we built a Terraform module that deploys a Lambda function that pauses those things on a schedule. So, if—obviously, on the weekend, if your team is not working, or overnight, you can pause your infrastructure running in your non-production accounts. And we've also added [00:25:40 unintelligible], like, send those events EventBridge so we can track the usage and the stop and start times. And in the past few months since we've deployed that module, we've saved 290,000 hours, resource hours, which is essentially bending time by saving 33 years in a three-month span.

Corey: There's something incredible to be said about that. It's fantastic to hear stories like that. The challenge, of course, is explaining to accounting that, “Oh, so what is your 18-month prediction look like as far as spend goes?” The honest answer that doesn't lead to nonsense is usually going to look an awful lot like, “Well, it's going to depend as a function of user growth and traffic.” This doesn't actually answer the question, but it does kick the can over to the BI folks rather than engineering in many cases. Again, culture is always the hardest part.

Alfonso: Yeah, I do not [laugh] envy the folks that have to go and figure out the forecasting long term with cloud spend because it's such a hard problem to solve. It's essentially impossible to forecast when you have variable growth and customer usage and you’re growing as a company. It's really hard to track.

Corey: Yeah, we've started doing that as a consulting option on our side, but invariably, it's always a second level of offering after first let's figure out what's in the account, and do an optimization pass, and understand what's going on in a meaningful, intelligent, intuitive way. That's never easy, and figuring out how to get there is always—I guess it's a journey more than it is a destination, which is usually the sort of thing you say about digital transformations when you're discussing that about a cloud bill. That's what drives accountants to drink.

Alfonso: Yep, absolutely. [laugh].

Corey: So, changing gears slightly, you and I first met a few years back. You're an organizer for DevOpsDays Charlotte. And I have a whole story about the time I attended there—in fact I’ve been there a couple of times now. But first, tell me about how you got, more or less, dragged into helping to organize a DevOpsDays, which if you look it up in the dictionary is the poster child for ‘thankless job.’

Alfonso: I don't know about that. It’s definitely rewarding. But yeah, it was a really interesting story how it came about. I remember attending DevOpsDays DC in 2015. I believe it was the first one; it may have been the second one, and Nathan Harvey is one of the organizers there. 

And they proposed an open space or breakout topic on organizing DevOpsDays. And I went to that, and Nathan Harvey showed up, and the only other person who showed up was Jason Hand, who was one of the organizers for DevOpsDays Denver at the time. So, I had them all to myself for 30 minutes, so I kind of just asked them what the process was, what do they learn from it? What were some of the pain points? 

And I went home from that I was like, man, that sounds really, really cool. It's awesome to get a community of folks together that are all interested in this DevOps thing. And a few months later—that was in June—in November we started the first ever DevOpsDay Charlotte. In a really crappy hotel by the airport. [laugh]. Not knowing what we were doing, but it was the first time and we learned a lot. And this past February, we ended up celebrating the fifth anniversary of DevOpsDay Charlotte and you also were at.

Corey: That still remains one of my favorite talks I've ever given. And it's probably time I told that backstory in public a bit. So, a few years ago, I gave a conference talk on things I've learned interviewing for jobs: how to handle job interviews, how to handle salary negotiation, the usual stuff you'd expect. The problem is, is all of this came from my own experience. So, what I accidentally built was how to handle job interviews as a white guy in tech, which is not the most accessible or inclusive talk, so I stopped giving it. 

And what I really liked about the keynote was it let me resurrect a highly, highly modified form of that talk that I co-presented with a friend of mine, Sonia Gupta, who before entering—and later leaving—tech, was an attorney, both as a district attorney as well as a defense attorney. So, she had a lot of thoughts on how negotiation worked, and she made the talk 1000 times better. It probably remains one of the most impactful talks I've ever given because usually, let's not kid ourselves, it's dumb jokes about cloud technology that's not going to stand the test of time. But interviewing skills are the sort of things that absolutely will pay dividends for someone's entire career. And I still get outreach over that talk. People discover the video every month and people ask questions.

Alfonso: That was a fantastic talk. We had a lot of feedback after that talk: the people loved that talk, and it was really, really impactful for them in, kind of, how they think about salary negotiation and job interviewing as a whole. That was an awesome talk.

Corey: Yeah, I hope to give another talk with her again, at some point. Just absolutely a fantastic experience. I found that on some level—and this is going to sound ridiculous, and I don't even care, I'm going for it anyway—I’ve gotten bored with some of the aspects of giving a talk where I get on stage, I know exactly how the entire talk is going to progress start to finish. Having someone else on stage makes it a lot more entertaining for me: you can bounce things off of someone, you're never going to give anything approaching the same talk twice, just because it goes back and forth so well, and it's really neat seeing that interaction. I've learned a lot because since then, I've started this podcast, among other things, and I've gotten better at having the ad-lib recording style of conversation where you don't really know where it's going to end up. I love that style of talk, and I'd love to see more people doing that.

Alfonso: Yeah, I think those are some of my favorite talks. You can tell it's not necessarily scripted, there's a little bit of improv mixed in, especially whenever you're on stage. [laugh]. But those are usually pretty rewarding and enjoyable.

Corey: I don't usually show people behind the scenes, but generally speaking, the way this podcast starts is we talked for five or ten minutes beforehand, just to get a broad outline of, make sure that the bio I have is up to date, and figure out, “Okay, what do you want to make sure we do or don't talk about?” “Cool, let's go.” But I don't have a giant list of topics I'm staring at of going point to point to point. I do it entirely in an improv basis. 

That's not because I'm awesome at this, it's because I'm terrible at planning ahead, so you've got to be good at improv when that stuff happens. And more often than not, it leads somewhere pretty uplifting and positive. But it's neat seeing that sort of thing unfold on talks because no two people have the same talk preparation approach, so it really forces you to learn to collaborate with people in a different way.

Alfonso: Yeah, and tying into just different talk formats. One of the other things I really love about DevOpsDays is the open space format, which, you know, you've attended plenty of them and have experience with, but it's just more of an audience-led style of interaction. People walk away from those initially, kind of like, “Eh, I don't know, if I'm going to like that,” or, “I don't know if I want to go to those.” But typically, once they go to a couple, they really enjoy that and it ends up being a favorite part of the conference for them. Because they get to talk to their peers and understand actual war stories, right? Like, things that are happening in companies, and challenges people have solved, and challenges they're still facing and just talking to those things. It's a lot of fun.

Corey: Yeah. I think there's a lot of value to it. And frankly, I've stopped speaking most of the DevOpsDays for the last year or so, primarily because let's face it, I'm trapped at home along with everyone else, and there's only so many times I can talk into a microphone. But also, in part, it's because I think the DevOpsDays are a terrific way to get started talking, and I am thrilled to see voices we have not heard before from folks who don't look like me, who are able to get up and start telling their stories. 

My offer that I make periodically, and it stands; I'll renew it now, is if anyone wants to give a talk but doesn't know how to get started or doesn't think they have anything to say, please reach out to me. That's one of the things I'm incredibly passionate about is helping people tell stories. It's not as hard as it looks. There's a lot of things you can do to make it easier on yourself, a whole bunch of shortcuts and cheats as it were. I've done a bunch of tweet threads on this, but those are always ephemeral. It's probably time for another one. And I'm very interested in helping people get on stage and learn to tell the story because I love it. And I like watching other people catch the gift as well.

Alfonso: Yeah, that is awesome. And honestly, the DevOps and DevOpsDays communities, one of the most inclusive and welcoming communities that I've been a part of. It's really awesome to see people share knowledge and are super supportive, especially for first-time speakers. I've noticed a lot of the DevOpsDays conference intentionally asked for first-time speakers because they're going to provide support for them and welcome them. And that's one of the other aspects I really love about being part of DevOpsDays.

Corey: Yeah. I think that's probably a good place to leave it. If people want to learn more about who you are, what you're doing, possibly come and work with you, where can they find you?

Alfonso: Yep, I'm on Twitter at @alfonzo__c. And we are definitely hiring. If any of these problems sounds interesting to you, we're looking for senior platform engineers, and also software engineers across the board. So, you can find us at redventures.com.

Corey: And I have to say, based upon conversations I had when I was out there with you folks, I did not get a big company feel, you know, except for the fact that we were on this enormous, gorgeous campus. But I didn't get a cultural sense of what most people think when they hear enterprise company at a large scale. It's not one of those cultures where nothing ever changes and everything is sort of frozen in amber. I know that that tends to be a common perception, but I don't believe that that's true.

Alfonso: Yeah, no, you're right—

Corey: Not in your case.

Alfonso: —we definitely take pride in acting like a startup. That's one of our core principles.

Corey: I'll take it a step further. You act like a startup without the incredibly toxic, damaging, and trash fire aspects that people often associate with startups.

Alfonso: [laugh]. Fantastic point. Yes, we have some really compelling belief statements that really give me pride about working at Red Ventures. One of those is we believe we can be the change we want to see in the world. And we're a company that believes we have a responsibility to interrupt social injustice and take concrete action to drive change, and you'll see that as well. And happy to chat with anybody that's interested in finding out more.

Corey: Excellent. I will, of course, include links to you in the [00:35:28 show notes] as well. Thank you so much for taking the time to speak with me today. I really appreciate it.

Alfonso: No problem. Thanks, Corey, for having me on.

Corey: Alfonso Cabrera, director of platform engineering at Red Ventures. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review at Apple Podcasts or whatever platform you'd like, whereas if you've hated this podcast, leave a five-star review anyway, but also be sure to include a comment about how my talks aren't nearly as good as I think they are.

Announcer: This has been this week’s episode of Screaming in the Cloud. You can also find more Corey at ScreamingintheCloud.com, or wherever fine snark is sold.

This has been a HumblePod production. Stay humble.

Corey: Alfonso Cabrera, director of platform engineering at Red Ventures. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review at Apple Podcasts or whatever platform you'd like, whereas if you've hated this podcast, leave a five-star review anyway, but also be sure to include a comment about how my talks aren't nearly as good as I think they are.

Announcer: This has been this week’s episode of Screaming in the Cloud. You can also find more Corey at ScreamingintheCloud.com, or wherever fine snark is sold.

This has been a HumblePod production. Stay humble.