Hacking Old Hardware and Developer Advocate Presentations with Darko Mesaroš

Darko is a Senior Developer Advocate based in Seattle, WA. His goal is to share his passion and technological know-how with Engineers, Developers, Builders, and tech enthusiasts across the world. If it can be automated, Darko will definitely try to do so. Most of his focus is towards DevOps and Management Tools, where automation, pipelines, and efficient developer tools is the name of the game – click less and code more so you do not repeat yourself ! Darko also collects a lot of old technology and tries to make it do what it should not. Like deploy AWS infrastructure through a Commodore 64.

Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: This episode is sponsored in part by our friends at Chronosphere. When it costs more money and time to observe your environment than it does to build it, there’s a problem. With Chronosphere, you can shape and transform observability data based on need, context and utility. Learn how to only store the useful data you need to see in order to reduce costs and improve performance at chronosphere.io/corey-quinn. That’s chronosphere.io/corey-quinn. And my thanks to them for sponsor ing my ridiculous nonsense.

Corey: Do you wish your developers had less permanent access to AWS? Has the complexity of Amazon's reference architecture for temporary elevated access caused you to sob uncontrollably? With Sym, you can protect your cloud infrastructure with customizable, just-in-time access workflows that can be setup in minutes. By automating the access request lifecycle, Sym helps you reduce the scope of default access while keeping your developers moving quickly. Say goodbye to your cloud access woes with Sym. Go to symops.com/corey to learn more. That’s S-Y-M-O-P-S.com/corey

Corey: Welcome to Screaming in the Cloud. I’m Cloud Economist Corey Quinn and my guest today is almost as bizarre as I am, in a somewhat similar direction. Darko Mesaroš is a Senior Developer Advocate at AWS. And instead of following my path of inappropriately using things as databases that weren’t designed to be used that way, he instead uses the latest of technology with the earliest of computers. Darko, thank you for joining me.

Darko: Thank you so much, Corey. First of all, you know, you tell me, Darko is a senior developer advocate. No, Corey. I’m a system administrator by heart. I happen to be a developer advocate these days, but I was born in the cold, cold racks of a data center. I maintain systems, I’ve installed packages on Linux systems. I even set up Solaris Zones a long time ago. So yeah, but I happen to yell into the camera these days, [laugh] so thank you for having me here.

Corey: No, no, it goes well. You started my career as a sysadmin. And honestly, my opinion, if you asked me—which no one does, but I share it anyway—is that the difference between an SRE and a sysadmin is about a 40% salary bump.

Darko: Exactly.

Corey: That’s about it. It is effectively the same job. The tools are different, the approach we take is different, but the fundamental mandate of ‘keep the site up’ has not materially changed.

Darko: It has not. I don’t know, like, what the modern SRS do, but like, I used to also semi-maintain AC units. Like, you have to walk around with a screwdriver nonetheless, so sometimes, besides just installing the freshest packages on your Red Hat 4 system, you have to also change the filters in the AC. So, not sure if that belongs into the SRE manifesto these days.

Corey: Well, the reason that I wound up inviting you onto the show was a recent blog post you put up where you were able to deploy Kubernetes from the best computer from 1986, which is the TRS-80, or the Trash-80. For the record, the worst computer from 1986 was—and remains—IBM Cloud. But that’s neither here nor there.

What does it mean to deploy Kubernetes because, to be direct, the way that I tend to deploy anything these days, if you know, I’m sensible and being grown up about it, is a Git push and then the automation takes it away from there. I get the sense, you went a little bit deep.

Darko: So, when it comes to deploying stuff from an old computer, like, you know, you kind of said the right thing here, like, I have the best computer from 1986. Actually, it’s a portable version of the best computer from 1986; it’s a TRS-80 Model 102. It’s a portable, basically a little computer intended for journalists and people on the go to write stuff and send emails or whatever it was back in those days. And I deployed Kubernetes through that system. Now, of course, I cheated a bit because the way I did it is I just used it as a glorified terminal.

I just hooked up the RS 232, the wonderful serial connection, to a Raspberry Pi somewhere out there and it just showed the stuff from a Raspberry Pi onto the TRS-80. So, the TRS-80 didn’t actually know how to run kubectl—or ‘kube cuddle,’ what they call it—it just asked somebody else to do it. But that’s kind of the magic of it.

Corey: You could have done a Lambda deployment then just as easily.

Darko: Absolutely. Like that’s the magic of, like, these old hunks of junks is that when you get down to it, they still do things with numbers and transmit electrical signals through some wires somewhere out there. So, if you’re capable enough, if you are savvy, or if you just have a lot of time, you can take any old computer and have it do modern things, especially now. Like, and I will say 15 years ago, we could have not done anything like this because 15 years ago, a lot of the stuff at least that I was involved with, which was Microsoft products, were click only. I couldn’t, for the love of me, deploy a bunch of stuff on an Active Directory domain by using a command line. PowerShell was not a thing back then. You could use VB Script, but sort of.

Corey: Couldn’t you wind up using something that would effect, like, Selenium or whatnot that winds up emulating a user session and moving the mouse to certain coordinates and clicking and then waiting some arbitrary time and clicking somewhere else?

Darko: Yes.

Corey: Which sounds like the absolute worst version of automation ever. That’s like, “I deployed Kubernetes using a typewriter.” “Well, how the hell did you do that?” “Oh, I use the typewriter to hit the enter key. Problem solved.” But I don’t think that counts.

Darko: Well, yeah, so actually even back then, like, just thinking of, like, a 10, 12-year step back to my career, I automated stuff on Windows systems—like Windows 2000, and Windows 2003 systems—by a tool called AutoIt. It would literally emulate clicks of a mouse on a specific location on the screen. So, you were just really hoping that window pops up at the same place all the time. Otherwise, your automation doesn’t work. So yeah, it was kind of like that.

And so, if you look at it that way, I could take my Trash-80, I could write an AutoIt script with specific coordinates, and I could deploy Windows things. So actually, yeah, you can deploy anything with these days, with an old computer.

Corey: I think that we’ve lost something in the world of computers. If I, like, throw a computer at you these days, you’re going to be pretty annoyed with me. Those things are expensive, it’ll probably break, et cetera. If I throw a computer from this era at you, your family is taking bereavement leave. Like, those things where—there would be no second hit.

These things were beefy. They were a sense of solidity to them. The keyboards were phenomenal. We’ve been chasing that high ever since. And, yeah, they were obnoxiously heavy and the battery life was 20 seconds, but it was still something that—you felt like it is computer time. And now, all these things have faded into the background. I am not protesting the march of progress, particularly in this particular respect, but I do miss the sense of having keyboards didn’t weren’t overwhelmingly flimsy plastic.

Darko: I think it’s just a fact of, like, we have computers as commodities these days. Back then computers were workstations, computers were something you would buy to perform a specific tasks. Today, computer is anything from watching Twitch to going on Twitter, complaining about Twitter, to deploying Kubernetes, right? So, they have become such commodities such… I don’t want to call them single-use items, but they’re more becoming single-use items as time progresses because they’re just not repairable anymore. Like, if you give me a computer that’s five years old, I don’t know what to do with it. I probably cannot fix it if it’s broken. But if you give me a computer that’s 35 years old, I bet you can fix it no matter what happened.

Corey: And the sheer compute changes have come so fast and furious, it’s easy to lose sight of them, especially with branding being more or less the same. But I saved up and took a additional loan out when I graduated high school to spend three grand on a Dell Inspiron laptop, this big beefy thing. And for fun, I checked the specs recently, and yep, that’s a Raspberry Pi these days; they’re $30, and it’s not going to work super well to browse the web because it’s underpowered. And I’m sitting here realizing wait a minute, even with a modern computer—forget the Raspberry Pi for a second—I’m sitting here and I’m pulling up web pages or opening Slack, or God forbid, Slack and Chrome simultaneously, and the fan spins up and it sounds incredibly anemic. And it’s, these things are magical supercomputers from the future. Why are they churning this hard to show me a funny picture of a cat? What’s going on here?

Darko: So, my theory on this is… because we can. We can argue about this, but we currently—

Corey: Oh, I think you’re right.

Darko: We have unlimited compute capacity in the world. Like, you can come up with an idea, you’re probably going to find a supercomputer out there, you’re probably going to find a cloud vendor out there that’s going to give you all of the resources you need to perform this massive computation. So, we didn’t really think about optimization as much as we used to do in the past. So, that’s it: we can. Chrome doesn’t care. You have 32 gigs of RAM, Corey. It doesn’t care that it takes 28 gigs of that because you have—

Corey: I have 128 gigs on this thing. I bought the Mac studio and maxed it out. I gave it the hostname of us-shitpost-1 and we run with it.

Darko: [laugh]. There you go. But like, I did some fiddling around, like, recently with—and again, this is just the torture myself—I did some 6502 Assembly for the Atari 2600. 6502 is a CPU that’s been used in many things, including the Commodore 64, the NES, and even a whole lot of Apple IIs, and whatnot. So, when you go down to the level of a computer that has 1.19 megahertz and it has only 128 bytes of RAM, you start to think about, okay, I can move these two numbers in memory in the following two ways: “Way number one will require four CPU cycles. Way number two will require seven CPU cycles. I’ll go with way number one because it will save me three CPU cycles.”

Corey: Oh, yeah. You take a look at some of the most advanced computer engineering out there and it’s for embedded devices where—

Darko: Yeah.

Corey: You need to wind up building code to run in some very tight constraints, and that breeds creativity. And I remember those days. These days, it’s well my computer is super-overpowered, what’s it matter? In fact, when I go in and I look at customers’ AWS bills, very often I’ll start doing some digging, and sure enough, EC2 is always the number one expense—we accept that—but we take a look at the breakdown and invariably, there’s one instance family and size that is the overwhelming majority, in most cases. You often a—I don’t know—a c5.2xl or something or whatever it happens to be.

Great. Why is that? And the answer—[unintelligible 00:10:17] to make sense is, “Well, we just started with that size and it seemed to work so we kept using it as our default.” When I’m building things, because I’m cheap, I take one of the smallest instances I possibly can—it used to be one of the Nanos and I’m sorry, half a gig or a gig of RAM is no longer really sufficient when I’m trying to build almost anything. Thanks, JavaScript. So okay, I’ve gone up a little bit.

But at that point, when I need to do something that requires something beefier, well, I can provision those resources, but I don’t have it as a default. That forces me to at least in the back of my mind, have a little bit of a sense of I should be parsimonious with what it is that I’m provisioning out there, which is apparently anathema to every data scientist I’ve ever met, but here we are.

Darko: I mean, that’s the thing, like, because we’re so used to just having those resources, we don’t really care about optimizations. Like, I’m not advocating that you all should go and just do assembly language. You should never do that, like, unless you’re building embedded systems or you’re working for something—

Corey: If you need to use that level of programming, you know.

Darko: Exactly.

Corey: You already know and nothing you are going to talk about here is going to impact what people in that position are doing. Mostly you need to know assembly language because that’s a weeder class and a lot of comp-sci programs and if you don’t pass it, you don’t graduate. That’s the only reason to really know assembly language most of the time.

Darko: But you know, like, it’s also a thing, like, as a developer, right, think about the person using your thing, right? And they may have the 128 gig us—what is it you called it? Us-shitpost-1, right—that kind of power, kind of, the latest and greatest M2 Max Ultra Apple computer that just does all of the stuff. You may have a big ‘ol double Xeon workstation that does a thing.

Or you just may have a Chromebook. Think about us with Chromebooks. Like, can I run your website properly? Did you really need all of those animations? Can you think about reducing the amount of animations depending on screen size? So, there’s a lot of things that we need to kind of think about. Like, it goes back to the thing where ‘it works on my machine.’ Oh, of course it works on your machine. You spent thousands of dollars on your machine. It’s the best machine in the world. Of course, it runs smoothly.

Corey: Wait 20 minutes and they’ll release a new one, and now, “Who sold me this ancient piece of crap?” Honestly, the most depressing thing is watching an Apple Keynote because I love my computer until I watch the Apple Keynote and it’s like, oh, like, “Look at this amazing keyboard,” and the keyboard I had was fine. It’s like, “Who sold me this rickety piece of garbage?” And then we saw how the Apple butterfly keyboard worked out for everyone and who built that rickety piece of garbage. Let’s go back again. And here we are.

Darko: Exactly. So, that’s kind of the thing, right? You know, like, your computer is the best. And if you develop for it, is great, but you always have to think other people who use it. Hence, containers are great to fix one part of that problem, but not all of the problems. So, there’s a bunch of stuff you can do.

And I think, like, for all of the developers out there, it’s great what you’re doing, you’re building us so many tools, but always that take a step back and optimize stuff. Optimize, both for the end-user by the amount of JavaScript you’re going to throw at me, and also for the back-end, think about if you have to run your web server on a Pentium III server, could you do it? And if you could, how bad would it be? And you don’t have to run it on a Pentium III, but like, try to think about what’s the bottom 5% of the capacity you need? So yeah, it’s just—you’ll save money. That’s it. You’ll save money, ultimately.

Corey: So, I have to ask, what you do day to day is you’re a senior developer advocate, which is, hmm, some words, yes. You spend a lot of your free time and public time talking about running ancient computers, but you also talk to customers who are looking forward, not back. How do you reconcile the two?

Darko: So, I like to mix the two. There’s a whole reason why I like old computers. Like, I grew up in Serbia. Like, when I was young in the ’90s, I didn’t have any of these computers. Like, I could only see, like, what was like a Macintosh from 1997 on TV and I would just drool. Like, I wouldn’t even come close to thinking about getting that, let alone something better.

So, I kind of missed all of that part. But now that I started collecting all of those old computers and just everything from the ’80s and ’90s, I’ve actually realized, well, these things are not that different from something else. So, I like to always make comparisons between, like, an old system. What does it actually do? How does it compare to a new system?

So, I love to mix and match in my presentations. I like to mix it, mix and match in my videos. You saw my blog posts on deploying stuff. So, I think it’s just a fun way to kind of create a little contrast. I do think we should still be moving forward. I do think that technology is getting better and better and it’s going to help people do so much more things faster, hopefully cheaper, and hopefully better.

So, I do think that we should definitely keep on moving forward. But I always have this nostalgic feeling about, like, old things and… sometimes I don’t know why, but I miss the world without the internet. And I think that without the internet, I think I miss the world with dial-up internet. Because back then you would go on the internet for a purpose. You have to do a thing, you have to wait for a while, you have to make sure nobody’s on the phone. And then—

Corey: God forbid you dial into a long-distance call. And you have to figure out which town and which number would be long distance versus not, at least where I grew up, and your parents would lose their freaking minds because that was an $8 phone call, which you know, back in the ’80s and early ’90s was significant. And yeah, great. Now, I still think is a great prank opportunity to teach kids are something that it costs more to access websites that are far away, which I guess in theory, it kind of does, but not to the end-user. I digress.

Darko: I have a story about this, and I’m going to take a little sidestep. But long-distance phone calls. Like in the ’80s, the World Wide Web was not yet a thing. Like, the www, the websites all, just the general purpose internet was not yet a thing. We had things called BBSes, or Bulletin Board Systems. That was the extreme version of a dial-up system.

You don’t dial into the internet; you dial into a website. Imagine if you have a sole intent of visiting only one website and the cost of visiting such a website would depend on where that website currently is. If the website is in Germany and you’re calling from Serbia, it’s going to cost you a lot of money because you’re calling internationally. I had a friend back then. The best software you can get were from American BBSes, but calling America from Serbia back then would have been prohibitively expensive, like, just insanely expensive.

So, what this friend used to do, he figured out if he would be connected to a BBS six hours a day, it would actually reset the counter of his phone bill. It would loop through a mechanical counter from whatever number, it would loop back again to that number. So, it would take around six and some hours to complete the loop the entire phone counting metric—whatever they use back in the ’80s—to kind of charge your bill, so it’s effectively cost him zero money back then. So yeah, it was more expensive, kids, back then to call websites, the further away the websites were.

[midroll 00:17:11]

Corey: So, developer advocates do a lot of things. And I think it is unfair, but also true that people tend to shorthand those of those things do getting on stage and giving conference talks because that at least is the visible part of it. People see that and it’s viscerally is understood that that takes work and a bit of courage for those who are not deep into public speaking and those who are, know it takes a lot of courage. And whereas writing a blog post, “Well, I have a keyboard and say dumb things on the internet all the time. I don’t see why that’s hard.” So, there’s a definite perception story there. What’s your take on giving technical presentations?

Darko: So, yeah. Just as you said, like, I think being a DA, even in my head was always represented, like, oh, you’re just on stage, you’re traveling, you’re doing presentations, you’re doing all those things. But it’s actually quite a lot more than that, right? We do a lot more. But still, we are the developer advocate. We are the front-facing thing towards you, the wonderful developers listening to this.

And we tend to be on stage, we tend to do podcasts with wonderful internet personalities, we tend to do live streams, we tend to do videos. And I think one of the key skills that a DA needs to have—a Developer Advocate needs to have—is presentations, right? You need to be able to present a technical message in the best possible way. Now, being a good technical presenter doesn’t mean you’re funny, doesn’t mean you’re entertaining, that doesn’t have to be a thing. You just need to take a complex technical message and deliver it in the best way possible so that everybody who has just given you their time, can get it fully.

And this means—well, it means a lot of things, but it means taking this complicated topic, distilling it down so it can be digested within 30 to 45 minutes and it also needs to be… it needs to be interesting. Like, we can talk about the most interesting topic, but if I don’t make it interesting, you’re just going to walk out. So, I also lead, like, a coaching class within internally, like, to teach people how to speak better and I’m working with, like, really good speakers there, but a lot of the stuff I say applies to no matter if you’re a top-level speaker, or if you’re, like, just beginning out. And my challenge to all of you speakers out there, like, anybody who’s listening to this and it has a plan to deliver a video, a keynote, a live stream or speak at a summit somewhere, is get outside of that box. Get outside of that PowerPoint box.

I’m not saying PowerPoint is bad. I think PowerPoint is a wonderful tool, but I’m just saying you don’t have to present in the way everybody else presents. The more memorable your presentation is, the more outside of that box it is, the more people will remember it. Again, you don’t have to be funny. You don’t have to be entertaining. You just have to take thing you are really passionate about and deliver it to us in the best possible way. What that best possible way is, well, it really depends. Like a lot of things, there is no concrete answer to this thing.

Corey: One of the hard parts I found is that people will see a certain technical presenter that they like and want to emulate and they’ll start trying to do what they do. And that works to a point. Like, “Well, I really enjoy how that presenter doesn’t read their slides.” Yeah, that’s a good thing to pick up. But past a certain point, other people’s material starts to fit as well as other people’s shoes and you’ve got to find your own path.

My path has always been getting people’s attention first via humor, but it’s certainly not the only way. In many contexts, it’s not even the most effective way. It works for me in the context in which I use it, but I assure you that when I’m presenting to clients, I don’t start off with slapstick comedy. Usually. There are a couple of noteworthy exceptions because clients expect that for me, in some cases.

Darko: I think one of the important things is that emulating somebody is okay, as you said, to an extent, like, just trying to figure out what the good things are, but good, very objectively good things. Never try to be funny if you’re not funny. That’s the thing where you can try comedy, but it’s very difficult to—it’s very difficult to do comedy if you’re not that good at it. And I know that’s very much a given, but a lot of people try to be funny when they’re obviously not funny. And that’s okay. You don’t have to be funny.

So, there are many of ways to get people’s attentions, by again, just throwing a joke. What I did once on stage, I threw a bottle at the floor. I was just—I said, I said a thing and threw a bottle at the floor. Everybody started paying attention all of a sudden at me. I don’t know why. So, it’s going to be that. It can be something—it can be be a shocking statement. When I say shocking, I mean, something, well, not bad, but something that’s potentially controversial. Like, for example, emacs is better than vim. I don’t know, maybe—

Corey: “Serverless is terrible.”

Darko: Serverl—yeah.

Corey: Like, it doesn’t matter. It depends on the audience.

Darko: It depends on the audience.

Corey: “The cloud is a scam.” I gave a talk once called, “The Cloud is A Scam,” and it certainly got people’s attention.

Darko: Absolutely. So, breaking up the normal flow because as a participant of a show, of a presentation, you go there you expect, look, I’m going to sit down, Corey’s going to come on stage and Corey says, “Hi, my name is Corey Quinn. I’m the CEO of The Duckbill Group. This is what I do. And welcome to my talk about blah.”

Corey: Tactically, my business partner, Mike, is the CEO. I don’t want to I don’t want to step too close to that fire, let’s be clear.

Darko: Oh, okay [laugh]. Okay. Then, “Today’s agenda is this. And slide one, slide two, slide three.” And that the expectation of the audience. And the audience comes in in this very autopilot way, like, “Okay, I’m just going to sit there and just nod my head as Corey speaks.”

But then if Corey does a weird thing and Corey comes out in a bathtub. Just the bathtub and Corey. And Corey starts talking about how bathtubs are amazing, it’s the best place to relax. “Oh, by the way, managing costs in the cloud is so easy, I can do it from a bathtub.” Right? All of a sudden, whoa [laugh], wait a second, this is something that’s interesting. And then you can go through your rest of your conversation. But you just made a little—you ticked the box in our head, like, “Oh, this is something weird. This is different. I don’t know what to expect anymore,” and people start paying more attention.

Corey: “So, if you’re managing AWS costs from your bathtub, what kind of computer do you use?” “In my case, a toaster.”

Darko: [laugh]. Yes. But ultimately, like, some of those things are very good and they just kind of—they make you as a presenter, unpredictable, and that’s a good thing. Because people will just want to sit on the edge of the seat and, like, listen to what you say because, I don’t know what, maybe he throws that toaster in, right? I don’t know. So, it is like that.

And one of the things that you’ll notice, Corey, especially if you see people who are more presenting for a longer time, like, they’ve been very common on events and people know them by name and their face, then that turns into, like, not just presenting but somebody comes, literally not because of the topic, but because they want to hear Corey talk about a thing. You can go there and talk about unicorns and cats, people will still come and listen to that because it’s Corey Quinn. And that’s where you, by getting outside of that box, getting outside of that ‘this is how we present things at company X,’ this is what you get in the long run. People will know who you are people will know, what not to expect from your presentations, and they will ultimately be coming to your presentations to enjoy whatever you want to talk about.

Corey: That is the dream. I really want to thank you for taking the time to talk so much about how you view the world and the state of ancient and modern technologies and the like. If people want to learn more, where’s the best place for them to find you?

Darko: The best way to find me is on twitch.tv/aws these days. So, you will find me live streaming twice a week there. You will find me on Twitter at @darkosubotica, which is my Twitter handle. You will find me at the same handle on Mastodon. And just search for my name Darko Mesaroš, I’m sure I’ll pop up on MySpace as well or whatever. So, I’ll post a lot of cloud-related things. I posted a lot of old computer-related things, so if you want to see me deploy Kubernetes through an Atari 2600, click that subscribe button or follow or whatever.

Corey: And we will, of course, include a link to this in the show notes. Thank you so much for being so generous with your time. I appreciate it.

Darko: Thank you so much, Corey, for having me.

Corey: Darko Mesaroš, senior developer advocate at AWS, Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry and insulting comment that you compose and submit from your IBM Selectric typewriter.

Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.