Whiteboard Confessional: Console Recorder: The Thing AWS Should Have Built

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Corey Quinn: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semi-polite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real-world forces us to build, and that the best to call your staging environment is “theory” because invariably, whatever you’ve built works in the theory, but not in production. Let’s get to it.

On this show, I talk an awful lot about architectural patterns that are horrifying. Let’s instead talk for a moment about something that isn’t horrifying. CHAOSSEARCH. Architecturally, they do things right. They provide a log analytics solution that separates out your storage from your compute. The data lives inside of your S3 buckets, and you can access it using APIs you’ve come to know and tolerate, through a series of containers that live next to that S3 storage. Rather than replicating massive clusters that you have to care and feed for yourself, instead, you now get to focus on just storing data, treating it like you normally would other S3 data and not replicating it, storing it on expensive disks in triplicate, and fundamentally not having to deal with the pains of running other log analytics infrastructure. Check them out today at CHAOSSEARCH.io.

You’ll notice that I periodically refer to various friends of the show as code terrorists. It’s never been explained, until now, exactly what that means and why I use that term. So, today, I thought I’d go ahead and help shine a little bit of light on that. One of the folks that I refer to most frequently is Ian Mckay, a fine gentleman based out of Australia. And he’s built something that is both amazing and terrible all at the same time, called Console Recorder. But let me back up before I dive into this monstrosity. Let’s talk about how we get things that we build in AWS into production. There are basically four tiers. Tier one is using the AWS web console itself, we click around and we build things. Great. Tier two is we use CloudFormation like sensible folks. Tier three is Terraform with all of its various attendant tooling, and then there’s the ultra tier four that I do, which is we use the AWS console and then we lie about it. Some folks are gonna play around here and say that oh, you should use the CDK, or something like that, or custom scripts that wind up spinning up production. 

And all of those are well and good, but only recently did CloudFormation release the ability to import existing resources. And even then, much like Terraform import, it’s pretty gnarly and not at all terrific. So, what do you wind up generally doing? Well, if you’re like me, you’ll stand up production resources inside of an AWS account. You will click around in the console—I always start with the console, not because I don’t know how these other tools work, that’s a side point, but rather because that helps me get a sense for how these services are imagined by the teams building them. They tend to assume that everyone who interacts with them is going to go through the console at some point, or at least they should. So, it gives me access and exposure to what their vision of this service is. Then once you’ve built something up, it often finds its way into production, if you at all like me, where I’ll spin something up just to test something and it works, and oh my stars, and suddenly you just want to get it out and not worry about it, so you don’t go back and rebuild it properly. 

So, now you’re left with this hand-built thing that’s just flapping around out there in production. What are you supposed to do? Well, according to the AWS experts, if we’re allowed to use that term to describe them, you’re supposed to smack yourself on the forehead, be convinced that you’re fundamentally missing the boat here, throw everything you’ve just built away and go back and do it properly. Which admittedly seems a little bit on the nose for those of us who’ve done exactly this far more times over the course of our career than we would care to count. So, today, however, I posit that there’s an alternate approach that doesn’t require support from AWS, which, to be honest, long ago seems to have given up on solving this particular problem in a way that human beings can understand. And I’d like to tell you about that, after this brief message from our sponsor.

In the late 19th and early 20th centuries, democracy flourished around the world. This was good for most folks, but terrible for the log analytics industry because there was now a severe shortage of princesses to kidnap for ransom to pay for their ridiculous implementations. It doesn’t have to be that way. Consider CHAOSSEARCH. The data lives in your S3 buckets in your AWS accounts, and we know what that costs. You don’t have to deal with running massive piles of infrastructure to be able to query that log data with APIs you’ve come to know and tolerate, and they’re just good people to work with. Reach out to CHAOSSEARCH.io. And my thanks to them for sponsoring this incredibly depressing podcast. 

This brings us back to where we started this conversation, with defining what a code terrorist was and pointing out Ian Mckay, out of Australia, who’s built something absolutely terrifying called Console Recorder for AWS. And this is a browser extension that works in Chrome, it works in Firefox, possibly others I stopped looking after those two. And what it does is you click a button in your browser when you start doing something inside the AWS console, and it does exactly what it says it would on the tin, where it starts recording what you’re doing. Their icon turns to a bright red record symbol, and you do whatever it is you’re going to do to spin something up. When you’re done, you hit the button again and say stop recording, and it opens a brand new tab. 

Inside of this tab lies wonder and magic that I am thrilled to be able to share with you. It’s terrifying, I mean, he is a code terrorist, but it is transformative. Specifically, it winds up spitting out the programmatic way of doing whatever the hell it is that you just did, but it spits that out in Boto 3, which is the Python SDK, it spits it out in CloudFormation, in Terraform, in Troposphere, which is a Python library built on top of CloudFormation—don’t get me started, it has its own challenges—CDK, for folks who can’t stop talking about the CDK long enough to build anything. And for those of us who tend to use our bash scripts, that’s right, it’s also got support for the AWS CLI. All of these are within sub-tabs within this page. And in all of them, it uses that specific language to explain what syntax and what commands you use to spit out those resources correctly, not at the high level of, oh, just do this magic. You could rerun these commands and spit out exactly what you had just done in the console. Oh, it also supports the JavaScript SDK for this. And lastly, the Go SDK, which is, in parentheses, version one. Now understand that Go is, of course, a project out of Google, so there’s always two versions of it. The version that they’re working on, and the version that is already deprecated. 

So, you wind up with a whole bunch of different options for ways to express that configuration of things you just clicked around inside of the console. But wait, there’s one more tab I want to talk to you about. IAM. Specifically, it defines very granular permissions for what you just did. For example, I just used this, before I recorded this, to spin up some S3 buckets. And it gives me very specific granular permissions for list bucket only on the bucket that I created. It does have some duplication because sometimes the console makes the same call several times, so, you’re going to want to edit this before turning it into something, but it’s a matter of taking things away, not necessarily of putting them back in. It’s always easier to cut than it is to add when it comes to overly wordy syntactic boilerplate. So, the reason that I’m bringing this up on the Whiteboard Confessional is because this is the sort of thing that A) should not have to exist, but it does. B) this is something that AWS should have built or something like this a long time ago of, it’s catching every API call you’re making whenever you build anything through the console, or through some weird script, or whatever tool you care to use, there is no reason in the world that they shouldn’t be able to capture those things and go ahead and tell you, oh, here’s how you would do that programmatically. 

I don’t mean to turn this into a cloud comparison story, but GCP already does this in the console. Hey, that thing you just did, here’s the command line interface if you were to do that right here in Cloud Shell in the browser. It’s incredibly helpful being able to take the thing you just clicked your way through and turning it into a command-line story that involves a lot of different fuzzy parameters, different syntaxes, different API syntaxes because of course there’s no API consistently here. That was a bit of a long story there, wasn’t it? And see, it really does tend to show the certain, I guess, depressing lengths that we’ll go to, in order to be able to A) build things and B) then replicate them in a borderline production-like environment. I should never be talking about this other than for this quick one-off thing that I built as an experiment. But I find myself using this more and more and more. That’s kind of a problem because it speaks to the fact that whenever I do something, I’m usually not the only person doing it—unless we’re talking about Route 53 is the database—it tells me that getting infrastructure as code is still too heavy of a lift for an awful lot of folks out there. 

The fact that this is not an extension that I dreamed up, or I wasn’t the only person that Ian told. I heard about this through the community. The fact that this has to exist, it speaks to the relatively immature state of infrastructure as code for production use in environments where you may not be starting out that way. It seems like AWS and the industry at large divide people into two buckets, one, uncultured swine like me who build things in the console, or the second bucket, very large companies that already have a robust CI/CD infrastructure pipeline, where they already have all of the tooling needed to build these things safely and sanely in production. In practice, most folks are somewhere between those two extremes. And the fact that it took a random third party building something like this before CloudFormation import even existed, tells me that this is a very clear pain for a lot of people. I talked to Ian on the Screaming in the Cloud podcast. It turns out that building this entire monstrosity took him a couple of months of nights and weekends work, and he did it by himself. And it’s disturbingly well polished for something like that. If he can build this in his spare time, why have we been asking for and not getting this for the entirety of AWS’s lifetime as a cloud services provider? That’s my rant. That’s my position on this. And frankly, I’m ashamed to admit that I love and use this service. Take a look. You can Google Console Recorder for AWS, the source is on GitHub. You need not trust Ian or me for this, but give it a try. For some of you out there, it’ll be transformative. 

Thank you for listening to the AWS Morning Brief Whiteboard Confessional. I’m Cloud Economist Corey Quinn, and I’ll talk to you next week. 

Announcer: Thank you for joining us on Whiteboard Confessional. If you have terrifying ideas, please reach out to me on twitter at @quinnypig and let me know what I should talk about next time.

This has been a HumblePod production. Stay humble.