Whiteboard Confessional: The Right and Wrong Way to Interview Engineers

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Corey: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semi-polite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real-world forces us to build, and that the best to call your staging environment is “theory”. Because invariably whatever you’ve built works in the theory, but not in production. Let’s get to it.

Sponsorships can be a lot of fun sometimes. ParkMyCloud asked, “Can we have one of our execs do a video webinar with you?” My response was, “Here’s a better idea. How about I talk to one of your customers instead, so you can pay to make fun of you.” And turns out, I’m super-convincing. So, that’s what’s happening. Join me and ParkMyCloud’s customer, Workfront, on July 23rd for a no-holds-barred discussion about how they’re optimizing AWS costs, and whatever other fights I manage to pick before ParkMyCloud realizes what’s going on and kills the feed. Visit parkmycloud.com/snark to register. That’s parkmycloud.com/snark.

Welcome. I am Cloud Economist Corey Quinn, and this is the AWS Morning Brief: Whiteboard Confessional; things that we see on whiteboards that we wish we could unsee. Today I want to talk about the worst whiteboard confessions of all time, and those invariably all tend to circle around what we ask candidates to do on a whiteboard during job interviews. There are a whole bunch of objections, problems, and other varieties of crappy opinions around whiteboarding as part of engineering job interviews, but they're all a part of the larger problem, which is that interviewing for engineering jobs fundamentally sucks. There are enough Medium articles on how trendy startups have cracked the interview to fill an S3 bucket. So, I'm going to take the contrarian position that all of these startups and all of these people who claim to have solved the problem, suck at it. 

And these terrible questions fall into a few common failure modes, most of which I've seen when they were levied at me back in my engineering days, and I was exercising my core competency of getting rapidly ejected from other companies. So, I spent a lot of time doing job interviews, and I kept seeing some of the same things appear. And they're all, of course, are different. But let’s start with some of the patterns. The most obnoxious one by far is the open-ended question of how would you solve a given problem? And as you start answering the question, they're paying more attention than you would expect. Maybe someone's on their laptop, quote-unquote ‘taking notes’ an awful lot. And I can't ever prove it, but it feels an awful lot—based upon the question—like, this is the kind of problem where you could suddenly walk out of the interview room, walk into the conference room next door and find a bunch of engineers currently in a war room trying to solve the question you were just asked. 

And what I hate about this pattern is it's a way of weaseling free work from interview candidates. If you want people to work on real-world problems, pay them. One of the best interviews I ever had is by a company that no longer exists called Three Rings Design. They wanted me to stand up some three-tiered web app, and turn on monitoring, and standard basic DevOps-style stuff; they gave an AWS account credential set to me. But what made them stand out is that they said, “Well, this is a toy problem. We're not going to use this in production, surprise. It's a generic question. But we don't believe in asking people to do work for free, so when you submit your results, we'll pay you a few hundred bucks.” And this was a standard policy they had for everyone who made it to that part of the interview. It was phenomenal, and I loved that approach. It solved that problem completely. But it's the only time I've ever seen it in my entire career. 

A variant of this horrible technique is to introduce the same type of problem, but it's with the proviso that this is a production problem that we had a few months ago. It's gone now, but how would you solve it? Now on its face, this sounds like a remarkably decent interview question. It's real-world. They've already solved it. So, whatever answer you give is not likely to be something revolutionary that's going to change how they approach things. So, what's wrong with it? Well, the problem is, is that in most cases, the right answer is going to look suspiciously like whatever they did to solve the problem when it manifested. 

I answered a question like this once with, “Well, what would strace tell me?” And the response was, “What does strace do?” I explained that it attached to processes and looked at the system calls that that process was making, and their response was, “Oh, yeah, that would have caught the problem. Huh. Okay, pretend strace doesn't exist.” Now it's not just the question of how you would solve the problem, but how you would solve the problem while being limited to their particular, often myopic, view of how systems work and how infrastructure needs to play out. This manifests itself the same way, by the way, in the world of various programming languages, and doing traditional developer engineer’s. It's awful because it winds up forcing you to see things through a perspective that you may not share. Part of the value of having you go and work somewhere is to bring your unique perspective. And, surprise, there's all these books on how to pass the technical interview. There are many fewer books on how to freaking give one that doesn't suck. And I wish that some people would write that book and that everyone else would read it. You can tell an awful lot about a company by how they go about hiring their people.

Corey: This episode is sponsored in part by ChaosSearch. Now their name isn’t in all caps, so they’re definitely worth talking to. What is ChaosSearch? A scalable log analysis service that lets you add new workloads in minutes, not days or weeks. Click. Boom. Done. ChaosSearch is for you if you’re trying to get a handle on processing multiple terabytes, or more, of log and event data per day, at a disruptive price. One more thing, for those of you that have been down this path of disappointment before, ChaosSearch is a fully managed solution that isn’t playing marketing games when they say “fully managed.” The data lives within your S3 buckets, and that’s really all you have to care about. No managing of servers, but also no data movement. Check them out at chaossearch.io and tell them Corey sent you. Watch for the wince when you say my name. That’s chaossearch.io.

Another very common failure mode for job interviews is when they ask a bunch of trivia questions. “What flag to this command does the following thing?” And what sucks there is it's a question of, “Have you seen this particular thing? And did it stick with you well enough to be able to more or less recite the manual back?” I don't know about you, but when I'm hiring, I believe in trying to find people who can add to the team, and if the best thing that someone can add is their ability to memorize and spit back trivia from man pages, I’m not that interested. 

I would much rather see what else they can bring. If you either know it or you don’t, that doesn't tell me much other than have you encountered this one thing in the wild. At some point, do enough job interviews, and you'll be able to pass any interview because you're able to answer all of their toy problems appropriately. One of my biggest pet peeves is when the job interview questions, or the way you're expected to answer them, bear almost no relation to how things work in real life. 

The Google phone screen for development roles is notorious for this. They make you write code in a blank Google Doc—pseudocode in most cases—but yeah, no one in the real world actually does this, in the same way that when you're trying to solve a problem, you're not writing code on a whiteboard in front of your coworkers unless you work on the third layer of hell. That's not how we think about these things. In one tab, I have a search engine open; in the other, I'm reading where I've—and also frantically copying and pasting from Stack Overflow, and a third I might be asking folks in a Slack channel, or an IRC network, or even on Twitter, how to solve particular issues that I'm running into. That's how real-world development works. It's not staring at a Google Doc, of all things, and it's not trying to solve these, I guess, ridiculous algorithm questions that sort out already solved problems.

So, that covers a lot of things that are terrible. What do I like to ask? Well, a common question that I've asked over the years that I love has been open-ended questions where people get the chance to talk about things that they're good at. Very often interviews turn into a game of ‘let me find out what you're the worst at and then needle you about it.’ It turns into, effectively then, hiring people based on an absence of weakness, rather than for particular strengths. And when you focus on that part of the story, it very quickly becomes this awful dynamic of having to set up these discussions where, “Oh, tell me what your worst at, and then I could beat the crap out of you on it for 45 minutes.” You can tell so much about a company by how they hire their people. Is that the dynamic of coworker you want to engage with? Not in my case. I've walked out of interviews for things like that because it doesn't go well. Yes, there's privilege inherent in being able to do that, but they're never nicer to you than when they're trying to get you to work there, and it's presumably all downhill from there. 

So, one question I love to ask is, have you heard of TinyURL? That actually does have a yes or no answer? And then the answer is no, you can explain what TinyURL does in about 30 seconds. It takes a long URL; it converts that to a much shorter URL, and stores it in a database. When someone visits the short URL, it returns a redirect to the longer URL. The end. You are now up to speed. 

And the question I like to give is, pretend that I know nothing about technology other than my ability to write large checks. This is known as being a venture capitalist. Now, I am writing you a large check to build a TinyURL equivalent service; something that will take in a short URL and return a redirect to a long URL. Go. What do I need? And it's completely open-ended, and what's fun about this is it's a small toy problem that encapsulates a whole bunch of different things. And you can go into database design, you can go into networking, you can go into how the system should be structured. You can talk about how you’d build this in Terraform, or CloudFormation or God forbid, something like Puppet or CFEngine. You can cover a lot of ground. You can even talk about the algorithmic complexity of some of the search queries and turn it into a whole interesting series of questions. 

But you'll notice that people will generally start moving in a direction when you give them a question like this, that leads directly to the thing that they're best at. Networking people will talk about the networks; systems people will talk about the systems, and it's great. It gives people an opportunity to talk about the things that they are best at, and it lets them shine, which is really what I'm after in the context of a job interview.

Then you can make the question more complex: “Okay, so it works for six months. This thing you build is awesome. Now it's slow. Where in the system is it going to be slow?” Well, people will be able to identify bottlenecks, ideally, and then you can start throwing in other things like I want it to be in multiple regions, and I want it to be able to withstand a regional outage, and how will your system scale to that point? 

And you can push back with constraints like this. And generally, it becomes somewhat frustrating at times, if you're not careful, so you want to make sure you frame it appropriately, but you also get to see how people have constructive discussions about constraints. If they get angry and start yelling at you, well, this is when they're supposed to be on their best behavior. In practice, having them go back and forth with you about different aspects is great. You want to get the questioning to a point where one of you is saying, I don't know in response to something the other person says because if you don't hit that depth, you're generally not getting to a point where you see the limits of someone's knowledge, and how they react when they reach it. 

Now, this is not a one size fits all interview question, but you can get an entire interview out of it if you want to. I'm not saying it's a silver bullet; I'm just saying that it's how I’ve liked to approach these things in the past for systems interviews. And it's certainly better than asking people to work for free or to do horrifying things that speak to terrible things about your culture.

This has been the AWS Morning Brief: Whiteboard Confessional ranting about interviews. I am Cloud Economist Corey Quinn, and this is the AWS Morning Brief. If you've enjoyed this podcast, please leave a five-star review on Apple Podcasts, whereas if you hated this podcast, please leave a five-star review on Apple Podcasts anyway, and implement QuickSort in the comments.

Thank you for joining us on Whiteboard Confessional. If you have terrifying ideas, please reach out to me on twitter at @quinnypig and let me know what I should talk about next time.

Announcer: This has been a HumblePod production. Stay humble.