Episode Summary
Today Corey sits down with swyx, head of developer experience at Airbyte, and so much more! They begin by chatting about swyx’s career history, professional motivation, and an industry taboo: following the money. Then Corey and swyx move into a discussion about the surprisingly challenging nature of developer experience and what it means to “learn in public.” swyx talks about expertise and how to quantify and demonstrate learning. Corey and swyx discuss swyx’s book “The Coding Career Handbook” and career coaching. swyx shares about his most recent foray into management in the era of zoom meetings, and conclude the conversation by talking about data integration and swyx’s latest job at Airbyte.
Episode Show Notes & Transcript
About swyx
swyx has worked on React and serverless JavaScript at Two Sigma, Netlify and AWS, and now serves as Head of Developer Experience at Airbyte. He has started and run communities for hundreds of thousands of developers, like Svelte Society, /r/reactjs, and the React TypeScript Cheatsheet. His nontechnical writing was recently published in the Coding Career Handbook for Junior to Senior developers.
Links Referenced:
- “Learning Gears” blog post: https://www.swyx.io/learning-gears
- The Coding Career Handbook: https://learninpublic.org
- Personal Website: https://swyx.io
- Twitter: https://twitter.com/swyx
Transcript
Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.
Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don’t leave managing your database to your cloud vendor because they’re too busy launching another half-dozen managed databases to focus on any one of them that they didn’t build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.
Corey: Let’s face it, on-call firefighting at 2am is stressful! So there’s good news and there’s bad news. The bad news is that you probably can’t prevent incidents from happening, but the good news is that incident.io makes incidents less stressful and a lot more valuable. incident.io is a Slack-native incident management platform that allows you to automate incident processes, focus on fixing the issues and learn from incident insights to improve site reliability and fix your vulnerabilities. Try incident.io, recover faster and sleep more.
Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. Some folks are really easy to introduce when I have them on the show because, “My name is, insert name here. I built thing X, and my job is Y at company Z.” Then we have people like today’s guest.
swyx is currently—and recently—the head of developer experience at Airbyte, but he’s also been so much more than that in so many different capacities that you’re very difficult to describe. First off, thank you for joining me. And secondly, what’s the deal with you?
swyx: [laugh]. I have professional ADD, just like you. Thanks for having me, Corey. I’m a—
Corey: It works out.
swyx: a big fan. Longtime listener, first time caller. Love saying that. [laugh].
Corey: You have done a lot of stuff. You have a business and finance background, which… okay, guilty; it’s probably why I feel some sense of affinity for a lot of your work. And then you went into some interesting directions. You were working on React and serverless YahvehScript—which is, of course, how I insist on pronouncing it—at Two Sigma, Netlify, AWS—a subject near and dear to my heart—and most recently temporal.io.
And now you’re at Airbyte. So, you’ve been focusing on a lot of, I won’t say the same things, but your area of emphasis has definitely consistently rhymed with itself. What is it that drives you?
swyx: So, I have been recently asking myself a lot of this question because I had to interview to get my new role. And when you have multiple offers—because the job market is very hot for DevRel managers—you have to really think about it. And so, what I like to say is: number one, working with great people; number two, working on great products; number three, making a lot of money.
Corey: There’s entire school of thought that, “Oh, that’s gauche. You shouldn’t mention trying to make money.” Like, “Why do you want to work here because I want to make money.” It’s always true—
swyx: [crosstalk 00:03:46]—
Corey: —and for some reason, we’re supposed to pretend otherwise. I have a lot of respect for people who can cut to the chase on that. It’s always been something that has driven me nuts about the advice that we give a new folks to the industry and peop—and even students figuring out their career path of, “Oh, do something you love and the money will follow.” Well, that’s not necessarily true. There are ways to pivot something you’d love into something lucrative and there are ways to wind up more or less borderline starving to death. And again, I’m not saying money is everything, but for a number of us, it’s hard to get to where we want to be without it.
swyx: Yeah, yeah. I think I’ve been cast with the kind of judgmental label of being very financially motivated—that’s what people have called me—for simply talking about it. And I’m like, “No. You know, it’s number three on my priority list.” Like, I will leave positions where I have a lot of money on the table because I don’t enjoy the people or the products, but having it up there and talking openly about it somehow makes you [laugh] makes you sort of greedy or something. And I don’t think that’s right. I tried to set an example for the people that I talk to or people who follow me.
Corey: One of the things I’ve always appreciated about, I guess, your online presence, which has remained remarkably consistent as you’ve been working through a bunch of different, I guess, stages of life and your career, is you have always talked in significant depth about an area of tech that I am relatively… well, relatively crap at, let’s be perfectly honest. And that is the wide world of most things front-end. Every time I see a take about someone saying, “Oh, front-end is junior or front-end is somehow less than,” I’d like to know what the hell it is they know because every time I try and work with it, I wind up more confused than I was when I started. And what I really appreciate is that you have always normalized the fact that this stuff is hard. As of the time that we’re recording this a day or so ago, you had a fantastic tweet thread about a friend of yours spun up a Create React App and imported the library to fetch from an endpoint and immediately got stuck. And then you pasted this ridiculous error message.
He’s a senior staff engineer, ex-Google, ex-Twitter; he can solve complex distributed systems problems and unable to fetch from a REST endpoint without JavaScript specialist help. And I talk about this a lot in other contexts, where the reason I care so much about developer experience is that a bad developer experience does not lead people to the conclusion of, “Oh, this is a bad interface.” It leads people to the conclusion, “Oh, I’m bad at this and I didn’t realize it.” No. I still fall into that trap myself.
I was under the impression that there was just this magic stuff that JS people know. And your tweet did so much to help normalize from my perspective, the fact that no, no, this is very challenging. I recently went on a Go exploration. Now, I’m starting to get into JavaScript slash TypeScript, which I think are the same thing but I’m not entirely certain of that. Like, oh, well, one of them is statically typed, or strongly typed. It’s like, “Well, I have a loud mechanical keyboard. Everything I do is typing strongly, so what’s your point?”
And even then we’re talking past each other in these things. I don’t understand a lot of the ecosystem that you live your career in, but I have always had a tremendous and abiding respect for your ability to make it accessible, understandable, and I guess for lack of a better term, to send the elevator back down.
swyx: Oh, I definitely think about that strongly, especially that last bit. I think it’s a form of personal growth. So, I think a lot of people, when they talk about this sending the elevator back down, they do it as a form of charity, like I’m giving back to the community. But honestly, you actually learn a lot by trying to explain it to others because that’s the only way that you truly know if you’ve learned something. And if you ever get anything wrong, you’ll—people will never let you forget it because it is the internet and people will crawl over broken glass to remind you that you’re wrong.
And once you’ve got it wrong, you will—you know, you’ve been so embarrassed that you’ll never forget it. So, I think it’s just a really good way to learn in public. And that’s kind of the motto that I’m kind of known for. Yeah, we can take the direction anywhere you want to go in JavaScript land. Happy to talk about it all day. [laugh].
Corey: Well, I want to start by something you just said where you’re doing the learning in public thing. And something I’ve noticed is that there are really two positions you can take—in the general sense—when you set out to make a bit of a reputation for yourself in a particular technical space. You can either do the, “I’m a beginner here, same as the rest of you, and I’m learning in public,” or you can position yourself as something of an expert. And there are drawbacks and advantages to both. I think that if you don’t look as wildly over-represented as I do, both of them are more fraught in different ways, where it’s, “Oh, you’re learning in public. Ah, look at the new person, she’s dumb.”
Or if you’re presenting yourself as an expert, you get nibbled to death by ducks on a lot of the deep technical nuances and well, actually’ed to death. And my position has always been and this is going to be a radical concept for some folks, is that I’m genuinely honest. I tend to learn in public about the things that I don’t know, but the things that I am something of a subject matter expert in—like, I don’t know, cloud billing—I don’t think that false modesty necessarily serves me particularly well. It’s yeah, I know exactly what I’m talking about here. Pretending otherwise it’s just being disingenuous.
swyx: I try to think of it as having different gears of learning in public. So, I’ve called this “Learning Gears” in a previous blog post of mine, where you try to fit your mode of learning to the terrain that you’re on, your domain expertise, and you should never over-represent the amount that you know because I think people are very rightly upset when there are a lot of people—let’s say on Twitter, or YouTube, or Udemy even—who present themselves as experts who are actually—they just read the docs the previous night. So, you should try not to over-represent your expertise.
swyx: I try to think of it as having different gears of learning in public. So, I’ve called this “Learning Gears” in a previous blog post of mine, where you try to fit your mode of learning to the terrain that you’re on, your domain expertise, and you should never over-represent the amount that you know because I think people are very rightly upset when there are a lot of people—let’s say on Twitter, or YouTube, or Udemy even—who present themselves as experts who are actually—they just read the docs the previous night. So, you should try not to over-represent your expertise.
But at the same time, don’t let your imposter syndrome stop you from sharing what you are currently learning and taking corrections when you’re wrong. And I think that’s the tricky balance to get which is constantly trying to put yourself out there while accepting that you might be wrong and not getting offended when or personally attacked when someone corrects you, inevitably. And sometimes people will—especially if you have a lot of followers, people will try to say—you know, someone of your following—you know, it’s—I kind of call this follower shaming, like, you should act, uh—invulnerable, or run every tweet through committee before you tweet after a certain sort of following size. So, I try to not do that and try to balance responsibility with authenticity.
Corey: I think that there’s something incredibly important about that, where there’s this idea that you either become invulnerable and get defensive and you yell at people, and down that path lies disaster because, believe it or not, we all get it wrong from time to time, and doubling down and doubling down and doubling down again, suddenly, you’re on an island all by yourself and no one respectable is going to be able to get there to help you. And the other side of it is going too far in the other direction, where you implicitly take any form of criticism whatsoever as being de facto correct. And I think that both paths don’t lead to super great places. I think it’s a matter of finding our own voices and doing a little bit of work as far as the validity of accepting a given piece of feedback goes. But other than that, I’m a big fan of being able to just more or less be as authentic as possible.
And I get that I live in a very privileged position where I have paths open to me that are not open to most folks. But in many respects so to you are one of the—easily—first five people I would think of if someone said, “Hey if I need to learn JavaScript for someone, who should I talk to first?” You’re on that list. And you’ve done a lot of things in this area, but you’ve never—you alluded to it a few minutes ago, but I’m going to call it out a little more pointedly—without naming names, let’s be clear—and that you’re never presented as a grifter, which is sort of the best way I can think of it of, “Well, I just learned this new technology stack yesterday and now I’m writing a book that I’m going to sell to people on how to be an expert at this thing.” And I want to be clear, this is very distinct from gatekeeping because I think that, “Oh, well, you have to be at least this much of an expert—” No, but I think that holding yourself out as I’m going to write a book on how to be proud of how to become a software engineer.
Okay, you were a software engineer for six months, and more to the point, knowing how to do a thing and knowing how to teach a thing are orthogonal skill sets, and I think that is not well understood. If I ever write a book or put something—or some sort of info product out there, I’m going to have to be very careful not to fall into that trap because I don’t want to pretend to be an expert in things that I’m not. I barely think I’m an expert in things that I provable am.
swyx: there are many ways to answer that. So, I have been accused a couple of times of that. And it’s never fun, but also, if you defend yourself well, you can actually turn a critic into a fan, which I love doing.
Corey: Mm-hm.
Corey: Mm-hm.
swyx: [laugh].
Corey: Oh yes.
swyx: what I fall back to, so I have a side interest in philosophy, based on one of my high school teachers giving us, like, a lecture in philosophy. I love him, he changed my life. [Lino Barnard 00:13:20], in case—in the off chance that he’s listening. So, there’s a theory of knowledge of, like, how do you know what you know, right? And if you can base your knowledge on truth—facts and not opinions, then people are arguing with the facts and not the opinions.
And so, getting as close to ground truth as possible and having certainty in your collection of facts, I think is the basis of not arguing based on identity of, like, “Okay, I have ten years experience; you have two years experience. I am more correct than you in every single opinion.” That’s also not, like, the best way to engage in the battlefield of ideas. It’s more about, do you have the right amount of evidence to support the conclusions that you’re trying to make? And oftentimes, I think, you know, that is the basis, if you don’t have that ability.
Another thing that I’ve also done is to collect the opinions of others who have more expertise and present them and curate them in a way that I think adds value without taking away from the individual original sources. So, I think there’s a very academic way [laugh] you can kind of approach this, but that defends your intellectual integrity while helping you learn faster than the typical learning rate. Which is kind of something I do think about a lot, which is, you know, why do we judge people by the number of years experience? It’s because that’s usually the only metric that we have available that is quantifiable. Everything else is kind of fuzzy.
Another thing that I’ve also done is to collect the opinions of others who have more expertise and present them and curate them in a way that I think adds value without taking away from the individual original sources. So, I think there’s a very academic way [laugh] you can kind of approach this, but that defends your intellectual integrity while helping you learn faster than the typical learning rate. Which is kind of something I do think about a lot, which is, you know, why do we judge people by the number of years experience? It’s because that’s usually the only metric that we have available that is quantifiable. Everything else is kind of fuzzy.
But I definitely think that, you know, better algorithms for learning let you progress much faster than the median rate, and I think people who apply themselves can really get up there in terms of the speed of learning with that. So, I spend a lot of time thinking about this stuff. [laugh].
Corey: It's a hard thing to solve for. There’s no way around it. It’s, what is it that people should be focusing on? How should they be internalizing these things? I think a lot of it starts to with an awareness, even if not in public, just to yourself of, “I would like advice on some random topic.” Do you really? Are you actually looking for advice or are you looking—
swyx: right.
Corey: For validation? Because those are not the same thing, and you are likely to respond very differently when you receive advice, depending on which side of that you’re coming from.
swyx: Yeah. And so, one way to do that is to lay out both sides, to actually demonstrate what you’re split on, and ask for feedback on specific tiebreakers that would help your decision swing one way or another. Yeah, I mean, there are definitely people who ask questions that are just engagement bait or just looking for validation. And while you can’t really fix that, I think it’s futile to try to change others’ behavior online. You just have to be the best version of yourself you can be. [laugh].
Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.
Corey: So, you wrote a book that is available at learninpublic.org, called The Coding Career Handbook. And to be clear, I have not read this myself because at this point, if I start reading a book like that, and you know, the employees that I have see me reading a book like that, they’re going to have some serious questions about where this company is going to be going soon. But scrolling through the site and the social proof, the testimonials from various people who have read it, more or less read like a who’s-who of people that I respect, who have been on this show themselves.
Emma Bostian is fantastic at explaining a lot of these things. Forrest Brazeal is consistently a source to me of professional envy. I wish I had half his musical talent; my God. And your going down—it explains, more or less, the things that a lot of folks people are all expected to know but no one teaches them about every career stage, ranging from newcomer to the industry to senior. And there’s a lot that—there’s a lot of gatekeeping around this and I don’t even know that it’s intentional, but it has to do with the idea that people assume that folks, quote-unquote, “Just know” the answer to some things.
Oh, people should just know how to handle a technical interview, despite the fact that the skill set is completely orthogonal to the day-to-day work you’ll be doing. People should just know how to handle a performance review, or should just know how to negotiate for a raise, or should just know how to figure out is this technology that I’m working on no longer the direction the industry is going in, and eventually I’m going to wind up, more or less, waiting for the phone to ring because there’s only three companies in the world left who use it. Like, how do you keep—how do you pay attention to what’s going on around you? And it’s the missing manual that I really wish that people would have pointed out to me back when I was getting started. Would have made life a lot easier.
swyx: Oh, wow. That’s high praise. I actually didn’t know we’re going to be talking about the book that much. What I will say is—
Corey: That’s the problem with doing too much. You never know what people have found out about you and what they’re going to say when they drag you on to a podcast.
swyx: got you, got you. Okay. I know, I know, I know where this is going. Okay. So, one thing that I really definitely believe is that—and this happened to me in my first job as well, which is most people get the mentors that they’re assigned at work, and sometimes you have a bad roll the dice. [laugh].
And you’re supposed to pick up all the stuff they don’t teach you in school at work or among your friend group, and sometimes you just don’t have the right network at work or among your friend group to tell you the right things to help you progress your career. And I think a lot of this advice is written down in maybe some Hacker News posts, some Reddit posts, some Twitter posts, and there’s not really a place you to send people to point to, that consolidates that advice, particularly focused at the junior to senior stage, which is the stage that I went through before writing the book. And so, I think that basically what I was going for is targeting the biggest gap that I saw, which is, there a lot of interview prep type books like Crack the Coding Career, which is kind of—Crack the Coding Interview, which is kind of the book title that I was going after. But once you got the job, no one really tells you what to do after you got that first job. And how do you level up to the senior that everyone wants to hire, right? There’s—
Corey: “Well, I’ve mastered cracking the coding interview. Now, I’m really trying to wrap my head around the problem of cracking the showing up at work on time in the morning.” Like, the baseline stuff. And I had so many challenges with that early in my career. Not specifically punctuality, but just the baseline expectation that it’s just assumed that by the time you’re in the workplace earning a certain amount of money, it’s just assumed that you have—because in any other field, you would—you have several years of experience in the workplace and know how these things should play out.
No, the reason that I’m sometimes considered useful as far as giving great advice on career advancement and the rest is not because I’m some wizard from the future, it’s because I screwed it all up myself and got censured and fired and rejected for all of it. And it’s, yeah, I’m not smart enough to learn from other people’s mistakes; I got to make them myself. So, there’s something to be said for turning your own missteps into guidance so that the next person coming up has an easier time than you did. And that is a theme that, from what I have seen, runs through basically everything that you do.
swyx: I tried to do a lot of research, for sure. And so, one way to—you know, I—hopefully, I try not to make mistakes that others have learned, have made, so I tried to pick from, I think I include 1500 quotes and sources and blog posts and tweets to build up that level of expertise all in one place. So hopefully, it gives people something to bootstrap your experience off of. So, you’re obviously going to make some mistakes on your own, but at least you have the ability to learn from others, and I think this is my—you know, I’m very proud of the work that I did. And I think people have really appreciated it.
Because it’s a very long book, and nobody reads books these days, so what am I doing [laugh] writing a book? I think it’s only the people that really need this kind of advice, that they find themselves not having the right mentorship that reach out to me. And, you know, it’s good enough to support a steady stream of sales. But more importantly, like, you know, I am able to mentor them at various levels from read my book, to read my free tweets, to read the free chapters, or join the pay community where we have weekly sessions going through every chapter and I give feedback on what people are doing. Sometimes I’ve helped people negotiate their jobs and get that bump up to senior staff—senior engineer, and I think more than doubled their salary, which was very personal proud moment for me.
But yeah, anyway, I think basically, it’s kind of like a third place between the family and work that you could go to the talk about career stuff. And I feel like, you know, maybe people are not that open on Twitter, but maybe they can be open in a small community like ours.
Corey: There’s a lot to be said for a sense of professional safety and personal safety around being—having those communities. I mean, mine, when I was coming up was the freenode IRC network. And that was great; it’s pseudo-anonymous, but again, I was Corey and network staff at the time, which was odd, but it was great to be able to reach out and figure out am I thinking about this the wrong way, just getting guidance. And sure, there are some channels that basically thrived on insulting people. I admittedly was really into that back in the early-two-thousand-nothings.
And, like, it was always fun to go to the Debian channel. It’s like, “Yeah, can you explain to me how to do this or should I just go screw myself in advance?” Yeah, it’s always the second one. Like, community is a hard thing to get right and it took me a while to realize this isn’t the energy I want in the world. I like being able to help people come up and learn different things.
I’m curious, given your focus on learning in public and effectively teaching folks as well as becoming a better engineer yourself along the way, you’ve been focusing for a while now on management. Tell me more about that.
swyx: I wouldn’t say it’s been, actually, a while. Started dabbling in it with the Temporal job, and then now fully in it with Airbyte.
Corey: You have to know, it has been pandemic time; it has stood still. Anything is—
swyx: exactly.
Corey: —a while it given that these are the interminable—this is the decade of Zoom meetings.
swyx: [laugh]. I’ll say I have about a year-and-a-half of it. And I’m interested in it partially because I’ve really been enjoying the mentoring side with the coding career community. And also, I think, some of the more effective parts of what I do have to be achieved in the planning stages with getting the right resources rather than doing the individual contributor work. And so, I’m interested in that.
I’m very wary of the fact that I don’t love meetings myself. Meetings are a means to an end for me and meetings are most of the job in management time. So, I think for what’s important to me there, it is that we get stuff done. And we do whatever it takes to own the outcomes that we want to achieve and try to manage people’s—try to not screw up people’s careers along the way. [laugh]. Better put, I want people to be proud of what they get done with me by the time they’re done with me. [laugh].
Corey: So, I know you’ve talked to me about this very briefly, but I don’t know that as of the time of this recording, you’ve made any significant public statements about it. You are now over at Airbytes, which I confess is a company I had not heard of before. What do y’all do over there?
swyx: [laugh]. “What is it we do here?” So Airbyte—
Corey: Exactly. Consultants want to know.
swyx: Airbyte’s a data integration company, which means different things based on your background. So, a lot of the data engineering patterns in, sort of, the modern data stack is extracting from multiple sources and loading everything into a data warehouse like a Snowflake or a Redshift, and then performing analysis with tools like dbt or business intelligence tools out there. We like to use MetaBase, but there’s a whole there’s a whole bunch of these stacks and they’re all sort of advancing at different rates of progress. And what Airbyte would really like to own is the data integration part, the part where you load a bunch of sources, every data source in the world.
What really drew me to this was two things. One, I really liked the vision of data freedom, which is, you have—you know, as—when you run a company, like, a typical company, I think at Temporal, we had, like, 100, different, like, you know, small little SaaS vendors, all of them vying to be the sources of truth for their thing, or a system of record for the thing. Like, you know, Salesforce wants to be a source of truth for customers, and Google Analytics want to be source of truth for website traffic, and so on and so forth. Like, and it’s really hard to do analysis across all of them unless you dump all of them in one place.
So one, is the mission of data freedom really resonates with me. Like, your data should be put in put somewhere where you can actually make something out of it, and step one is getting it into a format in a place that is amenable for analysis. And data warehouse pattern has really taken hold of the data engineering discipline. And I find, I think that’s a multi-decade trend that I can really get behind. That’s the first thing.
Corey: I will say that historically, I’m bad at data. All jokes about using DNS as a database aside, one of the reasons behind that is when you work on stateless things like web servers and you blow trunks and one of them, oops. We all laugh, we take an outage, so maybe we’re not laughing that hard, but we can reprovision web servers and things are mostly fine. With data and that going away, there are serious problems that could theoretically pose existential risk to the business. Now, I was a sysadmin and a, at least mediocre one, which means that after the first time I lost data, I was diligent about doing backups.
Even now, the data work that we do have deep analysis on our customers’ AWS bills, which doesn’t sound like a big data problem, but I assure you it is, becomes something where, “Okay, step one. We don’t operate on it in place.” We copy it into our own secured environment and then we begin the manipulations. We also have backups installed on these things so that in the event that I accidentally the data, it doesn’t wind up causing horrifying problems for our customers. And lastly, I wind up also—this is going to surprise people—I might have securing the access to that data by not permitting writes.
Turns out it’s really hard—though apparently not impossible—to delete data with read-only calls.
swyx: [crosstalk 00:28:12].
Corey: It tends to be something of just building guardrails against myself. But the data structures, the understanding the analysis of certain things, I would have gotten into Go way sooner than I did if the introduction to Go tutorial on how to use it wasn’t just a bunch of math problems talking about this is how you do it. And great, but here in the year of our lord 2022, I mostly want a programming language to smack a couple of JSON objects together and ideally come out with something resembling an answer. I’m not doing a whole lot of, you know, calculating prime numbers in the course of my week. And that is something that took a while for me to realize that, no, no, it’s just another example of not being a great way of explaining something that otherwise could be incredibly accessible to folks who have real problems like this.
I think the entire field right now of machine learning and the big data side of the universe struggles with this. It’s, “Oh, yeah. If you have all your data, that’s going to absolutely change the world for you.” “Cool. Can you explain how?” “No. Not effectively anyway.” Like, “Well, thanks for wasting everyone’s time. It’s appreciated.”
swyx: Yeah, startup is sitting on a mountain of data that they don’t use and I think everyone kind of feels guilty about it because everyone who is, like, a speaker, they’re always talking about, like, “Oh, we used our data to inform this presidential campaign and look at how amazing we are.” And then you listen to the podcasts where the data scientists, you know, talk amongst themselves and they’re like, “Yeah, it’s bullshit.” Like, [laugh], “We’re making it up as we go along, just like everyone else.” But, you know, I definitely think, like, some of the better engineering practices are arising under this. And it’s professionalizing just like front-end professionalized maybe ten years ago, DevOps professionalized also, roughly in that timeframe, I think data is emerging as a field that is just a standalone discipline with its own tooling and potentially a lot of money running through it, especially if you look at the Snowflake ecosystem.
So, that’s why I’m interested in it. You know, I will say there’s also—I talked to you about the sort of API replication use case, but also there’s database replication, which is kind of like the big use case, which, for example, if you have a transactional sort of SQL database and you want to replicate that to an analytical database for queries, that’s a very common one. So, I think basically data mobility from place to place, reshaping it and transferring it in as flexible manner as possible, I think, is the mission, and I think there’s a lot of tooling that starts from there and builds up with it. So, Airbyte integrates pretty well with Airflow, Dexter, and all the other orchestration tools, and then, you know, you can use dbt, and everything else in that data stack to run with it. So, I just really liked that composition of tools because basically when I was a hedge fund analyst, we were doing the ETL job without knowing the name for it or having any tooling for it.
I just ran a Python script manually on a cron job and whenever it failed, I would have to get up in the middle of night to go kick it again. It’s, [laugh] it was that bad in 2014, ’15. So, I really feel the pain. And, you know, the more data that we have to play around with, the more analysis we can do.
Corey: I’m looking forward to seeing what becomes of this field as folks like you get further and further into it. And by, “Well, what do you mean, folks like me?” Well, I’m glad you asked, or we’re about to as I put words in your mouth. I will tell you. People who have a demonstrated ability not just to understand the technology—which is hard—but then have this almost unicorn gift of being able to articulate and explain it to folks who do not have that level of technical depth in a way that is both accessible and inviting. And that is no small thing.
If you were to ask me to draw a big circle around all the stuff that you’ve done in your career and define it, that’s how I would do it. You are a storyteller who is conversant with the relevant elements of the story in a first-person perspective. Which is probably a really wordy way to put it. We should get a storyteller to workshop that, but you see the point.
swyx: I try to call it, like, accessibly smart. So, it’s a balance that you want to make, where you don’t want to talk down to your audience because I think there are a lot of educators out there who very much stay at the basics and never leave that. You want to be slightly aspirational and slightly—like, push people to the bounds of their knowledge, but then not to go too far and be inaccessible. And that’s my sort of polite way of saying that I dumb things down as service. [laugh].
Corey: But I like that approach. The term dumbing it down is never a phrase to use, as it turns out, when you’re explaining it to someone. It’s like, “Let me dumb that down for you.” It’s like, yeah, I always find the best way to teach someone is to first reach them and get their attention. I use humor, but instead we’re going to just insult them. That’ll get their attention all right.
swyx: No. Yeah. It does offend some people who insist on precision and jargon. And I’m quite against that, but it’s a constant fight because obviously there is a place at time for jargon.
Corey: “Can you explain it to me using completely different words?” If the answer is, “No,” the question then is, “Do you actually understand it or are you just repeating it by rote?”
swyx: right.
Corey: There’s—people learn in different ways and reaching them is important. [sigh].
swyx: Exactly.
Corey: Yeah. I really want to thank you for being so generous with your time. If people want to learn more about all the various things you’re up to, where’s the best place to find you?
Corey: And we will include links to both of those in the [show notes 00:33:37]. Thank you so much for your time. I really appreciate it.
swyx: Thanks so much for having me, Corey. It was a blast.
Corey: swyx, head of developer experience at Airbyte, and oh, so much more. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice or if it’s on the YouTubes thumbs up and subscribe, whereas if you’ve hated this podcast, same thing, five-star review wherever you want, hit the buttons on the YouTubes, but also leaving insulting comment that is hawking your book: Why this Episode was Terrible that you’re now selling as a legitimate subject matter expert in this space.
Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.
Announcer: This has been a HumblePod production. Stay humble.