Ken Collins is a Staff Engineer at Custom Ink focusing on DevOps and eCommerce architecture with an emphasis on emerging opportunities. Custom Ink is approaching its 20th year in business and is entering its second phase in Cloud adoption where Ken helps an increasing growing engineering team succeed using AWS-first well-architected patterns. Ken lives near Norfolk, VA and organizes the area’s Ruby User Group.

Links Referenced: 

Speaker 1: Hello, and welcome to Screaming in the Cloud with your host, Cloud Economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of Cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey Quinn: This week’s episode is generously sponsored by Digital Ocean. I’d argue that every cloud platform biases for different things. Some bias for having nearly every feature you could possibly want as a managed service at varying degrees of complexity. Others bias for, “Hey! We heard there was money in the cloud and we’d like it if you would give us some of that!” Digital Ocean is neither. From my perspective, they bias for simplicity. 

Corey Quinn: I wanted to validate that so I polled a few friends of mine about why they were using Digital Ocean for a few things, and they pointed out a few things. They said it was very easy and clear to understand what you were doing and what it took to get up and running when you started something with Digital Ocean. That other offerings have a whole bunch of shenanigans with root access and IP addresses and effectively consulting the bones to make those things to work together. Digital Ocean makes it simpler. In 60 seconds they were able to get root access to a Linux box with an IP. That’s it. That was a direct quote except for the part where I took out a bunch of profanity about other cloud providers. 

Corey Quinn: The fact that the bill wasn’t a whodunnit murder mystery was compelling as well. It’s a fixed-price offering. You always know what you’re going to wind up paying in a given month. Best of all, you don’t have to spend 12 weeks going to cloud school to understand all their different offerings. They also include monitoring and alerting across the board and they’re not exactly small-time. Over 150,000 businesses and three-and-half million developers are using them. So give them a try. Visit do.co/screaming, and they’ll give you a free $50 credit to try it. That’s do.co/screaming. Thanks again to Digital Ocean for their support of Screaming in the Cloud. 

Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. I'm joined this week by Ken Collins, who is a Staff Engineer at Custom Ink. Ken, welcome to the show.

Ken: Thanks, Corey. It's great to be here. Long time listener, first time caller.

Corey: Well, it's funny you mention that, because it turns out that I'm relatively familiar with Custom Ink. For those who were around a year or so ago, or a little less than that, I did a fundraising t-shirt where I had a picture of US East 1 as a burning dumpster on the front of it. All proceeds went to benefit St. Jude's Research Center, which benefits kids with cancer, an issue that virtually no one who was isn't monster is on the other side of. You folks were the house that handled fulfillment and printing of those t-shirts. So I've kept a loose eye on you folks ever since, and it turns out you have some interesting stories to tell about Cloud as well. So this is a fun and interesting combination of something that happened in the real world as well as something that interacts with my own ridiculous nonsense.

Ken: Yeah, everybody wears t-shirts.

Corey: You'd be surprised. I'm known for wearing suits for a living.

Ken: Under your blazer.

Corey: So let's start at the beginning here. What is your background? How did you enter into this whole world of Cloud-native development?

Ken: First, thanks for running that campaign and picking that fundraising platform that we do. I was glad to see that you were able to do that successfully and also contribute money to a needy cause. I've been at Custom Ink for about maybe six years now, maybe seven. I think six. It's a very old company, they've been around for 20 years now, I think. It's a great story. We've been working on getting our Cloud adoption skills up over the past two or three years. That's what my role plays into, I'm the Staff Engineer, I help make technology and architecture decisions across disparate teams and functional groups. I find as a longtime Rubyist, I've been retooling my career, which I think a lot of Rubyists may end up doing as we learned to program the Cloud more.

Corey: Something that I find interesting is that the streams crossed where I tripped over you, not because of any of the t-shirt stuff, but because you wound up releasing something called Lamby. That's L-A-M-B-Y for those who are sitting and not reading the transcript. What is that exactly?

Ken: Lamby is an open-source project that is basically a thin wrapper around your Rails project to help ship your Rails application to AWS Lambda.

Corey: Gotcha. We'll get into that in a little bit, but I found that just to be a fascinating story in its own right. I've had the privilege of walking around these parts of the Custom Ink office, and you folks are bigger than I think most people would think. It's a t-shirt company, that's effectively, what, three people with a screen printing press and that's the end of it. It turns out you have multiple floors in a non-trivially sized building. How big are you folks?

Ken: I think we're about 1,700 employees now. We maybe have about 60 engineers, and those are split up mostly between our Fairfax DC office and a prod team that runs a company called Represent as well. And maybe about half a dozen web ops people. It's a nice, large size company that still got some smaller roots. But we definitely have a lot of production facilities. We've got a huge production facility in Texas, Reno, so that's where most of the employees end up sitting, is in sales, service, and production facilities.

Corey: You have been in business for just shy of 20 years at this point. I'm going to go out on a limb and guess that you weren't a Cloud-native company, given that there really was no Cloud 19 years ago. If we take a look at that, what does your architecture look like?

Ken: Let's see, so a lot of these things went back before I joined. But mainly, I think Custom Ink was one of the first large companies that had a Rails app in production. We still deal with that today, it's a big monolith. We have traditionally two monoliths, like most companies do, where you have a front end and a back end architecture. After about 19 years, we started breaking that up more. We now have hundreds of applications and services, some of them on AWS Lambda, but most of our stuff looks like EC2. About four years ago, we finished a a major shift from your co-located facilities and actually moved into AWS. But it very much looks like a lift and shift. We had a lot of employees back in the day, like, say, Seth Vargo or Nathen Harvey, who used to run a lot of our config management. They've since moved on to bigger and better things, but our Cloud infrastructure, as we moved it into there, it looks very much like a bunch of config management and Rails running on EC2.

Corey: Seth was a guest on this show in its early days, and it's always interesting to me to see just how cyclical everything is, how everyone ties back into the same thing. The world is never quite as big as we seem to think that it is. As you've wound up progressing, back in the heyday of Ruby on Rails, which was what, I want to say early to mid-aughts, and now it seems to have fallen into something of disfavor. I understand that's something of a controversial statement, but there was an awful lot of stuff that wasn't written in the last six weeks that runs powerful companies that do interesting things. These aren't just companies that are doing the Twitter for pet style nonsense, rewriting their stack every month because of something they read on Hacker News. There's validity to software that solves business problems, and a lot of that is written in Ruby. How do you square working on a stack that is, in many cases, perceived as old news because it's stable, it doesn't crash every three minutes, so therefore it's not the programming language of the future, with having to solve real world business problems? I mean, you print t-shirts, it's hard to get more tangible than that.

Ken: I think when you say stack, I hear the words Rails, and the newer stack I'm looking at is Lambda. It's easy to get distracted and buy into technologies that may solve problems that are interesting but don't deliver business value. I'd like to think that one of the things that I do at Custom Ink is always keep that thumb on the scale when we're doing things that reminds people about what we're eventually doing. Customers don't care if we're running in EC2, if we're running in Lambda, or any other thing like that. They want a website that works. They want to be able to design on a t-shirt and have it look good. You may be able to solve some interesting problems in some smaller pieces of architecture, but a lot of times, we're exploring React in different ways. Rails is an easy choice, and I think there's going to be a second coming of Rails after V6 is released. Using new technologies like Lambda, what I like to call full stack serverless, to borrow a term, and pushing bigger, larger things into AWS Lambda.

Corey: So what does that adoption curve look like for you? When you go down the path of building out something that historically has been something of a monolith, and breaking that apart into individual components? First, you have the technical piece, and that's, I guess, an interesting topic in its own right. But I think what resonates a bit more with folks who are looking at that themselves is, what does that cultural transformation start to look like?

Ken: Yeah, I think we're going to hit two sides of that. One of the benefits of Custom Ink is that we have this well understood lay lines of where our architecture and our platform needs to be. After so many years of working, it's easy for us to see where this monolith needs to be broken up. Over about the last year, we've done quite a few high profile lambdas that are just very small microservices. They do one thing very well. Some of them may compose designs on product images, others may resize clip art when you're working in the lab. But those can be solved and you don't really have to think too much about the architecture, they do their one thing well. I think the thing that we're looking at now with how to move the mindset into more Cloud native, especially around Lambda, not getting into containers yet, is just thinking how much you can actually put inside of that lambda, what it can actually do.

Many people will build, say, a node framework, like what are some of the popular ones out there? I think it was called... Oh, I can't remember. But you could put a node framework that does a web application framework, and you could shift it off, and it would be small, per se, some people would think it's small. But I almost guarantee you that if you think of Rails on Lambda, you might think that it's a big thing, and it's a big mind shift to move something that monolithic. Rails is quite small when you look at comparison to node projects.

I think the mind shift is in two things. Realizing when you're dealing with a microservice, and realizing when you're pushing into full stack serverless with Rails and Lambda. And each of those have interesting problems to solve and topics to bring up.

Corey: So why go in a Cloud-native direction? Obviously, what you've built has been working for you. It's gotten you this far, for lack of a better term. What was the advantage of moving off of that legacy architecture?

Ken: Well, I don't think we're going to know for another few years or so. But the bet is that we can better adopt Cloud-native technologies, the full offering of AWS, and ship faster. Many of our applications now, some people might find this horrific, we're deploying applications through Capistrano and that legacy manner. Shifting to Lambda is a way that you can get inside the ecosystem of AWS. From there, you can branch out and say, okay, now let me go learn about DynamoDB or Route 53 for latency based routing, or these N number of other offers that they have out there, and start to make good, isolated platform decisions, where larger amounts of our teams can make these decisions and get things out without having to be tied down to our ecosystem. Where you don't have to hook things into Capistrano for deploy. In fact, you don't even have to deploy at all. You can start bringing up more modern things like code pipelines and automatic deployments and stuff, green/blue deploys, et cetera.

Corey: Gotcha. That's not necessarily to say that there's a right or wrong path here. It's one of those areas where there is absolutely a capability story of taking advantage of new technological developments. It's just always interesting to me to see how that manifests at a company that isn't venture funded, where engineers generally don't tend to have carte blanche to do whatever they want. Seeing how companies that are, I guess, doing things here in the real world with a solid business model behind them, are thinking about these things.

Ken: We always have the new apps that we have to build or the new services that are tied to business OKRs, and we're generally left free on how to do that from an engineering side. We have good architects that help makes decisions for us on what investments we want to make in certain technologies. But the way to do that is really left up to the team. We even have some groups now working on a new container based system that might be ECS, that might be Fargate, it might be Kubernetes, who knows?

Corey: It shouldn't be Kubernetes. It shouldn't be Kubernetes, but that's okay.

Ken: Yeah, we don't want to give money to the Greek god of spending money.

Corey: Exactly. No, it's always fun to wind up seeing how this stuff tends to come to be. I'm not entirely sure there's any right or wrong direction as far as how most of that goes. But I'm curious about your personal transformation, as well. You started off as someone who was a longtime Rubyist, and now you've been transitioning into what looks at an awful lot like a Cloud-native developer. What's that been like?

Ken: Well, I think I've been putting it into three phases here. One is learning about. The other is choosing new tools and then building pipelines around those tools and those technologies, and those run concurrently and synchronistically together, and even sometimes recursively where one dovetails or other way back into the beginning.

For the learning side, that's been really hard. I've, myself, have gone out and got AWS certified from, I believe it's called the Developer Associate. Being someone who hasn't gone through academic training, I don't have a CS degree, I'm very much self taught, I was very hesitant to get certified. I feel in some ways, certification is just to learn enough to pass a test. But learning about and working in the tool has more value. But I've ended up finding that certification is a good way to learn about something. It's like why you go to conferences. You don't go to conferences to learn things, you go to learn about them. Learning about AWS, it's just going to be a full time job. That certification gets you a kickstart into that, and hopefully gets you hooked into always keeping up with AWS offerings, hopefully with things like Last Week in AWS or any other avenue. But that learning is the first part, is the key. You have to know about all these things in AWS because they're certainly asking you to architect solutions around cobbling and all of these things together. And if you don't know the right stones to reach for, then you're not going to do well.

Corey: I think you just hit on something very important. That the purpose of conferences is not to learn something new, it's to learn about them. I feel like an awful lot of folks go to the big AWS conferences and others with a mind towards attending specific technical sessions, to learn specific implementation details. At least for me, I've never found that to make a whole lot of sense. You want to wind up figuring out what those tools are, of course, and you want to see how those things wind up being used in the real world. But especially at a conference where all of the sessions wind up on the internet two days later, I'm not convinced of the utility of flying halfway across the country to do that. I feel like people don't tend to come into conferences with the right strategy in mind.

Ken: Maybe it's because when they go to the conferences, their only dialogue is that one-on-one they're getting from the conference speaker. They're not maybe going up and engaging the speaker before or after the session. Maybe they're not having enough hallway conversations throughout the conference about those topics. I tend to agree. But I also like to go and learn about how other people are doing very minute problem solving and learn if I could fit that in my tool or not. I'd say if you're going to a conference to figure out if Kubernetes is right for you, then that's probably not the right solution.

Corey: I think that a lot of people also go the other direction, view it as just an excuse to go party somewhere for a few days. And sure, you can do that. But I don't find it to be the most productive use of anyone's time when we're talking about being around some of the smartest people on the planet when it comes to a particular technology.

Ken: One of the things we've done at Custom Ink is we pick out these new technologies and how we'd like to work with them, and we have a roadmap of keystone projects that help us understand the architecture. So we're not going to do a lot of meetings and make decisions about, all right, everybody, is Fargate or Kubernetes? Hands up or hands down. We're going to maybe look at a few projects, we're going to spin some up, and we're specifically going to call them out as these are exploratory projects that we're going to learn if we like something or not. Sometimes they may end in something being set down in stone and have a ways of working, but we're really cognitive about, hey, this is something new. This is something we want to try. We don't even know if we know enough about it to make a call or not. Let's just explore that and let a few people go deep into the tea and just drop down really low, go learn all this stuff, and when they surface back up, we'll share that laterally with the org.

Corey: I think there's a lot of value in being intentional in the technologies you select rather than just what seems to have the most stars on GitHub or something similar to that. It winds up almost, instead of trying to follow the herd, you're trying to solve for specific technical problems that you have. That said, you can take that too far, and reinventing everything from first principles and going off in your own direction, that is completely at odds with the rest of the larger industry. If nothing else, it makes hiring a lot harder.

Ken: Yeah, and this is one of the things. I do like GitHub stars. If you do star the Lamby project, I would love you to death. But I think Lamby, for me as a Rails developer, and I hope this is true for other Rails developers or anybody looking to do Ruby specifically on AWS Lambda. It's this incredible thing where I can take this ecosystem that I'm familiar with, I can ship it off to AWS Lambda, and it forces me to reconcile the other things that are going on at AWS and how to be more excellent in that ecosystem. I don't think there's any better way to learn if something is right for you. I'm not even worried about containers right now in my career, but I will be in a few months.

The one thing that I think that will fit together with Rails developers, and I hope they adopt with the Lamby project, is that once you get into this ecosystem and once you have this capacity to ship Rails to Lambda, you're forced to learn about these other things. I'm having a ball every day learning more about AWS. I think last week, I recently started putting some deep thoughts into how we deploy our applications across multiple regions. Not for disaster recovery, but for availability and redundancy. I would not have thought of that if I was spinning up a single dyno in Heroku or just putting things on the EC2 and filling them over the the DevOps wall, if you will.

Corey: Well, getting to that end, how do you wind up hiring effectively into an environment where you're starting to do something that is relatively unheard of in the larger industry? By that, I do mean Lamby, in that until last reinvent, when they announced a Ruby runtime for Lambda, if you wanted to do this, you were using things like Traveling Ruby and having to shove a whole bunch of things into place. And frankly, Rails itself never really seemed like much of a particularly Lambda friendly environment when everything is built in a monolithic sense where there's startup latencies and the rest. You've made that work with Lamby. So I guess a twofold question on that. The first is, how do you wind up effectively hiring people to start wrapping their heads around this who have the experience that you need? And secondly, how do you wind up seeing this evolving over time as best practices among Cloud technologies continue to evolve?

Ken: I think you have to take two approaches. One is, you have to hire people that you think have the right ability to learn and solve problems, and just train them into your ways of working, whether that be Ruby or Node or what have you. The other is maybe start to look a little bit remote for your hires. Traditionally, Custom Ink has been very centric to hiring in the Fairfax and northern Virginia area, but we're looking at maybe changing that. I think problem solving is the most important thing a programmer can do, besides, say, communication or something else. Rails is not so hard. It's easy to train somebody into Ruby and stuff. But it is a hard problem to solve. Rubyists aren't coming out of the walls anymore. They're just sitting there delivering business value softly in the background.

Corey: For better or worse, that's probably the right answer. I think all of us need to deliver business value, and it turns out that rewriting everything every three weeks is not necessarily the best way to do that for most shops. As you decided to start down this path of converting what you had running in Rails into Lambda functions or series, you wound up having to build your own framework to do this, called Lamby. Tell us a little bit more about that please.

Ken: Sure. I'd like to think it's only several dozen lines of code, and a lot of it's inspired from AWS themselves on how to get a Sinatra app running. Right now, Lamby basically just acts as a small wrapper around your code in Lando to send the handler to a rack compatible app, which a Rails app is. It doesn't do much more than that. Everything else that the project talks about on GitHub is mainly tooling around using AWS SAM, which is the serverless application model command line interfaces, to package and build the app and deploy it out to production. If anything, it's more deploy scripts than a framework for Rails.

The one thing that I really like about the project is that it actually, other than the internals of your app and how you use, say, S3 or DynamoDB or anything like that, the project does not care about your Rails app. There's no difference between a Rails app for an EC2 instance or a Rails app for Lambda. It's basically just your Rails app and you're delivering it. I like that approach a lot, and I think it's going to afford us some certain decisions later on in the project's life cycle when we eventually write wrappers for application load balancers versus API Gateway. But for now, mainly all the gem does is, it acts like a rack up file where it mounts your application for the Lambda handler and just sends direct messages to it all day long.

Corey: How constraining do you find a Lambda environment for the way that Ruby on Rails historically thinks about applications?

Ken: The only one that I've really hit upon now is, I've not really done any database connections. I know that there's the VPC cold start issues, so as soon as you attach a VPC to your Lambda, you're going to incur a penalty of maybe some 15, 20 seconds of startup time from cold. That's true if you use a Lambda with, say, ElastiCache, like Memcached as well. So there's those things that we haven't really touched on. A lot of the Rails apps that we've been prototyping in it are isolated image processing. We're taking advantage of DynamoDB versus, say, PostgreSQL or MySQL. And I have not found that too terribly discouraging. I'm an RDBMS guy, I love databases. It's been a hard mind shift to start thinking of DynamoDB, but I'd say it's database connections.

Corey: I think that historically, the challenge I always had was that Rails tried to do everything for you under the hood. And that's great, right up until it wasn't, where it started doing things where I didn't agree with the data model, active record, and a bunch of issues with databases under the hood. Effectively getting in, unwinding some of that here be dragons. It's always felt like it was terrific for rapid prototyping, something I needed up and running quickly. But as that application grew and scaled, breaking it apart, even upgrading from one version of Rails to the next, was hell. There's really not a nice way to put that. Finding ways to modernize that onto something that resembles a serverless stack feels like it would be something almost insurmountable. I wouldn't know where to begin. And yet you didn't just begin, you did it.

Ken: Maybe it's my proficiency in Rails, but I find it incredibly easy to... To me, it was an easy decision to go, hey, if I don't have an active record model, let me use the AWS record jam, which is a thin wrapper around DynamoDB. It even encourages you to mix into things that active record does, so out of the box Rails will give you active model validations. The same patterns are requested in the AWS Record gem, which basically will bring DynamoDB to the table for you. You mix in active model validations, and pretty much you just feel at home again.

It does require a little bit of tool sharpening and bringing things to bear on it. But I think the the gems out there to get things done, if you're looking at a full stack serverless, which I definitely recommend people do, there's definitely uses for small applications, where a Rails app just does one thing through a couple gems. But the problems are easily solved. If you're doing things like image processing or if you're not moving your oracle database over to serverless, then I think it'd be just fine.

Corey: So what's next for the project? What is it that's keeping it back from the next step on its path to greatness?

Ken: I'd like to do a few regional comp talks about it where we do some workshops. We've done a few internally for Custom Ink where we walk engineers through creating their own Rails app. Very much like a 201 class where you get a little bit past hello world. I think for me, it's basically going to be moving to, what it it, the Application Load Balancers for the event, versus API Gateway. I read a recent blog article that said that was a lot of cost overhead. Not a lot. Everything at Lambda is going to be cheap, but that you could probably get a little bit better performance and a little bit more cost squeezed out if you switch to the Application Load Balancer.

Other than that, it's really just building a community around it and finding out stories and people talking about how they solve certain problems. People can pull more resources together and understand that, hey, if I had an application that did one small thing, what's the pattern that you might choose from, or, my Rails application might be using more of the the Java script stuff. What's the best way to build the assets for that and ship them off to a CDN, et cetera.

Corey: How do you see the future of the project evolving now that Ruby is a first class runtime in the world of Lambda?

Ken: I think it's going to grow. I think there's going to be a lot of people, a lot of Rails programmers, that are, to date, shifting their careers, doing things like migrating Heroku instances to AWS, that are doing this type of consultancy that need to learn more about AWS. With tools like Lamby out there, they're going to be able to make some really good decisions for them on how certain Rails applications go into AWS Lambda and how it uses more of their ecosystem.

I'm excited to learn more about people's stories about what they can share on the the Lamby website. We can contribute more blog articles and also technical content for the product website on how other people can take those learnings. And that's really where I'd like to see it go. Just more success stories and more adoption and more documentation.

Corey: If people want to learn more, where can they find out about Lamby in general and you specifically?

Ken: Lamby is located at Lamby, L-A-M-B-Y, dot Custom Ink tech dot com. I'm pretty much metaskills on every other thing, whether that be Twitter or Xbox or LinkedIn or what have you.

Corey: Ken, thank you so much for taking the time to speak with me today. I appreciate it.

Ken: Thanks, Corey. I've had a good time.

Corey: Ken Collins, Staff Engineer at Custom Ink. I'm Corey Quinn. This is Screaming in the Cloud.

Speaker 1: This has been this week's episode of Screaming in the Cloud. You can also find more Corey at screaminginthecloud.com, or wherever fine snark is sold.

Speaker 2: This has been a HumblePod production. Stay humble.