I always empathized with Charlie Brown’s experience with his teacher in the “Peanuts” TV specials. Whenever she talked, the only sounds Charlie Brown heard were muffled horn noises. That’s effectively what I hear whenever I attend a webinar, sit through a formal class, or go to a conference talk that’s centered around trying to teach me the intricacies of a technology.
Everyone learns differently, and as I’ve said before, my preferred learning mechanism is shitposting. Why? The only way to get the technology to stick for me is to build something with it (and then, of course, shitpost about it).
Currently, I’m building a new production service with the intention of running it in containers orchestrated by Kubernetes — and that means it’s time for me to learn that Dread Platform.
I must have seen a good dozen presentations by now on how to get started with Kubernetes. As you might expect, they went in one of my ears and promptly out the other, since my skull is apparently a pretty poor container.
Building from scratch
I tend to consider most things through the framing that there are depths to understanding something. Linux is a great example of this. It’s become remarkably user-friendly (for certain values of “friendly”) over the years, but when things go sideways, it becomes something far beyond arcane. In my past life as a Linux systems administrator, I wanted to hit new depths of understanding by learning the fundamentals that underlie the usually functional system.
One tutorial that changed my career trajectory is “Linux From Scratch”, a lengthy book that guides you through building a Linux system entirely from source code. Would you ever do such a thing in production? I sure hope not! There are worlds better ways to deploy Linux to production, no matter what “production” might be in your particular case. There are better paths for servers, desktops, embedded systems, containers, and more. The heavy lifting has all been done for you already.
But that’s not what “Linux From Scratch” is about. Done correctly, you’re going to learn a heck of a lot over the week or month or however long you spend going through the process. It was exactly the kind of hands-on tutorial I needed for Linux. Fortunately, the same kind of thing exists for Kubernetes too.
Kubernetes the Hard Way
One of the best guides to Kubernetes I’ve yet found is authored by one of the best humans I’ve ever met: Kelsey Hightower. Several years ago, Kelsey wrote Kubernetes the Hard Way, which takes a similar approach to getting a Kubernetes cluster up and running. It doesn’t force you to compile source code yourself (although you could!), but it does walk you through building a custom Kubernetes cluster from scratch atop of six Google Cloud instances.
My only engagement with Kubernetes the Hard Way before now was to register and repoint the spoof domain KubernetesTheEasyWay.com. Go check out where it leads; I’ll wait. I’ll also repoint it to the highest bidder … but I digress.
I recently ran through Kelsey’s Kubernetes tutorial in an afternoon and decided immediately that I’d moved too quickly. I’d copied and pasted commands with only a cursory understanding of what they did rather than painstakingly and methodically researching every flag. This is my failure, not the tutorial’s. Fortunately, I know how to fix this in ways that get around my own reluctance to sit through tedium: re-implement Kubernetes the Hard Way on AWS instead of Google Cloud.
A much harder way to do Kubernetes
Just as AWS offers Elastic Kubernetes Service (EKS), Google Cloud offers Google Kubernetes Engine (GKE). It’s great — and it’s not what Kubernetes the Hard Way is about.
Instead of leveraging the higher-up-the-stack services that you should absolutely be using if you’re doing anything other than a learning exercise, it spins up six virtual machines that you can log into, and that’s it. You’re on your own from there. From there, you get to install Kubernetes from source.
What I learned is that Google Cloud’s approach to spinning up VMs is far more straightforward and less arcane than doing the same thing in EC2.
The painful part on both platforms is setting up the networking first. In the Google version, you need to create a VPC, then a subnet. Within that subnet, you create two firewall rules and add a load balancer and static IP.
In AWS Land, you do much the same, but it’s significantly more annoying. You create a VPC, then a subnet. But then you need to create an internet gateway and a route table. You attach that route table to the subnet. You then create a security group and add two rules to it.
After that’s done, you’re effectively back on the un-diverged path: You create six VMs or instances, and you continue with the tutorial.
The rest of the process doesn’t diverge all that much, though Google makes it easier to run commands on an instance from your workstation’s CLI. AWS’s approach to this is significantly wordier; I mostly bypassed it entirely.
The most maddening part of all of this was, at one point, the nodes simply would not come up in the cluster on AWS. I gave up and shut things down for the night, because I am first and foremost a Cloud Economist and dislike paying money for things that aren’t adding value. When I came back the following day, everything worked at boot, which is just the worst of all possible outcomes. “I dunno, have you tried restarting it?” is the rallying cry of the Windows world; it’s a last resort in *nix operating systems. (“A Unix admin must be root / A Windows admin must reboot,” as the nursery rhyme goes. What, your childhood wasn’t full of systems administration deep dives?!)
The fact that the cluster worked at boot is even worse, just because it means there’s Deep Magic going on that I don’t fully understand. Not understanding how something works is a dangerous sign, because you don’t want unknown things taking down production in the middle of the night. Plus, the whole point of this exercise was to understand how Kubernetes works.
What I learned by building a Kubernetes cluster from scratch (twice)
In all, I feel like I understand Kubernetes a lot better after going through this — which is a far cry from liking it, of course. That remains an unexplored area.
The process was undoubtedly more frustrating building on AWS. Kelsey works at Google Cloud, so for any doubters, creating Kubernetes the Hard Way using Google Cloud was absolutely the right decision; he’s not doing this to drum up marginal additional business for Google. Having done this twice now, Google Cloud’s simplicity is night and day from the finicky nature of AWS’s EC2 API.
My finalized working run-through now lives at KubernetesTheMuchHarderWay.com. More to come in this periodic Kubernetes exploration series…
