Last week, analyst and friend to all the industry RedMonk (specifically, James Governor) wrote a blog post detailing Spotify’s use of Backstage, an internal developer portal to their cloud infrastructure and more.
Despite being complimentary enough toward The Duckbill Group in general and me personally to make my mother blush, I felt like it was advising a “wrong direction” approach.
Earlier this week, a Protocol article entitled “Why Spotify loves being locked into Google Cloud” came out, and it became increasingly clear I needed to articulate my viewpoint in more depth than a Twitter rant would lend itself to.
What is Backstage?
Backstage self-describes as an open platform for building developer portals. In other words, it provides a company-specific way of implementing controls around cloud infrastructure, and it’s also plugin-powered to support new functionality. It came out of Spotify itself, and has been open source for ages.
In effect, it’s an onramp to building self-service internal tooling that enables engineers to get ready-to-go architectures that meet compliance, regulatory, and engineering requirements for working with a cloud vendor, and in a pre-approved way.
In the ideal case, they’re not just rolling out IaaS tooling; they’re rolling out things that are higher up the stack than that.
Beyond low-level primitives such as containers, load balancers, and other “basic” components, it also starts passing out things much higher up the stack. “Here’s nginx with our default config; here’s the blessed version of our Ruby on Rails stack that powers the application; here’s the CloudFront configuration that makes that stuff work from the larger internet.”
If this sounds familiar to folks deep in the AWS weeds, it may be because a first-party offering called AWS Proton appears to be taking tentative steps in this direction. Or it might be because of Yelp’s PaaSTA, which was a similar much touted then quietly smothered tool that did much the same thing in a highly opinionated way.
Developer portals can be used to enforce standardization, to enable multi-cloud use cases (which I have opined upon previously), and to increase visibility into the various hordes of microservices that are apparently how software gets built these days.
I’m of the opinion that this is the wrong direction. As a data point: Backstage is a project I’ve never once seen deployed anywhere wild.
What’s wrong with developer portals?
There are a few issues with building in-house tooling to wrangle cloud services.
The first (and most selfish) of them is that it robs a company’s engineers of an opportunity to develop reusable skills. If you become an expert in wrangling the internal developer portal that your employer has built, that’s not likely to serve you well at basically any other company on the planet.
Admittedly, “it makes it harder for me to quit” isn’t the most compelling reason an engineer can take to their management in order to advocate against such a thing, so let’s explore a few more.
Second, developer portals inherently lag the underlying service’s capabilities. If AWS, GCP or whomever enhances a service, it’s up to the developer portal’s maintainer to update and reflect the change (assuming they’re reading Last Week in AWS so they can catch it!).
Otherwise, the portal will rely overmuch on its “plugin-driven” model, which feels like a polite way of saying “pull requests welcome”—which, in turn, is a polite way of telling people to screw off and build it themselves.
That may be okay if you’re viewing developer portals purely through the lens of IaaS provisioning of containers, VMs, storage, and load balancers. But it begins to look very odd when you’re tying it in to CloudFlare domains, PagerDuty alerting, and higher-level differentiated offerings (like last week’s Nimble Studio release, as an example).
The third failure mode pops up when you’re viewing developer portals through the lens of just IaaS: They distill down to Kubernetes without the upsides.
“Wait, Kubernetes has upsides?” sounds like something you’d expect me to rejoin with, and yes, it absolutely does. It’s broadly deployed, you can find a bunch of folks to run it for you, and just because it looks super-strange to the underlying cloud provider as a monolithic application with very odd behavior patterns doesn’t mean it’s completely without merit.
Fundamentally, I think developer portals are what every SRE has been trying to build since the dawn of time: an internal-to-their-company version of Heroku.
Backstage has more potential than most attempts thanks to being actively developed at Spotify and under the purview of the CNCF. But I have still yet to see it at any company other than Spotify.
Let’s make sure I’m not being misunderstood here
Most of you aren’t spending north of $150 million a year on cloud infrastructure or trying to wrangle 500+ engineering teams like Spotify is quoted as in James’s piece.
At their scale, idiosyncrasies like Backstage sometimes start to make sense. They’re probably not “wrong” for their environment, context, or constraints—but they are not you, and you are not them. Viewing a Backstage rollout through a lens of “this is something we should explore doing” is almost certainly the wrong move for most of the people reading this.
Tread carefully! While the concert may look glorious on the big screen, you don’t always know what’s been going on… backstage.
