Seif is the bubbly Co-founder and CTO of Axiom where he has helped build the next generation of logging, tracing, and metrics. His background is at Xamarin, and Deutche Telekom and he is the kind of deep technical nerd that geeks out on white papers about emerging technology and then goes to see what he can build.

Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. This promoted guest episode is brought to us by my friends, and soon to be yours, over at Axiom. Today I’m talking with Seif Lotfy, who’s the co-founder and CTO of Axiom. Seif, how are you?

Seif: Hey, Corey, I am very good, thank you. It’s pretty late here, but it’s worth it. I’m excited to be on this interview. How are you today?

Corey: I’m not dead yet. It’s weird, I see you at a bunch of different conferences, and I keep forgetting that you do in fact live half a world away. Is the entire company based in Europe? And where are you folks? Where do you start and where do you stop geographically? Let’s start there. We over—everyone dives right into product. No, no, no. I want to know where in the world people sit because apparently, that’s the most important thing about a company in 2023.

Seif: Unless you ask Zoom because they’re undoing whatever they did. We’re from New Zealand, all the way to San Francisco, and everything in between. So, we have people in Egypt and Nigeria, all around Europe, all around the US… and UK, if you don’t consider it Europe anymore.

Corey: Yeah, it really depends. There’s a lot of unfortunate naming that needs to get changed in the wake of that.

Seif: [laugh].

Corey: But enough about geopolitics. Let’s talk about industry politics. I’ve been a fan of Axiom for a while and I was somewhat surprised to realize how long it had been around because I only heard about you folks a couple of years back. What is it you folks do? Because I know how I think about what you’re up to, but you’ve also gone through some messaging iteration, and it is a near certainty that I am behind the times.

Seif: Well, at this point, we just define ourselves as the best home for event data. So, Axiom is the best home for event data. We try to deal with everything that is event-based, so time-series. So, we can talk metrics, logs, traces, et cetera. And right now predominantly serving engineering and security.

And we’re trying to be—or we are—the first cloud-native time-series platform to provide streaming search, reporting, and monitoring capabilities. And we’re built from the ground up, by the way. Like, we didn’t actually—we’re not using Parquet [unintelligible 00:02:36] thing. We’re completely everything from the ground up.

Corey: When I first started talking to you folks a few years back, there were two points to me that really stood out, and I know at least one of them still holds true. The first is that at the time, you were primarily talking about log data. Just send all your logs over to Axiom. The end. And that was a simple message that was simple enough that I could understand it, frankly.

Because back when I was slinging servers around and you know breaking half of them, logs were effectively how we kept track of what was going on, where. These days, it feels like everything has been repainted with a very broad brush called observability, and the takeaway from most company pitches has been, you must be smarter than you are to understand what it is that we’re up to. And in some cases, you scratch below the surface and realize it no, they have no idea what they’re talking about either and they’re really hoping you don’t call them on that.

Seif: It’s packaging.

Corey: Yeah. It is packaging and that’s important.

Seif: It’s literally packaging. If you look at it, traces and logs, these are events. There's a timestamp and just data with it. It’s a timestamp and data with it, right? Even metrics is all the way to that point.

And a good example, now everybody’s jumping on [OTel 00:03:46]. For me, OTel is nothing else, but a different structure for time series, for different types of time series, and that can be used differently, right? Or at least not used differently but you can leverage it differently.

Corey: And the other thing that you did that was interesting and is a lot, I think, more sustainable as far as [moats 00:04:04] go, rather than things that can be changed on a billboard or whatnot, is your economic position. And your pricing has changed around somewhat, but I ran a number of analyses on your cost that you were passing on to customers and my takeaway was that it was a little bit more expensive to store data for logs in Axiom than it was to store it in S3, but not by much. And it just blew away the price point of everything else focused around logs, including AWS; you’re paying 50 cents a gigabyte to ingest CloudWatch logs data over there. Other companies are charging multiples of that and Cisco recently bought Splunk for $28 billion because it was cheaper than paying their annual Splunk bill. How did you get to that price point? Is it just a matter of everyone else being greedy or have you done something different?

Seif: We looked at it from the perspective of… so there’s the three L’s of logging. I forgot the name of the person at Netflix who talked about that, but basically, it’s low costs, low latency, large scale, right? And you will never be able to fulfill all three of them. And we decided to work on low costs and large scale. And in terms of low latency, we won’t be low as others like ClickHouse, but we are low enough. Like, we’re fast enough.

The idea is to be fast enough because in most cases, I don’t want to compete on milliseconds. I think if the user can see his data in two seconds, he’s happy. Or three seconds, he’s happy. I’m not going to be, like, one to two seconds and make the cost exponentially higher because I’m one second faster than the other. And that’s, I think, that the way we approached this from day one.

And from day one, we also started utilizing the idea of existence of Open—Object Storage, we have our own compressions, our own encodings, et cetera, from day one, too, so and we still stick to that. That’s why we never converted to other existing things like Parquet. Also because we are a Schema-On-Read, which Parquet doesn’t allow you really to do. But other than that, it’s… from day one, we wanted to save costs by also making coordination free. So, ingest has to be coordination free, right, because then we don’t run a shitty Kafka, like, honestly a lot—a lot of the [logs 00:06:19] companies who running a Kafka in front of it, the Kafka tax reflects in what they—the bill that you’re paying for them.

Corey: What I found fun about your pricing model is it gets to a point that for any reasonable workload, how much to log or what to log or sample or keep everything is no longer an investment decision; it’s just go ahead and handle it. And that was originally what you wound up building out. Increasingly, it seems like you’re not just the place to send all the logs to, which to be honest, I was excited enough about that. That was replacing one of the projects I did a couple of times myself, which is building highly available, fault-tolerant, rsyslog clusters in data centers. Okay, great, you’ve gotten that unlocked, the economics are great, I don’t have to worry about that anymore.

And then you started adding interesting things on top of it, analyzing things, replaying events that happen to other players, et cetera, et cetera, it almost feels like you’re not just a storage depot, but you also can forward certain things on under a variety of different rules or guises and format them as whatever on the other side is expecting them to be. So, there’s a story about integrating with other observability vendors, for example, and only sending the stuff that’s germane and relevant to them since everyone loves to charge by ingest.

Seif: Yeah. So, we did this one thing called endpoints, the number one. Endpoints was a beginning where we said, “Let’s let people send us data using whatever API they like using, let’s say Elasticsearch, Datadog, Honeycomb, Loki, whatever, and we will just take that data and multiplex it back to them.” So, that’s how part of it started. This allows us to see, like, how—allows customers to see how we compared to others, but then we took it a bit further and now, it’s still in closed invite-only, but we have Pipelines—codenamed Pipelines—which allows you to send data to us and we will keep it as a source of truth, then we will, given specific rules, we can then ship it anywhere to a different destination, right, and this allows you just to, on the fly, send specific filter things out to, I don’t know, a different vendor or even to S3 or you could send it to Splunk. But at the same time, you can—because we have all your data, you can go back in the past, if the incident happens and replay that completely into a different product.

Corey: I would say that there’s a definite approach to observability, from the perspective of every company tends to visualize stuff a little bit differently. And one of the promises of OTel that I’m seeing that as it grows is the idea of oh, I can send different parts of what I’m seeing off to different providers. But the instrumentation story for OTel is still very much emerging. Logs are kind of eternal and the only real change we’ve seen to logs over the past decade or so has been instead of just being plain text and their positional parameters would define what was what—if it’s in this column, it’s an IP address and if it’s in this column, it’s a return code, and that just wound up being ridiculous—now you see them having schemas; they are structured in a variety of different ways. Which, okay, it’s a little harder to wind up just cat’ing a file together and piping it to grep, but there are trade-offs that make it worth it, in my experience.

This is one of those transitional products that not only is great once you get to where you’re going, from my playing with it, but also it meets you where you already are to get started because everything you’ve got is emitting logs somewhere, whether you know it or not.

Seif: Yes. And that’s why we picked up on OTel, right? Like, one of the first things, we now support… we have an OTel endpoint natively bec—or as a first-class citizen because we wanted to build this experience around OTel in general. Whether we like it or not, and there’s more reasons to like it, OTel is a standard that’s going to stay and it’s going to move us forward. I think of OTel as will have the same effect if not bigger as [unintelligible 00:10:11] back of the day, but now it just went away from metrics, just went to metrics, logs, and traces.

Traces is, for me, very interesting because I think OTel is the first one to push it in a standard way. There were several attempts to make standardized [logs 00:10:25], but I think traces was something that OTel really pushed into a proper standard that we can follow. It annoys me that everybody uses a different bits and pieces of it and adds something to it, but I think it’s also because it’s not that mature yet, so people are trying to figure out how to deliver the best experience and package it in a way that it’s actually interesting for a user.

Corey: What I have found is that there’s a lot that’s in this space that is just simply noise. Whenever I spend a protracted time period working on basically anything and I’m still confused by the way people talk about that thing, months or years later, I’m starting to get the realization that maybe I’m not the problem here. And I’m not—I don’t mean this to be insulting, but one of the things I’ve loved about you folks is I’ve always understood what you’re saying. Now, you can hear that as, “Oh, you mean we talk like simpletons?” No, it means what you’re talking about resonates with at least a subset of the people who have the problem you solve. That’s not nothing.

Seif: Yes. We’ve tried really hard because one of the things we’ve tried to do is actually bring observability to people who are not always busy or it’s not part of their day to day. So, we try to bring into [Versal 00:11:37] developers, right, with doing a Versal integration. And all of a sudden, now they have their logs, and they have a metrics, and they have some traces. So, all of a sudden, they’re doing the observability work. Or they have actual observability, for their Versal based, [unintelligible 00:11:54]-based product.

And we try to meet the people where they are, so we try to—instead of actually telling people, “You should send us data.”—I mean, that’s what they do now—we try to find, okay, what product are you using and how can we grab data from there and send it to us to make your life easier? You see that we did that with Versal, we did that with Cloudflare. AWS, we have extensions, Lambda extensions, et cetera, but we’re doing it for more things. For Netlify, it’s a one-click integration, too, and that’s what we’re trying to do to actually make the experience and the journey easier.

Corey: I want to change gears a little bit because something that we spent a fair bit of time talking about—it’s why we became friends, I would think anyway—is that we have a shared appreciation for several things. One of which, at most notable to anyone around us is whenever we hang out, we greet each other effusively and then immediately begin complaining about costs of cloud services. What is your take on the way that clouds charge for things? And I know it’s a bit of a leading question, but it’s core and foundational to how you think about Axiom, as well as how you serve customers.

Seif: They’re ripping us off. I’m sorry [laugh]. They just—the amount of money they make, like, it’s crazy. I would love to know what margins they have. That’s a big question I’ve always had. I’m like, what are the margins they have at AWS right now?

Corey: Across the board, it’s something around 30 to 40%, last time I looked at it.

Seif: That’s a lot, too.

Corey: Well, that’s also across the board of everything, to be clear. It is very clear that some services are subsidized by other services. As it should be. If you start charging me per IAM call, we’re done.

Seif: And also, I mean, the machine learning stuff. Like, they won’t be doing that much on top of it right now, right, [else nobody 00:13:32] will be using it.

Corey: But data transfer? Yeah, there’s a significant upcharge on that. But I hear you. I would moderate it a bit. I don’t think that I would say that it’s necessarily an intentional ripoff. My problem with most cloud services that they offer is not usually that they’re too expensive—though there are exceptions to that—but rather that the dimensions are unpredictable in advance. So, you run something for a while and see what it costs. From where I sit, if a customer uses your service and then at the end of usage is surprised by how much it cost them, you’ve kind of screwed up.

Seif: Look, if they can make egress free—like, you saw how Cloudflare just did the egress of R2 free? Because I am still stuck with AWS because let’s face it, for me, it is still my favorite cloud, right? Cloudflare is my next favorite because of all the features that are trying to develop and the pace they’re picking, the pace they’re trying to catch up with. But again, one of the biggest things I liked is R2, and R2 egress is free. Now, that’s interesting, right?

But I never saw anything coming back from S3 from AWS on S3 for that, like you know. I think Amazon is so comfortable because from a product perspective, they’re simple, they have the tools, et cetera. And the UI is not the flashiest one, but you know what you’re doing, right? The CLI is not the flashiest one, but you know what you’re doing. It is so cool that they don’t really need to compete with others yet.

And I think they’re still dominantly the biggest cloud out there. I think you know more than me about that, but [unintelligible 00:14:57], like, I think they are the biggest one right now in terms of data volume. Like, how many customers are using them, and even in terms of profiles of people using them, it’s very, so much. I know, like, a lot of the Microsoft Azure people who are using it, are using it because they come from enterprise that have been always Microsoft… very Microsoft friendly. And eventually, Microsoft also came in Europe in these all these different weird ways. But I feel sometimes ripped off by AWS because I see Cloudflare trying to reduce the prices and AWS just looking, like, “Yeah, you’re not a threat to us so we’ll keep our prices as they are.”

Corey: I have it on good authority from folks who know that there are reasons behind the economic structures of both of those companies based—in terms of the primary direction the traffic flows and the rest. But across the board, they’ve done such a poor job of articulating this that, frankly, I think the confusion is on them to clear up, not us.

Seif: True. True. And the reason I picked R2 and S3 to compare there and not look at Workers and Lambdas because I look at it as R2 is S3 compatible from an API perspective, right? So, they’re giving me something that I already use. Everything else I’m using, I’m using inside Amazon, so it’s in a VPC, but just the idea. Let me dream. Let me dream that S3 egress will be free at some point.

Corey: I can dream.

Seif: That’s like Christmas. It’s better than Christmas.

Corey: What I’m surprised about is how reasonable your pricing is in turn. You wind up charging on the basis of ingest, which is basically the only thing that really makes sense for how your company is structured. But it’s predictable in advance, the free tier is, what, 500 gigs a month of ingestion, and before people think, “Oh, that doesn’t sound like a lot,” I encourage you to just go back and think how much data that really is in the context of logs for any toy project. Like, “Well, our production environment spits out way more than that.” Yes, and by the word production that you just used, you probably shouldn’t be using a free trial of anything as your critical path observability tooling. Become a customer, not a user. I’m a big believer in that philosophy, personally. For all of my toy projects that are ridiculous, this is ample.

Seif: People always tend to overestimate how much logs they’re going to be sending. Like so, there’s one thing. What you said it right: people who already have something going on, they already know how much logs they’ll be sending around. But then eventually they’re sending too much, and that’s why we’re back here and they’re talking to us. Like, “We want to ttry your tool, but you know, we’ll be sending more than that.” So, if you don’t like our pricing, go find something else because I think we are the cheapest out there right now. We’re the competitive the cheapest out there right now.

Corey: If there is one that is less expensive, I’m unaware of it.

Seif: [laugh].

Corey: And I’ve been looking, let’s be clear. That’s not just me saying, “Well, nothing has skittered across my desk.” No, no, no, I pay attention to this space.

Seif: Hey, where’s—Corey, we’re friends. Loyalty.

Corey: Exactly.

Seif: If you find something, you tell me.

Corey: Oh, if I find something, I’ll tell everyone.

Seif: Nononon, you tell me first and you tell me in a nice way so I can reduce the prices on my site [laugh].

Corey: This is how we start a price was, industry-wide, and I would love to see it.

Seif: [laugh]. But there’s enough channels that we share at this point across different Slacks and messaging apps that you should be able to ping me if you find one. Also, get me the name of the CEO and the CTO while you’re at it.

Corey: And where they live. Yes, yes, of course. The dire implications will be awesome.

Seif: That was you, not me. That was your suggestion.

Corey: Exactly.

Seif: I will not—[laugh].

Corey: Before we turn into a bit of an old thud and blunder, let’s talk about something else that I’m curious about here. You’ve been working on Axiom for something like seven years now. You come from a world of databases and events and the like. Why start a company in the model of Axiom? Even back then, when I looked around, my big problem with the entire observability space could never have been described as, “You know what we need? More companies that do exactly this.” What was it that you saw that made you say, “Yeah, we’re going to start a company because that sounds easy.”

Seif: So, I’ll be very clear. Like, I’m not going to, like, sugarcoat this. We kind of got in a position where it [forced counterweighted 00:19:10]. And [laugh] by that I mean, we came from a company where we were dealing with logs. Like, we actually wrote an event crash analytics tool for a company, but then we ended up wanting to use stuff like Datadog, but we didn’t have the budget for that because Datadog was killing us.

So, we ended up hosting our own Elasticsearch. And Elasticsearch, it costs us more to maintain our Elasticsearch cluster for the logs than to actually maintain our own little infrastructure for the crash events when we were getting, like, 1 billion crashes a month at this point. So eventually, we just—that was the first burn. And then you had alert fatigue and then you had consolidating events and timestamps and whatnot. The whole thing just seemed very messy.

So, we started off after some company got sold, we started off by saying, “Okay, let’s go work on a new self-hosted version of the [unintelligible 00:20:05] where we do metrics and logs.” And then that didn’t go as well as we thought it would, but we ended up—because from day one, we were working on cloud na—because we d—we cloud ho—we were self-hosted, so we wanted to keep costs low, we were working on and making it stateless and work against object store. And this is kind of how we started. We realized, oh, our cost, we can host this and make it scale, and won’t cost us that much.

So, we did that. And that started gaining more attention. But the reason we started this was we wanted to start a self-hosted version of Datadog that is not costly, and we ended up doing a Software as a Service. I mean, you can still come and self-hosted, but you’ll have to pay money for it, like, proper money for that. But we do as a SaaS version of this and instead of trying to be a self-hosted Datadog, we are now trying to compete—or we are competing with Datadog.

Corey: Is the technology that you’ve built this on top of actually that different from everything else out there, or is this effectively what you see in a lot of places: “Oh, yeah, we’re just going to manage Elasticsearch for you because that’s annoying.” Do you have anything that distinguishes you from, I guess, the rest of the field?

Seif: Yeah. So, very just bluntly, like, I think Scuba was the first thing that started standing out, and then Honeycomb came into the scene and they start building something based on Scuba, the [unintelligible 00:21:23] principles of Scuba. Then one of the authors of actual Scuba reached out to me when I told him I’m trying to build something, and he’s gave me some ideas, and I start building that. And from day one, I said, “Okay, everything in S3. All queries have to be serverless.”

So, all the queries run on functions. There’s no real disks. It’s just all on S3 right now. And the biggest issue—achievement we got to lower our cost was to get rid of Kafka, and have—let’s say, in behind the scenes we have our own coordination-free mechanism, but the idea is not to actually have to use Kafka at all and thus reduce the costs incredibly. In terms of technology, no, we don’t use Elasticsearch.

We wrote everything from the ground up, from scratch, even the query language. Like, we have our own query language that’s based—modeled after Kusto—KQL by Microsoft—so everything we have is built from absolutely from the ground up. And no Elastic. I’m not using Elastic anymore. Elastic is a horror for me. Absolutely horror.

Corey: People love the API, but no, I’ve never met anyone who likes managing Elasticsearch or OpenSearch, or whatever we’re calling your particular flavor of it. It is a colossal pain, it is subject to significant trade-offs, regardless of how you work with it, and Amazon’s managed offering doesn’t make it better; it makes it worse in a bunch of ways.

Seif: And the green status of Elasticsearch is a myth. You’ll only see it once: the first time you start that cluster, that’s what the Elasticsearch cluster is green. After that, it’s just orange, or red. And you know what? I’m happy when it’s orange. Elasticsearch kept me up for so long. And we had actually a very interesting situation where we had Elasticsearch running on Azure, on Windows machines, and I would have server [unintelligible 00:23:10]. And I’d have to log in and every day—you remember, what’s it called—RP… RP Something. What was it called?

Corey: RDP? Remote Desktop Protocol, or something else?

Seif: Yeah, yeah. Where you have to log in, like, you actually have visual thing, and you have to go in and—

Corey: Yep.

Seif: And visually go in and say, “Please don’t restart.” Every day, I’d have to do that. Please don’t restart, please don’t restart. And also a lot of weird issues, and also at that point, Azure would decide to disconnect the pod, wanted to try to bring in a new pod, and all these weird things were happening back then. So, eventually, end up with a [unintelligible 00:23:39] decision. I’m talking 2013, ’14, so it was back in the day when Elasticsearch was very young. And so, that was just a bad start for me.

Corey: I will say that Azure is the most cost-effective cloud because their security is so clown shoes, you can just run whatever you want in someone else’s account and it’s free to you. Problem solved.

Seif: Don’t tell people how we save costs, okay?

Corey: [laugh]. I love that.

Seif: [laugh]. Don’t tell people how we do this. Like, Corey, come on [laugh], you’re exposing me here. Let me tell you one thing, though. Elasticsearch is the reason I literally use a shock collar or a shock bracelet on myself every time it went down—which was almost every day, instead of having PagerDuty, like, ring my phone.

And, you know, I’d wake up and my partner back then would wake up. I bought a Bluetooth collar off of Alibaba that would tase me every time I’d get a notification, regardless of the notification. So, some things are false alarm, but I got tased for at least two, three weeks before I gave up. Every night I’d wake up, like, to a full discharge.

Corey: I would never hook myself up to a shocker tied to outages, even if I owned a company. There are pleasant ways to wake up, unpleasant ways to wake up, and even worse. So, you’re getting shocked for some—so someone else can wind up effectively driving the future of the business. You’re, more or less, the monkey that gets shocked awake to go ahead and fix the thing that just broke.

Seif: [laugh]. Well, the fix to that was moving from Azure to AWS without telling anybody. That got us in a lot of trouble. Again, that wasn’t my company.

Corey: They didn’t notice that you did this, or it caused a lot of trouble because suddenly nothing worked where they thought it would work?

Seif: They—no, no, everything worked fine on AWS. That’s how my love story began. But they didn’t notice for, like, six months.

Corey: That’s kind of amazing.

Seif: [laugh]. That was specta—we rewrote everything from C# to Node.js and moved everything away from Elasticsearch, started using Redshift, Redis and a—you name it. We went AWS all the way and they didn’t even notice. We took the budget from another department to start filling that in.

But we cut the costs from $100,000 down to, like, 40, and then eventually down to $30,000 a month.

Corey: More than a little wild.

Seif: Oh, God, yeah. Good times, good times. Next time, just ask me to tell you the full story about this. I can’t go into details on this podcast. I’ll get in a lot—I think I’ll get in trouble. I didn’t sign anything though.

Corey: Those are the best stories. But no, I hear you. I absolutely hear you. Seif, I really want to thank you for taking the time to speak with me. If people want to learn more, where should they go?

Seif: So, axiom.co—not dot com. Dot C-O. That’s where they learn more about Axiom. And other than that, I think I have a Twitter somewhere. And if you know how to write my name, you’ll—it’s just one word and find me on Twitter.

Corey: We will put that all in the [show notes 00:26:33]. Thank you so much for taking the time to speak with me. I really appreciate it.

Seif: Dude, that was awesome. Thank you, man.

Corey: Seif Lotfy, co-founder and CTO of Axiom, who has brought this promoted guest episode our way. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that one of these days, I will get around to aggregating in some horrifying custom homebrew logging system, probably built on top of rsyslog.

Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.