Dr. Galen Hunt founded and leads the Microsoft team responsible for Azure Sphere. The mission of his team is to ensure that every IoT device on the planet is secure and trustworthy. Previously, Dr. Hunt lead the Operating Systems Group at Microsoft Research and pioneered technologies ranging from confidential cloud computing to light-weight container virtualization, type-safe operating systems, and video streaming. Dr. Hunt was a member of Microsoft's founding cloud computing team and helped build Microsoft's first cloud operating system. Dr. Hunt holds 98 U.S. patents, a B.S. degree in Physics from the University of Utah, and Ph.D. and M.S. degrees in Computer Science from the University of Rochester.

Announcer: Hello and welcome to Screaming In The Cloud, with your host, Cloud economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of Cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming In The Cloud.

Corey Quinn: This week’s episode is generously sponsored by Digital Ocean. I’d argue that every cloud platform biases for different things. Some bias for having nearly every feature you could possibly want as a managed service at varying degrees of complexity. Others bias for, “Hey! We heard there was money in the cloud and we’d like it if you would give us some of that!” Digital Ocean is neither. From my perspective, they bias for simplicity. 

Corey Quinn: I wanted to validate that so I polled a few friends of mine about why they were using Digital Ocean for a few things, and they pointed out a few things. They said it was very easy and clear to understand what you were doing and what it took to get up and running when you started something with Digital Ocean. That other offerings have a whole bunch of shenanigans with root access and IP addresses and effectively consulting the bones to make those things to work together. Digital Ocean makes it simpler. In 60 seconds they were able to get root access to a Linux box with an IP. That’s it. That was a direct quote except for the part where I took out a bunch of profanity about other cloud providers. 

Corey Quinn: The fact that the bill wasn’t a whodunnit murder mystery was compelling as well. It’s a fixed-price offering. You always know what you’re going to wind up paying in a given month. Best of all, you don’t have to spend 12 weeks going to cloud school to understand all their different offerings. They also include monitoring and alerting across the board and they’re not exactly small-time. Over 150,000 businesses and three-and-half million developers are using them. So give them a try. Visit do.co/screaming, and they’ll give you a free $50 credit to try it. That’s do.co/screaming. Thanks again to Digital Ocean for their support of Screaming in the Cloud. 

Corey Quinn: Welcome to Screaming In The Cloud. I'm Corey Quinn. I'm joined today by Galen Hunt, a distinguished engineer and the managing director of Azure Sphere. Welcome to the show.

Galen Hunt: Thank you, Corey. It's great to be here.

Corey Quinn: So Azure Sphere is a lot of things and I'd like you to tell us what that is, but the most compelling part that I saw was in a single sentence on the website: "Our goal is to make IOT safe for society." Through the lens of that very inspiring statement, what is Azure Sphere?

Galen Hunt: What is Azure Sphere? So Azure Sphere is an end-to-end solution for addressing the security needs of IOT devices. Okay. It consists of three pieces. There are Azure Sphere compatible chips that are built by our silicon partners and incorporate intellectual property from Microsoft into them. There's an operating system that runs on those chips and then there's a Cloud service that works with the chips and the operating system to keep the devices based on them secure. And that's fundamentally what we're trying to do. We're trying to make sure that any device manufacturer can build a device based on Azure Sphere and ship it out and know that for the lifetime of that device, it is going to remain secured.

Corey Quinn: And that's, I guess, from a very naive perspective, not having much of a background in IOT myself, but I think of the internet of things, this entire world of devices that are living in my house. I go out and I buy a scale or something and it talks to the internet. In other words, I don't know, maybe it posts on Twitter to shame me whenever I gain weight. And that's awesome and I keep that thing for years on end.

So instead of focusing on a real device... For example, my Twitter for Pets company, my side project, decides to get into the IOT space. We're going to build combination toaster-refrigerators and it turns out that the product does not see a lot of market success because physics. And after selling a whopping three of these, we pivot. We post, "Our amazing journey has come to an end" on Medium and raise another round, because that's apparently how failure works today.

And we still have those three that are out there, and at that point, the Cloud services that we were paying for have been turned off. There's nothing for the other end to talk to, and assuming that there isn't a failure mode where we have just bricked the expensive thing that people have bought from us, you now have this thing sitting there, unpatched, in perpetuity...

Galen Hunt: Sitting on the internet.

Corey Quinn: Exactly. And now, one day, someone, maybe a state actor... Oh, sorry, in InfoSec, we call them nation state, which irritates a lot of people, the same way that on-premise instead of on-premises does. So we're just going to call this episode On-Premise Nation States, just to irritate everyone.

But once you look at that and it starts attacking things, there's a responsibility issue and there's a how do you even identify that that is a thing that your device is doing? If you think about that, it feels like an incredibly large-scale problem with no easy answers.

Galen Hunt: It's a huge problem, because in the old days, if you made some new device like this toaster-refrigerator combination... Gosh, I'd really love to have one, okay?

Corey Quinn: Oh, yeah. Saves so much space in the kitchen.

Galen Hunt: Yes, exactly.

Corey Quinn: And the unexplained fires have not been proven in court.

Galen Hunt: In the old days, you could build one of those and you could sell it to your customers and basically, your engineering job, your hard work, was done the day you shipped it, because you never saw that thing again. The problem is, when it's an IOT device, your hard work begins the day you ship it. It's the day that it goes into a customer's home or into an office or another environment and it gets connected to the internet. That's the day the Internet -- and the hackers come. And from then on, until that thing is disconnected permanently from the internet at the end of its life, it is at risk, from a security perspective.

And this is the fundamental thing. You know, IOT is super powerful, because it creates a connection between that device and the manufacturer or the customer and the manufacturer. It creates a connection, but every Internet connection's a two-way street. Right? And what that means, "Hello, hackers." So and what we're trying to do with Azure Sphere is we recognize it... This company that builds this refrigerator-toaster, they know how to build a refrigerator-toaster, let's hope. Knock on wood.

Corey Quinn: In theory, yes.

Galen Hunt: In theory, okay, but in practice, almost none of them know anything about internet security and it is a hard place to be. The internet's a very scary place. I have a former colleague who's a professor at Harvard, James Mickens, and he likes to say, "The internet is this cauldron of evil." Nation states, professional hackers, whatever you want to call it, and what we try to do is say, "Well, how can we package up the experience that Microsoft has?" Because, by the way, we've been doing this for a really, really long time. I've been at Microsoft 22 years. My entire career has been spent working on internet security, one form or another, trying to keep the hackers out.

And we said, "Okay, is there some way that we could take all of this expertise and experience that Microsoft has and package it up so that we could give it to device manufactures and then actually keep giving it to them so that we could help them keep building secure devices?" And that's what we've fundamentally created.

Corey Quinn: It seems to me, looking through... The way that I've historically seen Cloud services tend to manifest is... There's an economic challenge here where people are going to pay for a ridiculous IOT product like a toaster-fridge or a scale that fat shames you or whatever it is that you wind up buying but they're generally not going to want to pay a subscription for that because it doesn't tend to comport with our mental model of how services work. So people will go and they'll spend money, sometimes a lot of money on something like that, but they're not necessarily going to want to sign up for a recurring subscription model.

So the challenge then becomes you just need to be able to provide secure Cloud services for things that, in all likelihood, are going to be talking to the internet way longer than anyone thinks they will. It's, "Oh, I'll just get that scale for two or three years" and mine is coming up on 10 years old. I'm sure it's an attack vector for something now but I'm irresponsible. There's an economic story where if you have to pay at a monthly basis or per API call that thing makes to a Cloud provider, that, at some point, you are now spending more on the long-tail Cloud service than the thing made you in profit and you are losing money on every sale. How does that wind up tying into how customers are approaching IOT today from a security perspective?

Galen Hunt: Well, so one of the things we looked at is how do we make it... You want to make the security decision be a one time decision. Do I want a secure device or not? Okay. Hopefully, the answer is... The answer should always be, "Yes." Particularly, you don't want people asking on a month to month basis. "Do I want security this month or am I feeling lucky?" And in fact, the business model we came up with, the Azure Sphere, is it's a one time transaction. When the manufacturer decides to buy an Azure Sphere chip, they get with it from their distributor the chip and the license to our operating system and the license to our security service. And that includes the ongoing security work for both through a period of the expected lifetime of that device.

So let's say, a 10 year period, and it's a one time so nobody's paying money... 10 years in, seven years in, you're not paying more money to keep that device secure. And the other thing that we did with the Azure Sphere is we've actually separated out... If you typically looked at an embedded device, the device manufacturer takes an RTOS and they take their code and they put it together and they're responsible for everything. And what we've done is we've actually broken up the way the code is factored so that we can keep updating the operating system.

So there's new security vulnerabilities and new security threats and attacks come out, we can update the operating system. In fact, we will. We will update the operating system and the security features on the devices out in the field. So let's say, to use your refrigerator-toaster example... Let's say they go out of business or they say, "We're not going to support this thing", if it is based on Azure Sphere, Microsoft is going to keep supporting that and we're going to keep updating it and addressing security vulnerabilities until that thing is done.

Corey Quinn: To be honest, it wasn't even until this conversation where we look back at things like HeartBleed. When that came out, I was doing a fair bit of consulting with a number of different customers and talking to them, making sure they were patched, making sure my own stuff was patched. But not until now, did it occur to me, "You know, I wonder if that stupid scale of mine at home wound up getting patched or not." Almost certainly not because the company got acquired twice and who even knows at this point. It's basically a hazard to all around it in an emotional way and a physical way now but it's... This is not something anyone, even people who think about this stuff in a security context, are generally going to think of intuitively.

Galen Hunt: Yeah. Because you want to just buy that thing and install it and forget that you have to worry about it, right?

Corey Quinn: Yeah.

Galen Hunt: And that's exactly what we're trying to address here.

Corey Quinn: To be clear, there are remarkably few companies that could make a statement of, "If your company goes out of business, that's fine. We're going to continue to maintain security updates for the infrastructure for this IOT stuff." But if anyone's earned that, it's Microsoft at this point. The long tail legacy support for fascinating and varied use cases is borderline legendary. And for anyone who’s had to write code around some of this, it's kind of obnoxious to have to still work around, "Well, people are technically still using Internet Explorer." They announced that in the keynote at Build where the next version of Edge now has built in Enterprise support and two or three people in the audience just lost it, cheering.

And you look around, like, "Oh, those are the sad people." Because we lived that life. We know what that pain looks like. But the idea of being able to have a perspective of looking long term back at... This is important and it needs to be able to support this from a business continuity perspective. It's powerful and I think Microsoft gets that, arguably better than anyone else that...

Galen Hunt: We have been doing it a really long time. Like I said, I've been at Microsoft 22 years. And I remember when the Slammer and Blaster viruses came out and us having to figure out... I was on the task force at Microsoft to figure out, "Okay, how are we going to address these class of things and make sure that they don't happen again?" And all the skills that have to... And if you think about building a highly secure device, and that's the term I use... So highly secured is something I can just really depend on the fact that it's secured. There's a lot of skills that go into that. There's a lot of engineering up front that you have to do to get all the pieces together right so that you don't have... If you use a really bad random number generator so that even if you have this amazing crypto, well, it doesn't matter because you've thrown it away, the random number generator.

So it's a bunch of engineering and then there's this ongoing work that you have to do of, every time some new vulnerability like... What's the one you used?

Corey Quinn: HeartBleed.

Galen Hunt: HeartBleed. Okay. Like HeartBleed or the crack vulnerability in the WPA2, the Wi-Fi protocols, okay.

Corey Quinn: Oh, that brings me back.

Galen Hunt: Yeah, a year ago, or when these new things come out, somebody's got to look at that and say, "Does this apply to this advice and what are the changes we have to make?" So you've got to have an ongoing security expertise. And then you figure out a patch. Okay, you can say, "Oh, well here's how we're going to mitigate that. We're going to fix the patch." And then you've got to have this expertise of, "How do I actually roll it out to every fat shaming scale on the planet and make sure that everybody's device is actually updated? Do I roll trucks? Do I send emails or the devices automatically update themselves?"

So you have to have this operations logistic expertise on top of this ongoing security analysis expertise on top of the engineering expertise you have to have. And what we're basically trying to do is take all of that and offload that to Microsoft.

Corey Quinn: Well, where are the bounds of Azure Sphere in that sense, where if I build a device and I put this solution into it, it obviously controls the firmware. It winds up controlling the version of RTOS patching. Does it control, for example, the Wi-Fi aspect of it? Is that in bounds for this, assuming there's another Wi-Fi WPA2?

Galen Hunt: So it's pretty extensive because we own the entire operating system. And so for example, with Wi-Fi, if there was a crack vulnerability. Let's say someone's coming up with a new vulnerability... Actually, let's talk about the crack vulnerability. What had happened, it was a little over a year ago... Almost a year and a half now.

Corey Quinn: Why does it feel so much longer ago?

Galen Hunt: There's a lot of IOT security news out there, right? It just keeps coming. We had a fix, a verified fix, for that available within 24 hours of the vulnerability because one of the things we've also learned how to do very good at Microsoft is figure out what is the fix that we have to do for a particular vulnerability and how do we test our systems so that we actually know that the fix is correct, et. cetera. And then we had the deployment technology to build and deploy that out within hours to billions of devices.

Corey Quinn: And none of the customers who manufactured these things even had to think about this? It was simply done for them.

Galen Hunt: So if you were using Azure... If you had an Azure Sphere based device. Say you're a manufacturer and you build an Azure Sphere based device and you get woken up with this headline of crack vulnerability. If you're using Azure Sphere, what's your responsibility? Go back and go to bed. We got your back. It's our problem. And that's a key thing. We own the entire operating system stack on the device. On, not just the bits that we give you as a manufacturer, but literally the bits on the device. So that we're going to fix them out on the devices on the field and we also own the security services providing so that all the bandwidth for the updates and we do the updates both for the OS... We also provide an update channel for what we call the application. The OEMs code.

So the device manufacturer... Let's say the toaster-refrigerator, they come up with a new feature... I don't know. It's a thing that's going to shoot the ice cubes out into the toaster because everybody wants toasted ice, right? And it turns out that's just a software update. Well, they can... They can and they want to get that software update out to all their customers because who doesn't want toasted ice? Well, they create the new update and they turn it over the Azure Sphere security service and say, "Hey, deploy this out to all our customers", and we do the heavy lifting for that as well.

Corey Quinn: One thing that I've always found aligned with the security mentality is the way that I tend to approach Cloud economics. Specifically, in that, no one sets out to build a product or service for the least possible amount of money so waste creeps in, in the same way that almost no one sets out to build a product from day one to be the most secure thing in the world. They want to build a thing that ideally gains market traction and people buy it and security as the number one bullet point doesn't move almost any of these things unless it is a security device itself. So there's something to be said for using this service... And effectively at that point, you are taking the entire security issue and more or less outsourcing the work if not the responsibility to a provider that it just works.

And everything handles itself. That's compelling. That's the sort of story that I think is going to win the security wars, for lack of a better term. And I'm not talking about competitors' security wars. I'm talking about the ongoing battle against the cauldron of evil. It's how you wind up getting somewhere that you don't have to go out of your way to do the right thing. You've built a guardrail path where doing the right thing is easy, straightforward, and is, in some ways, much easier than doing the wrong thing.

Galen Hunt: Well, that was the objective. I launched this thing five years ago. Got it started, building the initial prototypes and everything. And that was the objective, was, "How do we make it so that security is so simple that everybody uses it?" Okay. And it was really critical to do that because, as you said, people don't immediately recognize, "Oh, why do I need security?" or "How much do I... " It's like, "Oh, I just want to do just enough security." Well, the problem is that the internet's a really, really dangerous place.

Corey Quinn: And it's not getting less so.

Galen Hunt: And it's not getting less so and just because you're new to internet security doesn't mean that the hackers are new to internet security. And so there's a pretty high bar of what it takes to build a device, even today, even if you just build for what are the known security issues right now. It's a really high bar and it really, really requires a lot of expertise and so we're trying to address that. The other thing I'll mention is you talk... People tend to say, "Oh, nobody's going to be willing to pay for security." We believe security is the differentiating value prop of IOT. Okay? Because when it really comes down to it, nobody wants their refrigerator-toaster that creates botulism or that blows up their house and the line between an IOT device and a dangerous device is really, really thin without security.

Corey Quinn: Oh, absolutely. But putting on the front of the box, "Won't burn your house down", in big letters is one of those, "huh!" That's selling a breakfast cereal, it's like, "Contains no rat poison." Well, it wouldn't have occurred to me to ask that question until you bring it out there. That's the marketing problem.

Galen Hunt: Yeah, well one of the things we have found... We've done a lot of looking at this. One of the things we did is we did a security survey with consumers across the United States and Europe. We interviewed somewhere about 3,000 individuals. We actually went and had face to face meetings and talked with them. And what our data showed is that most people, the vast majority of people... If they knew that a device was secure, they would buy a secure device over an insecure device, and they would pay more money for it.

Corey Quinn: From that perspective, is security framed as won't attack the underlying DNS infrastructure of the internet or is it contextualized more as privacy? I make a joke about a fat shaming scale but having it leak your personal information is, I think, a lot more resonant with people than some ephemeral, "Well, one day, the internet's going to be slow and broken" and my failure mode is, "I'm going to have to go outside for a little while."

Galen Hunt: Yeah. You kind of have to make it personal and one way I try not to scare people but if you just kind of step back and think of it like... So one of the things we're trying to do with Azure Sphere is make it even approachable for micro-controllers, the very cheapest class of computers. And to make it really personal, if you go into your home, it's a micro-controller that is keeping your furnace from creating carbon monoxide and poisoning your family. It is a micro-controller that is keeping your gas stove from exploding. It is a micro-controller that's keeping your dish washer and your washing machine from flooding your house. And today, those things are safe because they're not on the internet at all. But when they come on the internet, they have really got to be secure.

Corey Quinn: It's... We've talked a lot about ridiculous IOT approaches. Do you have an example of a customer or two that's doing it right? As much fun as it is to sit here and talk about terrible ideas that should never have been built, I'm more interested in a uplifting story. Who's using Azure Sphere today and making the world a more safe... making society a safer place for IOT?

Galen Hunt: We have a company in Europe called Eon that is doing home energy management systems and they've got car chargers and batteries in homes and solar power systems and you think about... There's a lot of electricity running in those things. They could actually... Those things could be dangerous but then Eon said, "No, want to make sure that these are trustworthy systems and are metering right and everything else." And so they've chosen to use Azure Sphere.

Corey Quinn: It's fascinating to see just the different verticals that these things tend to get used within. You talk about in almost the same paragraph, you talk about a retail establishment that sells coffee and a solar power company. And we're starting to see that the entire world is, in fact, becoming more connected. And it's... There are a lot of people who hear something like that, and I confess I'm generally one of them, who thinks, "Is this all good? Is this going to be something that leads to a better society or does it lead to a story where suddenly every bit of information about me is for sale on the dark net to the highest bidder?"

And that has been an area of growing concern. At this point, I've started thinking, "Oh, well, how many devices do I have on my internet connection at home?" And I realize, as I just think mentally, last time I looked at that to update something in my mobile app for the Wi-Fi, there's over 40 devices connected. There are three humans who live there. That seems a little excessive but everything starts to wind up being connected and this is going to be an area that is absolutely not going to go away anytime soon and it's getting safer.

Galen Hunt: Until Azure Sphere. We're going to make it safer. It is not going to go away and it's just going to keep coming. Security is necessary for privacy.

Corey Quinn: Yes.

Galen Hunt: Okay. Because if your devices aren't secure, it's like, OK. Well, if they're secure, then there's the question of, "What's my relationship with the manufacturer then? What's their privacy policy? Et. cetera" and things like that. But if it's not secure, hey, that stuff's open to any hacker that wants to come in. We've seen... We've seen headlines, IOT security headlines, fridges sending spam and baby monitors being used to spy on families or project messages into families. And so you really, really want these things to be secure.

Corey Quinn: As compelling as this sounds, it doesn't work, generally speaking, to think of security in a context of absolutes, like the idea of M&M security's always a challenge. You wind up breaking through the perimeter and now you have everything there. How does Azure Sphere tend to address that particular threat model, if at all?

Galen Hunt: Okay, so when we think about security... We actually published a paper that I co-authored about two years ago called The Seven Properties Of Highly Secured Devices, particularly to help explain to people how they should think about security because as we'd go out and talk to device manufacturers early on but a couple of years ago, we were just getting to kind of the prototype proof of concept stage. They would... Sometimes we'd have this conversation, they'd say, "We have some security, is it good enough?" So we'd try to help them frame that. And one of the topics we talk about in that paper is defense and depth and this is, "Do you have multiple layers of defense so that when something goes wrong, if somebody is able to circumvent one layer of your security, you've got another."

Give you a just kind of physical example. You think about, if you go into a fairly secure building, like, say, a courthouse or something like... A Microsoft office. Some of them or things. Or a bank. You'll go in and there will be locks on the door and there will be a guard and there might be a metal detector and there's video cameras and there's a safe. Okay. And that's because someone might be able to figure out how to break the lock on the door but then you've got a safe and then... Or you've got cameras so that you can figure out who it was. And you've got all these different layers and that's because, well, if you have only one layer of defense, you have a single point of failure and that means if something goes wrong either intentionally or accidentally in that piece, you don't have any security at all.

And the thing we found, most IOT devices that are out there today have really been built with... It's the M&M, hard on the outside, soft on the inside, security model instead of this defense and depth. And what we've done with Azure Sphere is we have multiple layers of defense and depth so within the hardware itself, we have three layers of defense. In the truest way, in the operating system itself, there are four layers of defense and depth in the operating system. And that's so that if hackers are able to find a vulnerability, get into one piece, they can't just keep going and be able to... In fact, we can actually... We, detect that they've gotten into a device and we can kick them out and renew the security on that device.

Corey Quinn: Fascinating. That's one of those areas that, I guess, makes a lot more sense once you get into the space but coming from an outside perspective, it would never have occurred to me to start thinking at that layer of complexity. It's a war that's probably never going to be won but you can absolutely embrace the stakes.

Galen Hunt: Yeah. And it's the... It's what's required out on the internet today.

Corey Quinn: I think we'll want to hear more about your thoughts on this. Where can I find out?

Galen Hunt: So I'm on Twitter. Galen_Hunt on Twitter. We also... They can go to the Azure Sphere website and find out more.

Corey Quinn: Thank you so much for taking the team to speak with me today. I appreciate it.

Galen Hunt: Thank you, Corey. It was a great conversation.

Corey Quinn: Galen Hunt, distinguished engineer and managing director at Azure Sphere. I'm Corey Quinn. This is Screaming In The Cloud.

Announcer: This has been this week's episode of Screaming In The Cloud. You can also find more Corey at screaminginthecloud.com or wherever fine snark is sold.

Announcer: This has been a HumblePod production. Stay humble.