Disclosing Vulnerabilities in the Cloud with Ryan Nolette

• AWS VDP on HackerOne: hackerone.com/aws_vdp
• AWS VDP inbox: [email protected]
• LinkedIn: www.linkedin.com/in/cloudy-with-a-chance-of-security
• AWS Vulnerability Reporting site: https://aws.amazon.com/security/vulnerability-reporting/
• Give your feedback on the recently expanded VDP program: https://pulse.aws/survey/MOOFGRLM

Transcript

Ryan: As somebody who was a researcher, I built it for something that would make my life better and makes me happier with the experience. I need more feedback from active researchers and community of what you want to see in the program.

Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Here today, we're going to talk about something near and dear to my heart that I've tried desperately to get away from and somehow can't quite seem to. Ryan Nolette is a Senior Security Engineer at AWS Outreach. Ryan, thank you for joining me.

Ryan: Thank you very much for having me.

I'm super excited to be here and chat with you today.

Sponsor: Backblaze B2 Cloud Storage helps you scale applications and deliver services globally. Egress is free up to 3X of data stored, and completely free between S3-compatible Backblaze and leading CDN and compute providers like Fastly, Cloudflare, Vultr, and CoreWeave. Visit backblaze.com to learn more. Backblaze—cloud storage built better.

Corey: So AWS having an outreach group in its security team on some level just seems like a bit of an aberration compared to most interactions I've had with large company security groups. The only type of outreach they tend to do is with cease and desist or preferably a baseball bat.

What does AWS outreach look like?

Ryan: Well, second of all, our bats are very high quality and they are Amazon Basics brand.

Corey: Oh, absolutely. Amazon essentials has made the shirt that I'm wearing today. I hear you. Everything is an Amazon in the front of it. It's how it works.

Ryan: Well, excellent, because we'll actually be talking about my shirt today as well as part of the talk. Because one of the topics that, uh, that I really want to go into today is the recent expansion of the AWS Vulnerability Disclosure Program. So, VDP is what it's commonly called. It's one of the methods of starting the coordinated vulnerability disclosure process, which is incredibly important in the software world, or really any developer. How it kind of interacts today with this wonderful security world that you're talking about.

is it gives a point of engagement between the large companies such as, you know, AWS and the security community at large, whether it's an individual, a company, even some governments and nation states.

Corey: Got it. So a Vulnerability Disclosure Program to my somewhat limited understanding, I usually find out about these things only after I've asked a, "That's funny question on Twitter," and then, "Oh no, what have I just unleashed," is the way of responsibly and collaboratively reporting security issues or suspected security issues to a vendor.

Is that directionally correct or is there nuance here that I'm failing to capture?

Ryan: There's a little bit of nuance. One of the things that I picked out from the question is the term "responsible disclosure." So that has become more of an industry term now, but it's actually a branded term. So it's, it's a specific terminology that's meant to describe the CVD process.

And CVD, once again, is the Coordinated Vulnerability Disclosure process.

Corey: And I'm using it in a more colloquial sense, as opposed to, instead of tweeting about it, I would go ahead and send in an email to a, or now, apparently, go on HackerOne, which you folks are recently making a significant expansion with.

Ryan: Exactly. You know, the key difference is that term coordination, and really what comes into that is scale. Responsible disclosure is usually more between a single entity and a single entity. Coordinated vulnerability disclosure could work at any scale. And that is something that AWS very much requires as part of our VDP program, is the ability to respond at scale.

We are incredibly large, and a lot of the technologies and issues that we deal with are very different once you get to a scale of our size. So, kind of talking about how we take in these new reports, how we end up actually doing the remediation and mitigation process, those are all super important things to talk about at scale and how it's different.

Corey: Yeah, I've, I tend to be much earlier in the process because again, all the stuff that I tend to find is of the form, instead of the form of, "Haha, I found a security issue," it's a, "that's funny." And one of the things that I've learned from a lot of folks over the years is even when you think you've found a security issue, I always find it best to say, "It appears," and "I think," as opposed to speaking declaratively, because the first time you tell someone, "You have a problem here," and it turns out that no, no, you're the one with a problem instead, no one will ever listen to you again. Because you're just someone who's going to sound off and sound confident when you really shouldn't be.

That was a lesson I think some people could really stand to benefit from, but my experience historically, when I found things that of the form of "That's interesting," is I would email into the AWS Security Questions email address, and I would get a human being responding to it, and it would sort of take off from there.

Is this replacing that? Expanding that? How is this manifesting?

Ryan: Yeah, it's expanding on it. So AWS has a VDP program, the Vulnerability Disclosure Program. It has had one in a very formalized sense for at least the last five or six years, and ten-plus years in a less formalized version. All we're doing with this HackerOne onboarding is really giving another input source for our VDP program.

So whether you're submitting through the AWS Security Inbox, which is [email protected], or if you're going to HackerOne, you can either search for "AWS" or you can go directly to it. I believe it's hackerone.com/aws_vdp. Either of those places, you can submit the same type of information.

We wanted feature parity, and really what we're trying to do is make it easier and quicker for the community to do submissions to AWS. And there's some additional benefits. So, the traditional way of having an inbox on the internet is it's the front door to the internet, so there's a lot of information that goes into it and a lot of filtering that has to occur.

With a more traditional portal or console experience, we're able to give a very structured, safe, consistent process and experience for researchers to submit. We're able to more clearly communicate to them what we believe should be in reports by including certain required fields, and we're able to communicate backwards to them what types of information is and is not in scope.

While it's an expansion of the existing program, it brings along an additional set of benefits. I think probably for the community, what they care about most is the user experience of it. Currently, right now, when you use an email to do any kind of communications, you're restricted by a response from either side.

There isn't a way to check current status unless you ask or somebody sends you an update. There isn't a way to easily collaborate with multiple people unless you have a CC line going and then that expands into all kinds of thread problems with people responding out of sequence in the emails.

Corey: Top-posting versus bottom-posting. I remember the mailing list and Usenet wars over this.

Ryan: Oh yes, those were the days. Give me that command line back so I can just grep it. But one of the nice experiences about, you know, having that portal console experience is that it gives you a lot of information. You know, all of your submissions are linked to a single user account, so you can check your entire history.

You can go into a single report and see all the comms between two people or 20 people if be. You're able to search and filter your own reports based on what you submitted on. Do you want to be interested in how many reports you sent about service 1, 2, 3? You can filter on that field. So it gives a lot better experience for people to pull information about what they're already working on.

Additionally, it allowed us to adopt a new safe harbor. So we've traditionally used a modified version of Disclose.io's Safe Harbor. And now we're using Gold Standard Safe Harbor, which was written by, I believe, one of the execs at HackerOne, but he's been very prominent in the security realm for a while.

And this is what he wrote and has been pretty popular adoption across the community. A lot of people have asked about it and whether or not we could adopt something similar to it, so as part of this launch, we adopted it across the entire VDP. So whether you submit security inbox or through HackerOne, you're under the same Safe Harbor, the ability to publicly and transparently describe the scope.

So we have two scopes, right? The scope of what can be submitted into the VDP, and then a tighter scope of what qualifies for a security advisory like a CVE or a GHSA. And a lot of times when you're dealing with companies, it's a black box really. You know, what can I submit? Does this count? Can I get a CVE for this? Does it even qualify? All those types of information, how to score the thing that you're submitting, all those details, we are posting. So there's no longer a question. You can control+F on that policy and find that information. So we've kind of built it off of what are the common questions that we get, and we're rolling that back in to answer.

That all kind of leans toward increased transparency. Alright, I, as a researcher myself, I've done a lot of submissions to different programs, the transparency aspect was always a point of frustration for me, because I want to know what's going on to see if I need to do work to help and provide guidance, or if I'm way off base and I should move on, and this increased transparency really allows me as a researcher to be more informed about the overall engagement. And finally, one of the really cool things about this is the approved statement. So traditionally, how do you show recognition for something that was submitted into a VDP program?

Corey: In many cases, the answer historically has been, "Eh, companies don't," and then they're surprised when the researcher posts a big blog post, and then they get very upset about the phrasing and the choice. It's like, for a vulnerability security researcher, this is the sort of thing that you build a career out of if you can show a track record of doing these things. Whereas companies would love to be able to fix the problem on some level and then never tell a soul in their idealized version of these things.

Because honestly, who wants to trumpet the things that they get wrong? Few of us, if any. I'm just weird like that. There's a certain sen- but again, I'm not saying that AWS is trying to sweep things under the rug. Far from it. Whenever I've engaged with the team, the first and overriding concern has always been, is there an issue here that affects customers?

And if so, how do we fix that immediately? At a few calls where I've had conversations before things went out, there's been someone from PR on the call. And I was always a little, "Hmm, is this about to be where they're going to attempt to spin things?" And never once did the PR person say anything until I dug into.

"Why are you here exactly?"

"Oh, I want to see how the context is and how you perceive this to be, because you're not going to be the only person that has these opinions, and I want to make sure that we're prepared. I'm here to learn. I'm not here to tell you what to say." Which I found incredibly gratifying.

Ryan: Honestly, that's one of the things that I think we try to push a lot is the emphasis on communication.

We make everyone on our team go through like writing classes and different forms of additional education, getting trained on the internal services and terminology and everything to make sure we have a consistent and clear response. And what I believe you're describing is kind of the difference between responsible disclosure and coordinated disclosure, right?

Companies that will traditionally accept a report and then be kind of quiet about it and nothing happens. Well, they're not doing any coordination. They are just accepting a report that somebody gave out of the goodness of their heart responsibly and getting nothing back out of it. Coordinated disclosure allows us not only to work with you as an individual, but once we analyze.

And validate your findings and your submission. We're going to do an additional scope analysis. So what internally is impacted? Is it a single application? Is it 100 service teams? Is it, you know, what is the scope of impact? And then if we find that we've submitted is actually in, say, an open source dependency that isn't us, but our service or application uses.

Well, we're not going to stop there. We are going to reach back out to you and say, you know, we found the issue. We understand why it looks like us, but it's a dependency. This is the dependency. Would you like assistance in submitting to them and coordinating with them? And then we're going to provide guidance and assistance for the researcher to do that.

Sometimes it's through, you know, direct email. We find a contact point and we just do an introduction. Sometimes we'll open a Vince case. Vince is a wonderful, wonderful tool that I don't think a lot of the community knows about where they specialize in coordination at mass scale.

Corey: I first learned that this existed with the recent CUPS vulnerability and disclosure.

Ryan: Yeah, that definitely used Vince, but I can see why that came out and why it's not commonly known. I, myself, who have been practicing security for 20 plus years, both at the endpoint side, you know, done the IT work, did cloud security, was a CISO, did all the fun stuff, right? I didn't know about it until I started doing, uh, the vulnerability disclosure portion of the security realm.

Because it was never needed for what I did previously, even when I did vulnerability management for my companies. So it is a very cool and useful thing. People that aren't aware of organizations like First or Cert should definitely give them a Google or whatever search engine you feel like using will DuckDuckGo them.

And what you're going to find out. is that there are these different entities that will do coordination for you and help you find the proper contact points for submission of vulnerabilities, and it's free. Anybody can submit to Vince. If you want to open a Vince case, you can open a Vince case and be like, "I found a vulnerability and I believe it impacts these vendors," and request their assistance at coordinating and disclosing that vulnerability with those vendors and getting fixes, getting CVEs generated. All that kind of information. So yeah, so kind of going back to how you said responsible disclosure, and then just never hearing anything back, and then blogging about it. That's not the experience we want people to have. We want people to be engaged. When you submit to our program, you're going to get A human response in 24 hours, right?

That's, that's one of our SLAs. After that, the actual submission is going to get reproduction done and validation, right? We want to triage it and make sure that it is an AWS security issue. If not, we help you find the contact point and assist there. If it is an AWS security issue, we are then going to take it and ask, "Are customers impacted? Is it service side? Where is it?" etcetera, because that's going to give us an idea of starting. Do we need to do security advisories? Do we need to do customer notification? Do we need to post publicly in mass about it? Things like that. And, um, as part of my talk at fwdcloudsec: Europe, we went over. the actual high level coordinated vulnerability disclosure process.

Corey: To be clear, I want to, I want to call something out here because you folks have been great at the, at the piece that a number of companies miss, which is continuing to remain in communication with the researcher while you're doing all this stuff behind the scenes.

Because you can't, you can't necessarily tell them, blow-by-blow what's happening behind the scenes. And you can't make this comparison, but I absolutely can, if we look at the last few years of Azure security issues, and the timelines that are reported, that the vulnerability researcher reports it, and like six weeks later, they're like, "Hello, is this thing on? Did I forget to hit send on the report?" And you have to goad them into action. I have never had that experience with AWS. Not once.

Ryan: I'm super happy to hear that about AWS.

Corey: And I've never heard of anyone having that reaction.

Ryan: I love that because we pride ourselves on the communication aspect.

Corey: On the being ignored. I have to, this is a full disclosure, in the early days, when it was an email with a ticket, it's like, "Okay, it's been four days. Should I not have received an email other than just an autoresponder of, 'Yup, we got it?'" And yeah, there was some, there was some, I think that at the time the response did not clarify timeline expectations and the rest.

Just because you have an internal SLA of X days to have a human talk to the person does not necessarily mean that the human knows that. So there's a, there was a lot of early days teething issues that, to my understanding, have been smoothed over remarkably well.

Ryan: I'd say that comes down to two points.

One, uh, we have an internal email rule that just filters all your stuff to feed the delete bin.

Corey: That's generally what I tend to expect happens with my emails. Cause that's me.

Ryan: You know, it's standard practice across the industry at this point, but what's kind of nice is that when it comes into the inbox, like we have the human triage team guy. You get a response saying that we received it, so you get that instant gratification and notification that yes, the email that you sent did make it across, but then we do have that 24 hour SLA, which is posted publicly on our Vulnerability Disclosure page. So that's the AWS Vulnerability Disclosure page and the policy page on our HackerOne VDP program.

Both places publicly state our SLAs for response. The commitment that we're always going to make for you.

Sponsor: This episode is brought to you by Gitpod.

Ever feel like you spend more time fighting your dev environment than actually coding? “Works on my machine issues” are too familiar and the VDI setup in your organization drives you mad?

Gitpod brings automated, standardized development environments to your laptop—and the cloud.

Describe your dev-environment-as-code and streamline development workflows with Automations. At the click of a button, you get a perfectly configured environment and you can automate tasks and services like seeding your database, provisioning infrastructure, running security or scanning tools or any other development workflows.

You can self-host Gitpod in your cloud account for free in under three minutes or run Gitpod Desktop locally on your computer.

Gitpods automated, standardized development environments are the fastest and most secure way to develop software.

Gitpod is trusted by over 1.5 million developers including some of the largest financial institutions in the world. Visit gitpod.io and try it for free with your whole team.

And also, let me know what you think about it. I've honestly been looking for this for a while, and it's been on my list of things to try. I'll be trying it this week. Please, reach out, let me know what you think.

Corey: Something else I didn't realize that you have to deal with, that I, because until I get to a point where I had to deal with it myself, is the sheer number of "security researchers," and you should hear the sarcastic quotes around that when I say it that way, who spam out these emails where they run some scanner on a website and say, "I found vulnerabilities in your system."

And I followed up with one once the first time I saw it, and it was, "You aren't hard failing on SPF, email delivery. It means people could spoof your email domain." It's- okay, I come from this world. I would take a 3 percent drop in deliverability of my email newsletter, because people use forwarding services.

Sure, they're not configured the right way in some cases, but I'd rather people read my words than be technically correct against an issue I really don't concern myself with. People are like, well, I still found it. Pay me. It's, you know, That is not how this works. They're basically shakedown artists. So you have to sort out the wheat from the chaff when it comes to figuring out, is this a real threat or is someone with an opinion or a naive security tool just trying to basically get people to pay them?

Ryan: Yeah, I think the industry terms for it is like "bounty beggars" and things like that. The people that are not really focused on improving security, but they're looking for the lowest hanging fruit to try and get some kind of monetary reward for it.

Corey: I'm glad you mentioned the word "bounty" because this gets a little closer to where I am at this point.

I used to work in security and then I decided no, if this makes me too happy I want something even more depressing than InfoSec. So I went into AWS Billing. So the economic incentives behind these things are fascinating to me, but as I've learned somewhat recently, a vulnerability disclosure program and a bug bounty program are two distinct things.

Ryan: Correct. There's actually three tiers that are really commonly issued out. The first tier is kind of what we have launched right now. The AWS, the VDP. So a VDP is really meant to be a open door to the internet. There's no reputation limits in order to report to it. There's a clearly defined scope, but it's incredibly broad scope and involves, in ours, particularly, we have close to 400 services, which is pretty much everything AWS has, whether it's open source, a managed service, one of our client apps, anything like that.

So it's a huge, huge scope, and it's really meant to have that engagement with the community. The second tier is usually called a VRP, a Vulnerability Reporting Platform. And that's where people start kind of playing around with the idea of moving to monetary, and it comes with a lot of issues of just implementation to begin with legal stuff, trying to figure out if somebody is really a contractor for you or not, if they submit a certain amount of things. There's a lot of questions about that people want to deal with.

And then the third tier is what people are most commonly called, "the bug bounty." And that's usually a private invite-only program. So you go public-wide, public-really small, and then private-small. And you kind of ramp up that way. It's a progression of maturity into the programs.

Corey: And in one of your slides at fwdcloudsec: Europe this year, you mentioned that there is an invite only AWS Bug Bounty Program, if I'm not mistaken.

Ryan: You are correct. It's actually one of the new reward systems that were publicizing for the VDP. So this was always an aspect that happened is that people that report to the security inbox, our VDP program, we would recommend and nominate different researchers to be invited into the private program.

But that's only one of the benefits of it. You know, we also have a leadership board. So with your reputation points, you can see if you're the top researcher of the day, week, month, year, ever. That kind of fun stuff. We do swag vouchers and stuff. We have promo codes. So this week alone, I've given out a ton.

Corey: Please tell me that this is not the epitome of the Bug Bounty Program, "Wow! You found a zero day remote code execution, hypervisor escape on EC2. We got you a t-shirt in return, but you get to pay the shipping on it."

Ryan: No, sir.

Corey: Seems like it'd be better not to have that kind of program.

Ryan: 100 percent agree. As part of building this program, we got an official merch store.

It's an AWS merch store. The shirt I'm wearing right now is actually from it. It's, it's actually really nice quality, which is why I chose to go that direction. But no, your rewards are incre- are entirely tied to severity. Right? One of the things that we are doing right now, the program is new, people are starting to use it instead of the security inbox. So pretty much anyone that submits a valid security issue or concern to AWS through VDP, when we resolve it, we're issuing you some swag. And the swag promo code covers a certain allotment from the merch store, and covers shipping. Yeah, I made sure because the first round didn't, and I had to ask questions about that.

Like, "Cool. Here's a present. Pay for shipping." No. No, no. That's not going to happen. But out of that, we also, a couple other like non-tactile things, like it's really hard to reward people across the world with physical items. So we're looking at other ways to do so. So skill badges is an attribute that the HackerOne platform has, and right now, the same thing as when you, how we issue swag, we're also issuing out a badge for the service that you have successfully submitted a valid report for. This will do two things. One, you can use badges wherever you want, LinkedIn, your resume, etc. And you're able to count from them how many successful valid reports per service you have done.

But also, we're going to be tiering these out, and after you do a certain number of submissions, you get a new gold badge, and so on and so forth to show you're upscaling. And we use those as part of the nomination process to figure out who's great at what services and would get special advice to special things.

Corey: I'm going to find one on Route 53, and I want that gold badge to actually be a physical trophy that you send me.

Ryan: If you find something nice and shiny, I will commit to getting you a trophy, but I get to choose what's on it. [laughs]

Corey: I think that's a fair approach because, oh yeah, good point. Someone will try and out snark me.

Oh god, that always ends well. I don't know, uh, for folks who are looking at me on the video, behind me I have a Minecraft pickaxe, foam thing, that people always ask me about. Like, where did that come from? I wound up writing an article once saying that AWS's approach to machine learning was like selling digital pickaxes into a gold rush, and the head of machine learning analyst relations sent me that in retaliation. It was genius. It's where is this creativity in all of the conference talks you give about this stuff? It sounds like Star Trek Technobabble. This is amazing. I get more commentary on that than I do the custom commissioned painting of Billy the Platypus fighting an AWS Billing dragon.

Ryan: I love the imagery, and quite honestly, like, that is a great, great marketing thing. I've been trying to get somebody to build inflatable bandhammers for swag giveaways at, like, conferences. You know, big inflatable one, has squeakers on the end, because one, it's easy to travel with. Two, your kids are going to love it.

And three, they can't hurt themselves with it. So it's like the ultimate giant bandhammer. And, uh, anybody who is listening and does that and brings it to a conference I'm at, I will hook you up with some swag because that is awesome.

Corey: That's a terrific idea. We have a booth at reInvent for the first time this year.

I'm going to run that past my team and see what they say. The problem is I'm not quite sure, "We fix AWS bills here," have a ban hammer. Like that, that doesn't quite align in the same way, but I love the idea so much I almost don't care.

Ryan: Ban large charges with the ban hammer. I don't know. We'll work on marketing stuff later.

Corey: We can definitely torture a metaphor to death to make this work. Yeah.

Ryan: I appreciate that. And that's what I appreciate about you is that commitment.

Corey: Exactly. It's the, uh, what is it? Uh, which has at least two or three leadership principles there. Bias for action. Dive deep. Yeah. Definitely

Ryan: Dive deep. Yes. Uh, are right a lot.

Corey: Oh, I tell people I'm right a lot, which is kind of the same thing, right?

Ryan: I mean, you got to fake it until you make it.

Corey: Disagree and commits, et cetera. Oh yeah. We have all the fun ones.

Ryan: I like that.

Corey: So what's next? You've, you effectively have fought what I can only assume were political headwinds to get this out the door, even if the only headwind that really existed was simply corporate inertia.

At AWS's size, that is no small thing. It's Incredible that this saw the light of day. It gives me hope that, okay, maybe there's a glimmer of old Amazon still shining through sometimes, and maybe there's going to be a different future than the last couple of years have been, but what are you aiming at next?

Ryan: So I am incredibly proud that we got this out, and it's a natural maturity progression to the program, right? As you grow and scale and work on the efficiency internally for how your mechanisms work, you're able to bring on more external sources and more input sources. So a couple of things I'd love to see us grow with is through the feedback mechanisms we have, whether it's us asking surveys on things like the cloud security forums or at conferences.

Corey: Oh, you do love your surveys. You carpet bomb the world with them.

Ryan: I love my surveys. Ours are a little different though, because we don't have an automated mechanism to do them. So I manually count stuff. But what is great is we have a few different ways to get feedback now than we didn't before.

Mainly, on back one platform after every report gets resolved, you have the ability to give some scoring on your experience, but you also have an open box to give feedback. And something that I constantly am asking people is, you know, as somebody who was a researcher, I built it for something that would make my life better and makes me happier with the experience.

I need more feedback from active researchers and community of what you want to see in the program. You know, you ask for more transparency. We have a lot of public documentation now. We gave that transparency. You wanted to make sure that we were meeting certain SLAs for responding. We're publicly committing to those SLAs.

So really, what's next? Well, people asked for like testimonials. Things they can put on their resume. Things that they can show their work for, more recognition methods. So besides badges, which are kind of cool on the platform, we're also doing public testimonials and personalized accolades. So this is something that will get customized for your report as well as the experience of working with you as a researcher.

So not specific to your report, but you as a researcher, what the engagement was like between us, um, and those are connected to your profile. So you're able to pull them as basically like accolades for you to show you had a great experience with a company like AWS.

Corey: How do you maintain that? Because I, again, I haven't seen this from AWS, but I've heard stories that are legion for a number of others, where the entire company's perspective is damage control and trying to convince people not to say a word ever about any of it. How do you fight back that, that natural force of, we have screwed up and we preferred not to see it on a billboard somewhere?

Ryan: It goes back to, you know, security is kind of job zero. I understand how people will be afraid to talk about their flaws and stuff, but security by obscurity has never been an effective method of defense for a long period of time.

No matter what in the digital age, information will always find a way to see the day of light, or, see the light of day, and, really, what we're focusing on isn't trying to sweep things under the rug, but make them valuable for the rest of the community. You know, we're able to do more disclosure now as well.

So part of the VDP is an option to disclose your report. Yes, there are some redactions in it, but it's mostly like PII and stuff that you probably don't want to get doxxed for. So we'll scan for that and make sure that gets redacted, but your report can get publicized. We give public statements now about all of your submissions.

So if you are writing a blog, not only will we collaborate with you, get internal subject matter experts to give a technical review of your blog and give feedback to make sure you're stating things in a technically accurate way, but we're also going to give a statement on AWS's stance so that way you can use it in your blog or your presentation or whatever you feel like doing for that.

Corey: You've become much more consistent about repor- about in the security updates that you put out on your security informational feed thanking whatever security researcher brought it to your attention. You didn't used to do that. You do now.

Ryan: That is correct. We have a group called Security Communications, our, our lead for that has been excellent with showcasing improvements based on community response and submissions from all across how AWS Security communicates So outreach, my team that I work on, I work as tech lead. We're just one of the teams in that org, and, but we also have all the messaging as our sister team, and we work heavily with them on getting out not only internal customer messaging, and the types of messages that you get as like your dashboard update or things like that, but also the mass communication emails to give guidance to possibly impacted users and companies, but also things like security bulletins, GHSA posts, CVEs, all those security advisories.

We write all of that stuff as well, so it's just a huge boon. So we're constantly improving the process and assessment for those types of messaging has been greatly improved over the last couple of years.

Corey: Messaging stuff is important. I was getting tired of, "Security is job zero," as a repeated talking point was on a bunch of conference slides, and great, I didn't believe it or internalize it or- until one of the most impressive things I've ever seen in a security disclosure was when, actually, AWS had an issue with AWS Glue. And the the analysis on it on the website was, included the phrase, "We have looked back at the previous six years of audit logs going back to the launch of the service and the only time that this was ever executed was when the researcher found it."

Which is- that makes me feel better than I was even before learning that there was an issue at all. It demonstrates an attention to detail, a defense in depth, the response wasn't, "What could my logs be?" It was one of the most reassuring things I'd seen, and it It really did, it really showed this was something that was internalized and believed rather than just dead words on a slide.

Ryan: If we ever do find impact on something that has been submitted, the the depth that we go through for analysis to find scoping and do risk assessment to AWS and its customers is incredible in my opinion, right? Not only like how far back we go, it's like, great, you know, years and years of data. Well, think of the scale of that data.

You know, if, if one customer can create petabytes of data in a day for logging. Now multiply that by all of our customers that use that service, then crawl through all of that, and still do that in a reasonably fast manner that has accurate results, and is able to give us confidence to say it is only executed once.

I've been part of multiple engagements in the last few years, and another one that was very similar for that kind of look back was one that, I believe it was Datadog, it was Nick Frechette with an Amplify issue. And what ended up occurring is that we went back, and we looked to see when it got fixed, when it got unfixed, all this, all the way back to the start of the service entirely, which is years and years old, and we're able to find some variants of the issue.

Um, so I know I kind of rambled there, but the emphasis on the lengths that we go to for the investigation portion of the responsible coordinated vulnerability disclosure process is just boggling sometimes. Just the amount of data that we go through and validate. It's really cool to see. I'm a big data nerd in general.

Probably know from being the co-author of AWS Detective is, uh, it's big data. We love big data.

Corey: You're working in the right place for it.

Ryan: Oh, yeah. And what's really cool for me as well, like previous lives of being blue team and then going to red team and somewhere in the middle is you get kind of a perspective of this is how somebody is thinking to get in because this is how you would get in.

Now let's figure out how to break that way forward so they can't get in, and now I'm a defender. But seeing what gets submitted and seeing how some people think, and you'd sometimes get to just put down the white paper, take a breath, and try to think of whatever trauma this person has gone through that has made them think possibly in this way to make this type of an attack vector work.

Like, what would ever make you look at doing something? And it's incredibly fascinating to me because the security community at large, they think different, right? Uh, it's not how to get from A-to-B-to-C-to-D, but going from A-to-D, and then B and C just worked out, and now you're backtracking, and how you made your connections, and you look like one of those crazy people maps on the wall where you're trying to find all the connections.

And, but the community is just putting them together based on their usage, their experience, the artistry of kind of the intuition of what they should keep poking at is always fascinating to me. I love that aspect.

Corey: I've always envied the skill set. It's one that I don't have. I just blindly trip over things every once in a while.

It's as I am. It's not the, "Eureka!" moment. It's the, "That's funny," moment.

Ryan: What I always love to say is like describe what, you know, the people do for a living in kind of our, our vector of the world. And I say, "Well, they go to the dark places on the internet, poke it with a stick and see what crawls out," right?

That's threat research and intel and all of that fun stuff, but what we don't talk about is, well, when the bugs crawl out of the dark, who's going to raid them? Who's going to, uh, spray that thing down? Who's going to make it known to everybody else that these bugs are crawling out? Well, that's the research community.

They already, you know, they shook the tree. They saw what came out. So now that they saw the bug, it's really kind of that moral/ethical choice. Do I tell people about what I saw crawl out? And that's kind of where the VDP comes in is people can share that information. They don't have fear of like retribution of it.

We have a Safe Harbor. We have a very clear scope of what is and is not in bounds. So as long as you read that you're, you're good to go, and I think that additional, I don't want to call it a safety blanket, but the additional assurances, I think, make the program very successful.

Corey: I think that that's, that's really, the proof is in the pudding and in staying the course for a, for a period of time.

I want to thank you for taking the time to speak with me. If people want to learn more, participate, or give you some of that vaunted feedback you're after, where's the best place for them to find you?

Ryan: The best place for feedback right now is either as the submission to the HackerOne program, which is hackerone.com/aws_vdp, or directly to our security inbox, which is [email protected]. Those are the best places to give feedback on the program specifically. But if you ever have questions or concerns, things like that, you can find me on the cloud security forum. I'm on that Slack.

It's a great place to network with other people that are working in the cloud, research or otherwise. And then on LinkedIn, I believe my LinkedIn URL is https://www.linkedin.com/in/cloudy-with-a-chance-of-security/. I am always happy to talk with people and network. You'll find me at a ton of conferences. One of the big perks of being in outreach is our emphasis on engagement with the community.

You'll find me at all the major conferences. I like to go around and chat with people and hang out. Black Hat, DEF CON, The B Sides, fwd:cloudsec, reInforce, reInvent where we have a presence, and if you want us to come check out a new conference, shoot us a message and let us know. We'd love to talk with you and meet more, and we'd love to probably host some more events and just get the community together to talk about what they're interested in, how we can improve, and what people like and don't like about it right now.

I really want to emphasize that this is a program by a researcher for researchers. I want your experience to be pleasant because I want to recognize the type of effort that you're putting in, the type of time, the hours, the creativity that you're putting in, and then also the fact that you're taking the effort and time to write it up and let us know.

Thank you. Like, I want to sincerely say thank you for that because it's really, really important.

Corey: And we will, of course, put links to this in the show notes. Thank you so much for taking the time to speak with me. I appreciate it.

Ryan: Thank you so much for having me. It's an absolute pleasure as always.

Corey: Ryan Nolette, Senior Security Engineer at AWS Outreach.

I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five star review on your podcast platform of choice. Whereas if you've hated this podcast, please leave a five star review on your podcast platform of choice, along with an angry comment that points out a vulnerability in my platform. Then I will follow my own policy internally, which at the moment is a post it note that simply says, "PANIC," and then call the police.