Last week, Ubiquiti filed a lawsuit against Brian Krebs for reporting he’d done previously around an alleged Ubiquiti security breach. Because they mentioned “Amazon Web Services” in one of the exhibits for the filing, I was tweeting about it within the hour, which should probably be a sobering realization for a certain team in Seattle once they put those pieces together–but that’s not really the point I want to make today.
The complaint is, bluntly, not particularly well drafted or articulated. It meanders in a bunch of different directions, it dances around some points, and basically is work product I’d not accept from a law firm I was hiring. As a result it’s taken me a bit of time to dissect exactly what its beef with Krebs is; should I have gotten any of my analysis here wrong, I’ll be absolutely thrilled to post a correction, and I’m remarkably easy to find.
The Timeline
On Jan 11, 2021 Ubiquiti sent an email to its customers (of which I’m one, but I missed the email). Brian Krebs wrote about it at the time which I also missed because I was still gasping for air after re:Quinnvent. The email stated that Ubiquiti had detected “unauthorized access to certain of our information technology systems hosted by a third party cloud provider,” which we now know is AWS. Note the phrasing: a casual reader could well take this to mean that the third party provider had been the ones compromised. At first glance, I sure did.
On March 30, 2021 Krebs posted an article stating that a Ubquiti insider going by “Adam” had reached out after his own attempts, as he characterized them, to blow the proverbial whistle both internally and to European authorities.
According to Adam, the hackers obtained full read/write access to Ubiquiti databases at Amazon Web Services (AWS), which was the alleged “third party” involved in the breach. Ubiquiti’s breach disclosure, he wrote, was “downplayed and purposefully written to imply that a 3rd party cloud vendor was at risk and that Ubiquiti was merely a casualty of that, instead of the target of the attack.”
Ubiquiti reiterated on its forum that there was no evidence that customer information was accessed. “Adam” retorted that of course there wasn’t, because Ubiquiti failed to keep logs that would show such things. Adam also attested that “[Ubiquiti] legal overrode the repeated requests to force rotation of all customer credentials, and to revert any device access permission changes within the relevant period.”
Ubiquiti’s stock price dipped roughly 20 percent, compared to where it had been trading in January. As of this writing, that is roughly where the stock price is hanging out.
On December 2, 2021 Krebs reported that federal prosecutors charged Nickolas Sharp, a senior developer at Ubiquiti with stealing data and attempting to extort Ubiquiti. Krebs’s article did not address this point, but the linked indictment states that Sharp had reached out to journalists, identified himself as an anonymous source inside Ubiquiti, and went on record blaming an outside attacker. At this point it’s reasonable to surmise that “Adam” is in almost certainty Nickolas Sharp.
And on March 29, 2022 Ubiquiti filed suit against Brian Krebs, which concludes this timeline.
The Complaint
Let me start by disclosing a conflict of interest beyond what I put on my disclosure page: My sole relationship with Ubiquiti is that of “being their customer, which apparently also means victim.” I have no business relationship with Brian Krebs or his company.
After sorting through all of the whining in the complaint, the heart of its allegations appear to be that Krebs has allegedly knowingly fabricated three specific things that are both untrue and defamatory about Ubiquiti.
I will address each of the three in turn.
(a) “Now a source who participated in the response to that breach alleges Ubiquiti massively downplayed a ‘catastrophic’ incident to minimize the hit to its stock price… ”
Even knowing what we know now (i.e., that Krebs’s source may very well have been the culprit!), this statement appears to be factually correct. The email sitting in my inbox says “unauthorized access to certain of our information technology systems,” and goes on to say “we are not currently aware of evidence of access to any databases that host user data, but we cannot be certain that user data has not been exposed.”
The disclosure buried deep within their February 5th 2021 10-Q filed with the SEC – but curiously not linked or cited directly in the complaint – mentioned only “[w]e are not currently aware of evidence of access to any databases that host user data, but we cannot be certain that user data has not been exposed….” Uhh… that’s a far cry from advising people that the attacker had enough access to provision themselves two backdoors, grab additional credentials, and suck down the company’s GitHub repositories as laid out in the DOJ charges . A scenario in which someone has accessed your production environment and you do not know what exactly they did with that access is about as close to “catastrophic” as infosec incidents get; the messaging spin presents as a transparent attempt to mitigate the damage of the disclosure. While it’s obviously speculation to claim that the motivation was to protect the stock price, that’s the sort of obvious inference that Ubiquiti will have to take pains to push back against, regardless of the burden of proof on the issue – and it remains to be seen if they have the evidence or persuasive capacity to do it convincingly. Plus, I somewhat doubt that a public company is prepared to go on record with a statement that aligns to “we don’t actually care about our stock price any,” given that shareholders will almost certainly consider that a breach of the company’s fiduciary duty.
And of course, Ubiquiti only sent its customers a “you may wish to rotate your credentials out of an abundance of caution.” Given everything disclosed so far via the DOJ charges and the complaint Ubiquiti filed, it seems relatively clear that Ubiquiti did not at the time (and very well may not now) know what customer data had been accessed or altered. In that light, it’s hard to classify their lack of forcing a password rotation as anything other than a massive downplaying of what had transpired. If you’ve gotta announce a breach anyway, I fail to see what additional benefit you derive from not forcing new credentials. I mean, you’re admitting to a security failure anyway, and it looks way more defensible when the dust settles and the accusations start flying around.
The second defamatory claim, which I’ve already partially cited above, reads:
(b) “According to Adam, the hackers obtained full read/write access to Ubiquiti databases at Amazon Web Services (AWS), which was the alleged ‘third party’ involved in the breach. Ubiquiti’s breach disclosure, he wrote, was ‘downplayed and purposefully written to imply that a 3rd party cloud vendor was at risk and that Ubiquiti was merely a casualty of that, instead of the target of the attack.’”
According to the FBI complaint and the credentials that Sharp had access to, read/write access to various databases is effectively a given based upon the DOJ’s indictment. I also want to be clear here: there is zero functional difference between access to these databases at AWS, and access to these databases had they instead hypothetically been hosted in a Ubiquiti physical data center. Going to great pains to point out it was a “third party cloud vendor” when so few other details were disclosed absolutely presents as Ubiquiti attempting to at the minimum strongly imply it was a third party’s failing instead of their own.
And the third allegedly defamatory claim:
(c) Finally, the article in its entirety implies that Ubiquiti suffered a severe data breach via a malicious external actor and Ubiquiti intentionally lied to its investors and the public about the nature and severity of the “breach” to protect its stock price and actively instructed its employees not to take reasonable precautions to protect customer data.
And here’s where I just don’t understand the point Ubiquiti is trying to make. Do they somehow think that a malicious insider having access to company data for weeks via a VPN is somehow better than a random outsider doing the same? I believe there’s a strong case to be made that it is in fact far worse!
“No, we didn’t get breached by an outside attacker; instead we hired a(n alleged) criminal and gave him access which he then used to extort us,” is absolutely not the strong statement of competence that they seem to think it is. It causes people to ask difficult questions, such as “hey, aren’t you the company that sent almost $47 million to thieves a few years ago?”
Security breaches happen; they aren’t fun, but they’re not exactly uncommon. Ubiquiti is, again, a publicly traded company! What the HELL does their internal governance look like?! There are certain actions that I cannot take in my company’s AWS accounts without triggering alerts to others at the company–and I’m not hosting the keys to customers’ infrastructure over here the way Ubiquiti is. There’s been a lot of speculation in various corners of the internet that suggest CloudTrail log retention was set to a single day (something AWS’s Security Control Policies can be configured to prevent in any reasonable environment), that the AWS root keys were accessed (something that sets off screaming alarms in virtually any off-the-shelf SaaS security tooling), and (via the DOJ) that sketchy access via VPN endpoints to company assets went undetected for weeks.
These are all signs of having utterly failed at establishing a reasonable security posture. It’s also highly relevant to Krebs’s defense of the lawsuit, so there’s effectively zero chance that the answers to a series of questions that distill down to “what is your internal AWS security posture” won’t be entered into the record as a part of the legal discovery process. There’s likewise effectively zero chance that those answers won’t be easily interpreted as “we at Ubiquiti, a company that builds networking gear to which security is paramount, absolutely suck at infrastructure security ourselves.” Forget anything Krebs wrote; “we didn’t take some very fundamental, easy to implement steps to control our AWS environments” is going to be far more damaging to Ubiquiti’s reputation than anything third parties are going to be able to say.
It seems to me that they’re accusing Brian Krebs of shoddy journalism at most; I have a very hard time understanding the defamatory nature that they’re ascribing to his writings. “Being bad at journalism” isn’t actionable, and the bar for defamation is wildly high in the United States.
My default position on the company has changed radically as a result of this lawsuit. Previously my position on the breach was sympathetic; Ubiquiti was a victim, either of an outsider or an internal bad actor! Now instead they’re very clearly attempting to stifle critical coverage of their blunder, setting fire to their brand perception, and bringing a breach most people had already forgotten about back to the current conversation.
The lesson for the rest of us is pretty clearly that we need to examine our auditing configuration. We should most definitely consider the separation of duties between folks who are highly trusted, and take pains to not allow the person with access to the root credentials for our AWS (and other provider) production accounts to also have access to our audit logs. And barring materially different information from what has been disclosed so far, I wouldn’t blame anyone for considering Ubiquiti to be a bully who does its level best to stifle criticism.
As someone who cares about AWS as a platform, I’m interested in how this unfolds from here. As someone who is something of a trusted voice with respect to “AWS security for folks without ‘Security’ in their job titles,” I’m kinda required to keep up on things like this. I’m paying careful attention to this lawsuit, and in particular to the anti-SLAPP motion I expect to be forthcoming from Krebs.
