https://www.bleepingcomputer.com/news/security/over-3-000-github-accounts-used-by-malware-distribution-service/

https://blog.knowbe4.com/how-a-north-korean-fake-it-worker-tried-to-infiltrate-us

https://arstechnica.com/security/2024/07/secure-boot-is-completely-compromised-on-200-models-from-5-big-device-makers/

https://www.darkreading.com/cybersecurity-operations/crowdstrike-outage-losses-estimated-staggering-54b

 https://cdn.prod.website-files.com/64b69422439318309c9f1e44/66a24d5478783782964c1f6f_CrowdStrikes%20Impact%20on%20the%20Fortune%20500_%202024%20_Parametrix%20Analysis.pdf

https://www.darkreading.com/vulnerabilities-threats/unexpected-lessons-learned-from-the-crowdstrike-event

Summary:

Episode 274: Malware on GitHub, North Korean Developer Scam & Secure Boot Failures In this episode of the Defensive Security Podcast, hosts Jerry Bell and Andrew Kalat discuss several notable security stories and issues. They start with a malware distribution service that leverages compromised GitHub accounts and WordPress sites. They then cover a security warning from KnowBe4 about hiring a supposed North Korean agent as a senior developer. They dive into the significance of two separate vulnerable firmware signing keys affecting over 500 hardware models. Lastly, they explore the massive financial impact of the recent CrowdStrike outage, with losses estimated at $5.4 billion. Throughout the episode, the hosts provide insights, potential solutions, and share personal experiences related to these cybersecurity challenges.

00:00 Introduction and Casual Banter

00:30 Funemployment and Retirement Reflections

01:54 Disclaimer and First Story Introduction

02:17 Malware Distribution via GitHub

04:24 WordPress Security Issues

8:09 North Korean Developer Incident

14:36 Lessons Learned and Recommendations

23:27 Secure Boot Vulnerabilities

29:19 Cloud Providers and Firmware Security

30:47 The Epidemic of Leaked Keys on GitHub

33:35 Challenges in Development and Security Practices

35:36 CrowdStrike Outage and Its Financial Impact

39:16 Legal and Technical Implications of the Outage

57:33 Concluding Thoughts and Future Plans

Transcript:

Episode 274 274

===

jerry: [00:00:00] Today is Wednesday, July 31st, 2024. And this is episode 274 of the defensive security podcast. My name is Jerry Bell and joining me tonight as always is Mr. Andrew Kalat.

Andrew: Good evening, Jerry. How are you? My good sir.

jerry: So good. It hurts. How are you?

Andrew: I’m doing good. it’s Wednesday, which is halfway through the week. So I can’t complain too much.

jerry: It’s just another day to me though.

Andrew: I, how are you enjoying your funemployment?

jerry: It is awesome. funny story, when my dad retired, he told me something sad. He said, one of the things that you don’t realize is that the weekend starts losing its appeal,

Andrew: Because every day is the weekend.

jerry: because it’s just another day and, holidays are just another day.

jerry: There’s not really something to look forward to when you’re working. You typically look forward to the weekend. It’s just another day. I am finding that to be true. I’m going to be [00:01:00] spending some time coming up down at the beach, which will be a whole different experience, not having to work and actually be at the beach, which will be cool.

Andrew: So you don’t have to wrap your laptop in plastic when you take it surfing with you anymore.

jerry: That is very true. No more conference calls while out on the boogie board.

Andrew: I will say the random appearance of sharks behind you on your zoom sessions will be missed.

Andrew: Of course, we’ll have to find a way to bring that back. I live in jealousy of your funemployment. I will just say that. But not that you didn’t work your ass off and earned it, right? This is 25 years of blood, sweat, and tears given to this industry to get you to this point. So you earned it

jerry: I’m going to have to be responsible again at some point, but I am having fun in the meantime.

Andrew: as well. You should

jerry: before we get into the stories for today I just want to remind everybody that the thoughts and [00:02:00] opinions we express on the show are ours and do not represent anybody else, including employers cats, farm animals, spouses children, et cetera, et cetera.

Andrew: there’s that one Lama in Belarus though, that agrees 100 percent with what we have to say.

jerry: Very true. Getting into the stories, we have one from bleeping computer and this one is titled over 3000 GitHub accounts used by malware distribution service. I thought this one was particularly interesting and notable. There is a malware distribution as a service that leverages both, let’s call them fake or contrived GitHub accounts, as well as compromised WordPress sites.

jerry: And the, what they’re effectively leveraging is the brand reputation of GitHub. And so they have a fairly complicated setup of driving. [00:03:00] Victims through watering hole attacks and SEO type lures to get people to these sites and they have different templates that entice people to download these encrypted zip files that are hosted on GitHub.

jerry: And what they’re taking advantage of is two things. Number one, people generally think that GitHub is a reputable place. To find files. And so you’re. Level of concern goes down when you download something that you think is coming from a reputable place. And I think the other, perhaps more problematic angle from my perspective, at least is GitHub is something that most companies allow access to.

jerry: it is something that, by design, many companies, not all, encourage their employees to interact with GitHub. And so you really can’t block it. [00:04:00] And or at least it’s more difficult to block it. And because it’s one kind of amorphous. Thing you, you don’t have the ability to granularly say you can go to this aspect or this part of GitHub and not this other part of GitHub.

Andrew: Yeah. I agree at all points. It’s absolutely leveraging and abusing the reputation of GitHub to get this malware out there and it’s effective. Using WordPress doesn’t surprise me, just about every day I see some other plugin has a massive vulnerability. So I’m not blaming WordPress, I’m blaming their plugin ecosystem as being highly toxic in the original sense.

jerry: I know that WordPress has a lot of detractors, especially in the security community, but It’s over 50 percent of the entire internet. Websites runs WordPress, right? That is pretty impressive.

Andrew: There’s something to be said for The amount of coverage or the amount of instances out [00:05:00] there equals how many bad guys are poking at you. So if you’re not widely deployed, you’re probably also not getting widely tested. So there is absolutely some of that aspect of, Hey, if you’re a well used tool, you’re likely to have more security problems.

Andrew: So statistically that makes sense, but it’s not a bad tool. Don’t get me wrong. It’s a super useful tool. It’s just amazing how often I see advisories about. Really nasty exploits on various plugins for WordPress.

jerry: Yeah. the barrier to entry for plugin development is incredibly low and there are just an absolute ton of them. There’s many thousands. So it isn’t surprising.

Andrew: People who are running WordPress sites are not super technical admins. They’re usually marketing folks or content generation folks. So, when they’re looking for, Hey, I need something that makes a pretty picture, do something like this in WordPress, they probably aren’t looking at with the same level of technical rigor you and I might.

jerry: I will tell you in in prior [00:06:00] jobs where we had customers hosted on our infrastructure, this was a big problem because, customers would walk away, right? There’s it’s so easy to set up. It’s so easy to set up a WordPress instance, which is by the way, like that’s part of its value proposition, but it’s also part of, I think it’s I think it contributes to the low ongoing attachment or ongoing care and feeding of it.

jerry: It’s so easy to set up and then just walk away from and it’s a big problem. I think that the WordPress team themselves have done a pretty good job of mitigating the issues to the extent they can. Most of the, most of it auto updates these days.

Andrew: Yeah.

jerry: More to go, right?

Andrew: what’s interesting to me is the way you describe that often sounds like the same problems we have with SaaS and cloud in general. It’s so easy to set up and walk away from and not manage it well. That we lead to all sorts of similar problems. This story is more about [00:07:00] GitHub and I agree with all your points.

Andrew: I just went down my WordPress rant rabbit hole, but yeah, I get it. GitHub is an interesting one. And I don’t have a lot of good solves for that one.

jerry: No, it’s It is not a, it’s not an easily solvable issue. I think this is one of those one of those cases where education will help certainly, proper end point detection, I should say end point protection. Will help being able to identify that somebody has downloaded and run something that is potentially malicious on their device.

jerry: But beyond that, unless you’re willing to take that leap and not not permit people to access GitHub, it’s hard to defend against. So by the way, if the industry as a whole decided, Hey, this is too risky, the bad guys would just move somewhere else.

Andrew: Yeah, absolutely. They’re leveraging whatever has a good solid reputation that has a sort of functionality. It’s not GitHub’s fault. And I think you’re right. If it [00:08:00] wasn’t GitHub, it’d be somebody else that serves the same purpose. function. It’s their victim of their own success in that sense.

jerry: 100%. So the the next story we have comes from the know b4 blog. Know B4 is. I think a fairly well known security education company. They are not without controversy in my my experience. I think they’re the ones that offer the the ransomware guarantee or warranty, if I’m not mistaken.

jerry: But this story is not about that. The story is about how they hired a North Korean, allegedly as a senior developer. So they they did what I would describe as a public service announcement describing how they had been duped by purportedly a North Korean, government agent who was trying to infiltrate their [00:09:00] company.

jerry: It’s a little unclear exactly what their mission was. But the story basically is that a this company was trying to hire a senior developer. They went through the interview process. They selected what they believed was the best candidate. They performed all their normal interviews, background checks and whatnot.

jerry: Then they made an offer. The offer was accepted. They dispatched a laptop to this person. And then suddenly that. Laptop was observed attempting to install malware. And that is what really set off the series of events that alerted them to what was going on. So in the end analysis, the the person they hired was again, somebody allegedly from North Korea.

jerry: They had used a stolen identity of a legitimate person in the U S. They [00:10:00] altered a photo, and it’s a little unclear to me exactly in what ways they show before and after of how the photo was altered, but it’s not entirely clear to me if that was the photo of the identity, the person whose identity they stole or somebody else that aspect isn’t clear to me, the laptop was sent to what they call an IT A laptop shipping mill, I think is what they call it.

jerry: Basically. It’s a place that will set these laptops up either on a network to allow access, low remote access from overseas, or they will send them out I think in this instance, it was allowing remote access in. So the the bad actor was apparently in North Korea, VPNing into the laptop and then connecting from the laptop, which I guess was located in California and accessing the company’s networks.

jerry: When KnowBe4’s security [00:11:00] team identified that there was malicious activity going on. They tried to contact this. Very new employee, and the new employee said, I can’t talk right now. I, I’m trying to, I’m trying to debug something, but I’ve got a personal or family issue going on. And so over the course of a couple of hours, the security team decided to isolate the laptop, and then things started to fall apart, and they realized exactly what happened.

jerry: It’s not entirely clear to me, by the way, from reading the article exactly how they stitched together that this person was a North Korean agent. That. That is not part of the at least the stories that I have read. But it’s interesting because this resonated with me personally. I worked for a large multinational company for a very long time.

Andrew: And you’re a North Korean agent.

jerry: And I yes, that’s, that is true. Sorry, everybody had to find out that way. And anyhow,

Andrew: Didn’t they all know? Is that breaking news?

jerry: [00:12:00] I think it might be for some,

Andrew: Oh you’re a very nice North Korean agent.

jerry: I try to be

Andrew: Yeah.

Andrew: Anyway. Yeah. It’s a crazy story. I, Sent this to some of my friends in HR and they’re like, what this happens? Yeah, it happens. So that tells me there’s some awareness interestingly according to The timeline that no before published their sock was on it. They Detected the users activity Suspicious activity began at 9 55 p.

Andrew: m. And by 10 20 p. m 25 minutes later. They’re like, nope shut me down.

jerry: yeah. But by all accounts, they did a great job.

Andrew: Yeah, and the SOC person was suspicious enough. And also the North Korean angle. It looks like, before we reached out to the FBI in Mandiant. And so it’s probably coming from one of those two who are saying, that this was a North Korean thing.

Andrew: Although from the story as well, it looks like this may not necessarily be malicious in terms of state sponsored. malicious [00:13:00] activity so much as it might’ve been just a way to earn legitimate money sent back to North Korea to fund whatever, as they say, at least according to what they quote in their blog post, the FBI was saying, sometimes this is just straight up people working in sanctioned countries, getting around the sanctions and doing legitimate work.

Andrew: Or, but this guy immediately started, Loading malware, which is not great.

jerry: Yeah. And realistically, if he hadn’t started loading malware, who, I don’t know if it would have been very easy to detect that this was going on.

Andrew: sure. So a lot of lessons learned though. A lot of takeaways.

jerry: Yeah. I, anyway I was saying that this is, this, It hit home for me because I have had similar, not with, foreign agents, North Korean agents or age, people in embargoed countries, but I have had experience with people falsifying who they [00:14:00] are when they get hired and it’s difficult to do, especially in these remote working.

jerry: Environments. I’m not saying that it can’t also be done in a face to face type environment, but I think that it is. Easier to pull off in, in today’s remote working environment. I’m still, by the way, a big proponent of remote working. Don’t misunderstand in any way. But I do understand this one.

jerry: I I can definitely sympathize someday. I will tell some funny stories about experiences I’ve. I’ve had, so there were, they did call out some tips and recommendations as a result of this. Again, this was intended to be a public service announcement. So they wanted to describe the, the event and what they learned from it.

jerry: So their tips to prevent were scan your remote devices and make sure no one remotes into these. So I think that was [00:15:00] really, addressing how the person was connecting remotely into the laptop. But that’s how, by the way, that’s not a even if you were to not permit VNC or remote desktop or things like that, it’s not a guarantee because there are plenty of cheap ass network KVMs, right?

jerry: So just be aware of that. Next one is better vetting. Make sure they, the employee are physically where they are supposed to be. I think also that’s a difficult. One a better resume scanning for career inconsistencies, by the way I’m assuming they are listing these tips because some aspect of the, that the root cause analysis is tied back to each of the, each of these things.

jerry: So I’m guessing that if they had looked more closely, they would have identified inconsistencies in the career of the person whose [00:16:00] identity they had stolen. And so I think what they’re saying is, make sure that everything ties out. Then next one is get these people on video camera and ask them about the work they’re doing so that you can see firsthand, a, that their picture looks like who you think you’re hiring and B that you can understand, that they are in fact who they say they are.

jerry: And I can definitely tell you firsthand that’s an important one.

Andrew: Yeah. But that’s getting easier and easier to fake.

jerry: Yeah.

Andrew: going to be an interesting problem as, deepfake AI technology is getting cheaper and more widely accessible. So

jerry: So Bob, our friend Bob, you told me about a situation in a Asian country where it is At least in the recent past fairly common for [00:17:00] what I’ll loosely call, or what he loosely called job fairs, where people who were job prospecting could go and hire an expert. Like they could, the person, this person would offer their services to go through interviews and.

Andrew: wow,

jerry: And, purport to be the person who’s looking for the job and they would rely on like the, the silly Westerners inability to distinguish between different Asians and they would end up hiring, And bringing on board somebody who was not the person they had actually interviewed.

jerry: And it was it was done at an industrial scale.

Andrew: obviously that is a short lived scam for the person getting hired, but I guess they get a paycheck or two out of the deal and move on.

jerry: I think, and I think sometimes it, sometimes I think it can work out for a long time, [00:18:00] unfortunately.

Andrew: I guess that’s true. Wow, that is crazy.

jerry: Anyway Next one is the laptop shipping address is different from where they are supposed to live at work is a red flag. So I’m assuming that this person declared that they had an address in place a, and they asked for the laptop to be put shipped to place B and that, that should be a definitely red flag.

jerry: I. I’m a little surprised that has to be called out. Seems obvious to me. They did have, they do have a couple of recommended process improvements, including a background where background checks appear to be inadequate and names are not consistent. So basically where that the person’s name And the results of their background check don’t necessarily line up correctly.

jerry: You need to make sure that you’re properly vetting your references and do not solely rely on email [00:19:00] based references. I think that’s also a good thing, although not airtight, especially if you’re trying to fend off, any type of I want to say nation state, but if it’s the adversary, it’s a kind of a low bar to top.

Andrew: If they’re already a scammer, they can probably get over that hurdle of having three valid, references who will lie for them.

jerry: That’s my thinking to implement enhanced monitoring for any continued attempts to access systems. That makes a very good sense. Obviously if. If you terminate this, by the way, I don’t think it’s unique to the situation. When you terminate someone, you need to make sure that their access is terminated globally and you don’t have back doorways into your environment.

jerry: A review and strengthen access controls and authentication processes. And then finally conduct security awareness training. And then they have a couple of what to look out for is like using voiceover IP numbers, discrepancies and addresses and dates of [00:20:00] birth conflicting personal information like marital status and then also family emergencies. I think, again. When the company soc tried to contact the employee and ask what was going on.

jerry: This person said, oh, I can’t talk. I’ve got a family emergency going on. And then sophisticated use of v VPNs and VMs for access for accessing custom or for company systems. I think that’s, by the way, very important one in something that, that I have. I know Bob was doing pretty pervasively was looking for evidence of unauthorized connections from known VPN sources,

Andrew: Right.

jerry: And then attempting to execute malware and subsequent coverups.

jerry: So I think all really good. I will also say, by the way, that I, I think, obviously, no, before published this to get their name out, it was intended to help, I what that one judge called corporate puffery, [00:21:00] which is my new, my, my new favorite term, the legal term now. But I wish I, it was shared far and wide, but it was very clear because this company had to hit the post, the followup blog post that this post was being interpreted as they’re disclosing a data breach.

jerry: Thought that angle is also very interesting that you have to be careful how you couch things and they, in, in the FAQ, by the way, they actually draw a delineation between how the InfoSec media industry covered their disclosure or their privacy. Public service announcement and how the general media disclosed it.

jerry: The InfoSec press, as they describe it, covered it as a lessons learned. Here’s here’s some things to look out for and did what they had hoped, but the general media basically just covered it as if it was just another data breach, [00:22:00] which I thought was quite interesting and something to be aware of.

Andrew: There’s two other things that I took away. Interesting that one, their EDR software caught the malware. So that’s important. I don’t know if it was CrowdStrike, but we’ll just mention that just for the, the circuit search algorithms, because everybody has to talk about CrowdStrike. The other thing that I thought was interesting is the way this is written.

Andrew: It’s clearly a bit of an internal document this the way that the Verbiage is used in the tenses and that sort of thing. So they say here.

Andrew: Let me find it really quick Ah, I don’t have to it’s good We have new employees in a highly restricted area when they start and they have no access to production systems So it sounds to me based on that, that they have some sort of tiering for brand new employees that they don’t automatically get access to anything, that they’re some sort of training period or ramp up period where they’re vetted or whatnot, whatever the purpose is for that particular sort of [00:23:00] sandbox that they’re left in initially.

Andrew: Not a bad idea for this sort of circumstance.

jerry: a great idea and a great call out. Yeah,

Andrew: Crazy story though. I’m really glad that, before. Published all this and we can learn from it. Kudos to them. Yeah, you’re right. There’s probably some cynical self serving nature to it, but nonetheless, it’s good fodder for us.

jerry: absolutely. Absolutely. And I think it’s happening probably a lot more than we, we want to admit. Moving on to the next story, this one comes from Ars Technica and the title is secure boot is completely broken on 200 plus models from five big device makers, So I think most people hopefully know what secure boot is.

jerry: It’s intended to to establish a level of trust in the firmware that is running on the hardware. And that was back in the early 2010s late Ts early 2000 tens a big deal because there were some pretty novel firmware [00:24:00] based attacks and the specter of really persistent and very difficult to eradicate firmware level mail malware was was running pretty rampant.

jerry: So the industry created this secure boot concept, and it relies on public key cryptography. And the one thing that everybody should realize by now is that we really.

jerry: And so there are two distinct problems here. That’s it’s identified in the article. One is that. Back in 2022 AMI, one of the signing keys from AMI was posted on a public GitHub repository. Now it was encrypted. It was strongly encrypted with a four character password. So it was very easy to decompile, reverse engineer.

Andrew: Indeed.

jerry: Yeah. So that basically means that systems which use [00:25:00] secure boot that, that relies on that particular key can be bypassed by an adversary. They can now load malware. They can basically sign a malicious version of firmware and run that on the system. So not

Andrew: It’ll be trusted. And the only way to fix this is basically to deploy firmware updates to the hardware to rotate the key, which is not going to happen.

jerry: And by the way once a system is infected, like if you have malicious firmware, it is very difficult. This is not like you can necessarily just wipe the hard drive and start over. It’s very difficult to eradicate the tools in detection capabilities for malicious firmware are very immature.

Andrew: Mhm. this is a tough one. When you’re relying purely on the secrecy of a [00:26:00] key, and if that key were to get leaked, in this case, it blows up that entire ecosystem. Wow, that’s rough. And the other thing that I’ll bring up is I have seen so many secrets get pushed to GitHub by accident. It’s so easy to do and it’s a huge problem that I don’t know that we talk about enough.

Andrew: Once it’s pushed too it’s almost impossible to ever wipe from that record because it’s permanently in that, repos history. So once that happens, you’re basically forced to rotate that secret. And in this case, you can’t because it’s relying on firmware you can’t centrally update. Without massive amounts of effort.

Andrew: So now you’ve got this problem forever

jerry: Yeah, correct. Now in the,

Andrew: for those hardware.

jerry: In this instance, in the other instance, which is maybe a little less bad I guess in some ways it’s better. It’s less bad in other ways. It’s worse. [00:27:00] What we just talked about affected 200, 200 or so models. There were another 300 models that were impacted by a different problem.

jerry: And that other problem was that a, one of the firmware manufacturers released prototype code that had a stock.

jerry: And in fact, the common name in this key is do not trust. So the hardware manufacturers did not actually rotate the key as they were supposed to. And so in the analysis of The first problem that we were just talking about, this company called Binerly was doing some analysis of other systems, looking for other problems.

jerry: And they found apparently a very pervasive problem across many different. Models of hardware that have this, stock test key [00:28:00] instead of an actual legitimate key. And so it’s literally the same key used everywhere. It was never intended to be used in production systems but apparently the hardware manufacturers think at the memo, I guess the one good news in all of this is that I think all of the hardware now is, or almost all of it is out of support and probably pretty old.

jerry: Not necessarily a good look in terms of our ability. And I like these. Who knows how many people have had these keys over the years and provided an opportunity for, bad actors to leverage the vulnerability. I would say for most people or for most organizations, it’s not like an exigent threat.

jerry: Because something else has to go horribly wrong, in order for a bad person, unless you’re talking about like a laptop or a desktop personal, like a PC from a server [00:29:00] perspective, something else has to go horribly wrong. They’ve got to, you’ve got to get, be compromised. And if they did, this provides a really gnarly method of persistence.

jerry: If that were to be properly leveraged, but the one, again, coming from. Where I was, there is a category of company that has to worry about this like very deeply, and that is. Cloud providers.

Andrew: that’s true. Somewhere there is hardware. Backing all that stuff up.

jerry: Yeah. And, because you have to have administrative rights on the system in order to flash it. You are like, I run a infosec. exchange and I have a bunch of bare metal servers that I rent from a data center. I have root, I could conceivably update the firmware on those.

Andrew: You could.

jerry: If I up, if I updated the firmware in a way that maintained persistence and my [00:30:00] hosting provider was not sophisticated, I could. Establish a means of persistence to compromise the next person who, or organization that is using that piece of hardware.

Andrew: You rent it. You flash it with your code. You un rent it. And they rent it to somebody else.

jerry: Yes,

Andrew: It doesn’t matter if they loaded fresh OS or whatever. It’s sitting there in firmware. It’s at the BIOS level.

jerry: exactly. So that those are the kinds of things that kept me up at night.

Andrew: But to your point, I don’t know how much of a real world hardcore threat this is. It’s a very interesting situation, but I don’t think that’s what’s going to burden most companies.

jerry: No. I think it, so you touched on something. I wanted to hammer home and that is our just terrible stewardship of keys particularly as it pertains to GitHub. I’m sure [00:31:00] there are others, but like GitHub is, I suspect if in 50 years going to be viewed as like the nexus point for many data breaches that were caused by leaked keys it’s

Andrew: It’s just too easy.

jerry: It’s an epidemic.

Andrew: and I’m not throwing shade at the developers. It is just so easy to do it that even the most well intentioned and diligent developer can get caught by this, even if you have tools that are supposed to check for secrets before that PR request is completed, they’re not foolproof, and the secrets look differently, and they’re not great.

Andrew: There’s no one perfect tool for this. So it’s, I don’t know, the only thing I can say, and it’s really tough when we’re talking about firmware based things, but everything else is try to design with the intent that you’ve got to rotate those keys on a regular basis and make it easier for yourself.[00:32:00]

Andrew: Much easier said than done.

jerry: Certainly true. I will say if you start out with that design principle in mind, it’s easier. It’s much easier than going back and trying to retrofit. So I, and I think that’s, I think that goes without saying, but it is doubly true in this instance. So keep an eye on those keys. Definitely rotate them when, and where you can minimally know what keys you have and how you would rotate them if they were disclosed.

Andrew: And I don’t want to get down in the weeds of all the coding practices, but because I’m certainly not an expert there, however, what I, as a very dangerous level of knowledge that I have just enough to be very dangerous, I understand there are ways to not directly call those secrets in your code or contain them in the code.

Andrew: You can call them as a reference and [00:33:00] otherwise teach your devs how to do that so that if they don’t have to worry about. That secret accidentally getting pushed is one very crude example of a very complicated problem that I’m not doing a good job talking about. But for those who know much better than I, try to help your devs.

Andrew: Now put secrets in GitHub, please. Thank you.

jerry: It’s, it is a difficult. And I will tell you, wrestled with this problem mightily, it requires you to create an ecosystem. And one of the challenges I’ve observed is that. And we are how best to say it, commoditizing development. I know that’s, perhaps a crass way of saying it, but we are, we’re putting people in development roles that don’t really have a lot of development background.

jerry: And we’re, we’re basically throwing them out of the nest and [00:34:00] saying, okay, it’s time to fly. And they know how to write code, but they don’t necessarily know that they’re not even, Aware that this class of problem exists until, somebody from the security team comes along and says, what did you do?

jerry: Let alone knowing how and where to actually store. And so very incumbent on, the leadership of the security team to make sure that ecosystem, both exists and your developers are aware of how to use it. Yeah. Otherwise you, this is going to keep happening.

Andrew: A single slide during onboarding isn’t good enough?

jerry: Oh, so it’s add that slide to your annual training and you’re good.

Andrew: All right. And maybe just something like just don’t share secrets. Just as a bullet point, move on. I think

jerry: So yeah don’t share. Yeah. Don’t share secrets, right? Don’t share your passwords. That’s right. I have, by the [00:35:00] way I have, by the way I have a been involved in in situations where something like this has happened. And I. Have asked the person, like at what point did you think it was okay?

jerry: This is a password. And he said, no, it’s not a password. It’s a secret.

Andrew: Right.

jerry: Not allowed to share passwords. There’s no, no prohibition on secrets. So

Andrew: makes my code go. I need it. It’s important.

jerry: very cautious about terminology. All right. Let’s keep plugging along here. The next one comes from dark reading and the title is CrowdStrike outage losses estimated at staggering 5.

jerry: 4 billion.

Andrew: And I’m going to go on record and say that’s probably low.

jerry: Yeah, I, it was funny in the short after immediate aftermath of the the event, or maybe a couple of days after somebody was asking the question of, how much money that [00:36:00] was lost. And at the time The number we just gotten the number of eight and a half million systems impacted. And so I did a bit of bad back of the net napkin math.

jerry: And I came up with between five and 10 billion, which is pretty interesting. That’s coming out to be a 5. 4 billion. I do think that by the way, the eight and a half million number is now being walked back. I haven’t heard a new number. But Microsoft seems to be signaling that eight and a half million is probably very conservative and CrowdStrike doesn’t seem to be very motivated to go correct the record.

Andrew: Yeah, from what I understand, Those were just the systems that were able to send a crash dump to Microsoft is what they based it on And they’re now saying that was probably just a subset of some so that 8. 5 million number that we’ve been talking about It’s probably Way low I would i’m just going to take a guess.

Andrew: It’s at least [00:37:00] 10x that number.

jerry: It’s a lot, it’s a lot, it was it was an interesting discontinuity because I think CrowdStrike has something like 15 percent of market share and this was like 1 percent or less than 1 percent of Windows systems that were impacted and it was hard to reconcile those. And now I, what, the only thing I could think of is that, That 1 percent included home users, but I don’t know, it just, it doesn’t seem right.

jerry: It’ll be very interesting to see what the actual number is. I think it’s certainly going to be a lot higher than eight and a half million, just based on, the news coverage how many freaking airports did we see pictures of people on ladders, resetting the displays?

jerry: You know, it’s mind boggling the amount of effort that went into recovering from this. I

Andrew: side story. I saw this amazing [00:38:00] article I won’t do it justice because I didn’t have it prepped but somebody figured out they could use a barcode scanner here To speed up the process because Windows treats a barcode scanner as a keyboard, and they figured out how to encode the actual command with the, um, the the encryption key they needed.

jerry: a key that bit like a key. Yeah.

Andrew: Thank you, the BitLocker key. And created custom barcodes for HPC that was tied to it. And instead of typing it all in, just got to the point where they needed, they scanned the barcode and boom, it was recovered. And I was like, I need to go find the story. It was amazing. Little bit of a hack that I thought was awesome.

jerry: Yeah. That was a,

Andrew: also I was thinking.

jerry: it was going to say it was a company in Australia that did that, I think if I remember, yeah, that was very clever.

Andrew: I was also thinking randomly when I saw that same picture of those people up in the ladder fixing that airport monitor. I’m like, if you’re down to that monitor, you probably got your critical stuff back.

jerry: That is very true. Although different teams, [00:39:00] probably different

Andrew: That’s true. That’s true. And I know there’s a lot more to talk about here, but one last thing I’ll say just since I stole the, we’ll probably get a lot better read on this because a whole bunch of legal actions is starting to spin up and that’ll all come out in discovery.

jerry: Delta airlines, notably, we were sharing some stories back and forth about Delta airlines before, before the recording here announced that they estimate their losses at half a billion dollars and they have retained the legal council and they’re contemplating legal action against both CrowdStrike and Microsoft.

Andrew: Yeah. The other interesting thing since the last we talked about this that came up, we were taking a little bit of a grain of salt, but we were talking a lot about why CrowdStrike was operating in the kernel and that, if that was a good idea or not, and why are they allowed to do it and should they be allowed to do it, et cetera, et cetera.

Andrew: And you can’t do that on a Mac. And one thing Microsoft came out and said is that based on an EU [00:40:00] legal They were forced due to antitrust type considerations to basically allow any security tool to have the same level of access as Microsoft’s built in security tools. Hence, they didn’t want Microsoft to have monopoly access to the operating system.

Andrew: So they needed to allow third parties to have the same capabilities. So if Microsoft defender has the kernel. Therefore, competitors need to also be able to have kernel access. That’s going to be an interesting thing to see play out. And I think Microsoft is walking a very careful line of pointing that out without playing too much of a victim.

Andrew: But it is an interesting caveat that I think adds to the nuance of the conversation we were having last week on this topic.

jerry: Yeah they’ve come out pretty forcefully and said that they’re intending to refactor access to the kernel. And I think they actually made the comment that they are intending to move in a direction that is much more like iOS, the Apple iOS [00:41:00] ecosystem. So that’ll be be quite interesting to watch.

Andrew: Yeah, I don’t blame them. They, Microsoft’s in a tough spot here, I think. And I’m not a Microsoft lover or an apologist, but they’ve caught a lot of flack for something that kind of wasn’t their direct fault, but somewhat the result of decisions and ecosystems and permissions that were granted. And I think they realize now that they just can’t succeed that.

Andrew: Ground or that they’re still going to be held accountable to what other people are doing in their operating system.

jerry: Yeah, I, in in the legal circles, there’s this concept of an attractive nuisance, and I have a feeling that’s going to be part of the basis of their, of claims against Microsoft is that, should have known.

jerry: And therefore, they’re somewhat culpable for not taking action. And anybody who, you know who’s had somebody slip and fall on their driveway, at least in the U S I don’t know [00:42:00] that this is a big problem outside of the U S you, you have that sort of, problem, right?

jerry: It’s an attractive nuisance. You should have done something about it. You should have known that this was going to be a problem. Somebody got hurt as a result. And now you’re you’re at least in part responsible. So again, don’t know. Not presupposing what a court would find. I don’t think this, by the way, is the kind of thing that would actually end up going to trial, but Hey, we’ll see this, by the way, this whole, this story here.

jerry: Is based on a report that comes from a company called parametrics and they are a I guess a think tank provides information to insurance companies. And so it was they who pinned the estimate estimated losses at half a billion, I’m sorry, at 5. 4 billion dollars. And they identified that healthcare companies were the most impacted followed by [00:43:00] banking as well as transportation companies.

jerry: We actually have a breakdown based on their telemetry. They have they, they’ve You know, they’ve identified different categories and how impact and how much each category of industry lost as a result of this net all totals up to the five point five point 4 billion. There were some interesting conclusions at the end of the report, which was linked in the article from dark reading.

jerry: And I’ll just read through those. And some of them, by the way, are a little confusing. Contrary to what I understood to be the case. So the first one is that recovery disparities between cloud and traditional infrastructures, they point out that cloud based organizations that were had their assets impacted in the cloud recovered much more quickly than those who had traditional infrastructure.

jerry: And on the one hand, I know that there were a lot of problems. But [00:44:00] on the other hand I’m wondering if that whole concept is skewed by the, the fact that you don’t have laptops and kiosks and whatnot, sitting in the cloud. And so those things that were impacted actually required you to dispatch someone to go out and do, whereas, You know you, with with infrastructure that’s sitting in a cloud, like somebody from their basement was able to just plug through them one, one by one without having to to actually dispatch people to go fix.

jerry: So I don’t know how much that played into it.

Andrew: I had one other thought there, which is that maybe a certain number of these systems that went down were abstracted in some sort of hypervisor and the management plane was still remotely available to sock. Or knock engineers who could fix it without actually having to put hands on physical keyboards at the devices.

jerry: That’s possible. Very possible.

Andrew: know, but because I’m assuming like the underlying infrastructure running it [00:45:00] didn’t go down. It was the actual virtual machines inside of it that were running CrowdStrike at that layer that, that caused, the blue screen. And can I don’t know, I don’t know if I can fix that easier than, having to go plug into a PC.

Andrew: USB port on a physical device.

jerry: I think they were able, if nothing else, they were able to use a virtual KVM to connect the systems.

Andrew: Yeah. So that’s much quicker. I would think.

jerry: Yeah, that’d be my expectation. So their second conclusion was misplaced priorities. They they call out the focus on non systemic cyber perils was ultimately a catalyst for the CrowdStrike outage.

jerry: And but my read of that is that we’re all focused on things like. Ransomware and worms and whatnot. We weren’t focused on the potential impacts of other components in our technology stack failing catastrophically.[00:46:00]

Andrew: Yeah, I challenge this. I think that’s some Monday morning quarterbacking.

jerry: Keep, again, keep in mind that I don’t think you can. I don’t think you can divorce that one from the next item, which is achieving portfolio diversification. So keep in mind that this was written for insurance companies,

Andrew: That’s fair. Carry on.

jerry: not necessarily for it companies. I, so what they’re pointing out here is that

jerry: there was a concentration of risk. Based on pervasive use of CrowdStrike and pervasive use of windy intersection of pervasive use of CrowdStrike and windows led to, this very systemic outage. And so if you are an insurance company, you should probably be thinking just like how, insurance companies don’t want to have all of their customers in Florida.

jerry: Because that, then if, hurricane comes [00:47:00] through, all of their customers at the same time are making simultaneous claims, they want to have diversified customer base. They want to have some people in Alabama and some people in New York and some people in Ohio and some in Florida. And so carrying that. Into in, into it. And so you don’t, do you want to ensure, do you want to have companies under your insurance umbrella who are only using windows and only using windows and CrowdStrike because now this is pointing out that creates. A situation that is not unlike having all of your customers your homeowners in Florida. It’s been a very one sided.

Andrew: I’m also very curious how these lawsuits are going to go. If they do indeed go, as we have very cynically EULAs that these companies write are not really. setting themselves up for a lot of liability. [00:48:00] And I saw a quote from the CEO of Delta and I won’t get it right, but basically he was saying, Hey, they got to be held accountable and we need to hold them accountable.

Andrew: And my initial thought, having been in a lot of discussions around procurement is that’s probably something your legal team should have done when you sign that contract. What does your legal team agree to and do they really have any actual liability in this circumstance. In my gut, I’ve never read the EULA for CrowdStrike.

Andrew: I’ve never bought CrowdStrike, so I don’t know. But having looked at many other EULAs, they have this limited liability stuff. And hey, yeah, we might screw up. This stuff isn’t perfect. And this is how much we’re liable for. So it will be very interesting, I think, how that might play out in the real world.

jerry: Yeah. I’ve seen snippets of CrowdStrikes EULA and contracts and, one of the, in the immediate aftermath, one of the notable discussion points was that CrowdStrikes. CrowdStrike does not [00:49:00] permit you, the customer to use CrowdStrike in a I’m paraphrasing, but in a in a context of life critical or safety critical uses.

jerry: And I think there were some other similar words. It is not uncommon, by the way, for company software companies To limit the liability based on, how much you paid, like you, you can’t necessarily sue them for damages that exceed the amount that you’ve paid them. That is a fairly common thing.

jerry: Now in the one reality is as as was stated by the CEO of Delta, they are going to go, they are going to go after Delta and it will be sorry after cross tracking, it will be. Interesting to see how that is perceived by the court. I will tell you, you, at least in the U S again, my context is all us law.

jerry: You can’t disclaim away. You [00:50:00] can’t write away in contract gross negligence, right? That is something that if you are found to be gross negligence, it doesn’t matter what you wrote. And, what the customer agreed to in a contract is that there can still be a finding against you.

jerry: And so I think that’s very likely the stance or the angle that Delta will take and probably some other companies as well. I think it’ll be very interesting to see how this plays out because, again, I’m not an attorney. I have worked with quite a few of them over the years. And I’ve seen a lot of a lot of contract cases go to court and be settled out of court.

jerry: I think personally, this is the kind of thing that would get settled out of court because CrowdStrike is not going to want, um, drawn out legal proceeding that is. Really giving them a deep, their IT processes and development processes, a deep [00:51:00] examination. And likewise, Delta probably doesn’t want a deep examination of exactly how they were using CrowdStrike and whether that was in concert with the terms of their license agreement and whatnot, the.

jerry: I think the challenge for CrowdStrike is,

jerry: I, again, I’m not saying this isn’t warranted, but there are a lot of companies right now that are probably in talks with legal counsel and what their options are. And I think, once one happens, this is going to quickly turn into either a class action case, or it’s going to turn into some, some revolving door of litigation.

jerry: And the reality is I think unless CrowdStrike goes to trial with this and they’re found to be not, not having been negligent in their contract term, Limitations are enforceable. They have, they stand a lot to [00:52:00] lose.

Andrew: Yeah, that’s true. The one thing that you touched on that I thought was interesting is the Yula bit of not to be used in critical infrastructure, life, and the one other area that I have any sort of familiarity with is aviation software, which is meant obviously to run aircraft and important functions on aircraft.

Andrew: And that software is incredibly expensive. And very highly regulated because it is meant for that critical use case. So what that sort of tells me is that we have this tension going on here of We want any relatively inexpensive software Almost commodity grade that can run everywhere to solve this problem, but that inexpensive software means that The level of care that goes into very high critical software.

Andrew: I’m not explaining this well, but if you look at what why it costs much money to write aviation software is because the level of diligence is [00:53:00] so high because the consequence of failure is so high, as well as all the regulatory oversight. And that’s where it comes into it. We don’t have that in this case, which is probably why.

Andrew: We have that EULA stipulation of don’t run it in a critical case. But if you’re expecting that level of due care and carefulness in coding, it comes with that price and would corporations be willing to pay that price instead of paying 150 bucks a node or whatever it is for CrowdStrike, are they willing to pay 5, 000 a node?

Andrew: That’s where the rubber meets the road.

jerry: If you look at the case of Delta, obviously they’re an airline and, they’re, they, they have planes that are the software that runs on them are life critical, but I don’t think that’s what, you know,

jerry: The things that failed, I think were the kind of the back office stuff, the planning and the scheduling of crews and and departures and and arrivals and whatnot, things that are not necessarily [00:54:00] important to an airplane staying up in the air, but rather to orchestrating the movement of planes around the world and passengers around the world.

jerry: And. That, that is not by itself, I think, a life critical thing. Although in aggregate, I, there was certainly a story about a person in Florida, an elderly person in Florida who missed his flight as a result of this and has not been seen since.

Andrew: Yeah, and I’m sure that there’s second order. I’m more, what I’m trying to draw a parallel between is, if you’re demanding software with zero defects, it’s much, much more expensive.

jerry: Oh, sure.

Andrew: And that’s where I’m going with this is there’s, whether we like it or not, there are trade offs. And so if you look at one of the extreme of software that runs, and I only say this because this is what I’m familiar with, I’m sure the same for like power plants and software that runs super critical stuff is also super expensive.

Andrew: So [00:55:00] if you want that level of dependability, you probably aren’t going to do it on commodity multifunction. Operating systems and hardware, and you’re probably not going to do it with lower cost tooling like this. So I guess what I’m trying to say is if companies expect zero defect, they’re going to have to pay for it.

Andrew: And we’re not in that scenario right now.

jerry: Temper your expectations, I think is what you’re saying.

Andrew: Yeah. And you can’t have, fast, cheap. Reliable pick one, kind of situation.

jerry: Yeah it’s fair. I think in this instance, this is one of those situations where it’s obvious in hindsight, it’s a simple failure. It’s a stupid failure. It didn’t rely on, highly sophisticated. Quality assurance processes in order to catch it. But it’s one of those things where it’s obvious in hindsight, it’s not necessarily obvious going in.

jerry: And so I, I think broadly speaking, [00:56:00] you’re right. We in it and really civilization and society in general have built this infrastructure around software and systems that, that we beat the shit out of our vendors, pardon my French, to get the lowest prices out of. And then things happen and we’re, we are

Andrew: And generally you might say it’s worked like how much, and

jerry: worked.

Andrew: Agree. Not

jerry: it’s worked spectacularly well.

Andrew: paying

jerry: Like we don’t, how many times do we have this discussion? This is not a common problem on the other side. I’ll just contrast that with every single month. Microsoft, and I think Oracle does it quarterly and, many other companies do too, release a cadre of patches for critical vulnerability.

jerry: So I think to some extent, on the one hand, we’ve built an [00:57:00] ecosystem that is tolerant of a certain level of. Software badness or a lack of quality. But on the other hand, when things like this come along, it highlights how interconnected and interdependent our systems are and that we’re not necessarily, the things that we’re buying are not as resilient as they need to be based.

jerry: And have the expectation that they would never fail,

Andrew: and not on the, Commodity multifunctional hardware and software we’re using.

jerry: All right. It was it was not obvious to hopefully with through magic of editing, not obvious that we just had like catastrophic connectivity failure and whatnot, but I think we’re going to cut it off and we’ll pick up the rest of the story next time.

Andrew: Sounds good.

jerry: I appreciate everybody’s time and attention, and I hope you learned something and, certainly appreciate your your patronage here.

jerry: So you can [00:58:00] follow our podcast here at our website at www. defensivesecurity. org. You can find back episodes just about any podcast service, including now, thanks to Mr. Callit, including YouTube.

Andrew: Audio only for the moment, just an experiment. We’ll see how it goes.

jerry: We’ll be we’ll be experimenting with video, but we both, unfortunately, have to undergo some plastic surgery things before we can

Andrew: About a 12 month intensive workout program with the world’s leading exercise physiologists, I believe as well.

jerry: Yes, but that’s to look forward to the future. You can follow me on social media. I’m primarily on my mastodon instance, Infosec.Exchange. You can follow me at @Jerry@Infosec.Exchange. You can follow Andy where

Andrew: On X @LERG L E R G and infosec.Exchange at L E R G.

jerry: awesome. [00:59:00] And with that, I wish everybody a great week. I hope you have a great weekend ahead and we will talk again very soon. Thank you all.

Andrew: Thanks very much. Bye bye.