In a recent podcast interview with Cybercrime Magazine's host, David Braue, Scott Schober, Cyber Expert, Author of "Hacked Again," and CEO of Berkeley Varitronics Systems, discusses the July 2024 CrowdStrike outage, explaining what happened and more.
Welcome to The Data Security Podcast sponsored by Cimcor. Cimcor develops innovative, next-generation file integrity monitoring software. The CimTrak Integrity Suite monitors and protects a wide range of physical, network, cloud, and virtual IT assets in real-time while providing detailed forensic information about all changes. Securing your infrastructure with CimTrak helps you get compliant and stay that way. You can find out more about Cimcor and CimTrak on the web at cimcor.com/cimtrak.
David: Scott, thanks for joining me today.
Scott: Wonderful to be here with you, David.
David: Always enjoy our chats. I wanted to hear your thoughts about one of the really quite dramatic things that happened very recently. This is one of those cyber incidents that really had a effect on, well, nearly everyone across the world. It was incredible what happened in July with CrowdStrike. Remind everybody what actually happened here. This was a major outage.s
Scott: This one is one that'll go down in the history books, I think, certainly. And this was a combination primarily CrowdStrike, but also Microsoft. And really, what happened back was about mid-July, CrowdStrike issued an automatic update, and they called it its Falcon Sensor Security Software. And it really was something positive. It was meant to or intended to detect a new attack technique that could really exploit Windows machines. Unfortunately, the update that CrowdStrike pushed out there really had kind of a defect in it, and it was something that was very critical in the Windows operating system, and the result was really just widespread system crashes. Not a whole lot of people focused much on the technical side of things. But it was really, I think, the immediate effect and the aftermath that we're still talking about because it was so significant.
There was estimates about 8.5 million computers were affected worldwide. And when I just think about how that big of an impact could happen, it just blows my mind. Recent estimates are somewhere in the neighborhood of about 10 billion dollars they're now putting it at. I know, early talkers, they were saying, oh, this could be close to a billion dollars damage. Here we are weeks later, the aftermath, and it's 10 times what they initially thought it was. So things are still unfolding. And in particular, it's really the airline industry that took probably the brunt of everything, I think, as far as the costs and the effects, and unfortunately the poor passengers, as a result of it. So pretty amazing. I think we're all still learning from this. To me, I always like to look at these things, not focus on all the negative. But there are some takeaways that everybody can appreciate, hopefully, going forward. It doesn't happen again.
David: So much has happened here that nobody expected. One of the problems, from my understanding, was that this actually made a change at a very low level in the Windows systems, which meant that the computers couldn't be booted. Basically, they couldn't just reboot and reinstall a new patch. They had to be manually updated. Every single computer.
Scott: You're absolutely right. And in fact, some of the media I was watching, and I think it was at airports and stuff, and they were showing people climbing up ladders and doing a physical, hard reset on some of these computers because they couldn't just do a quick, soft patch and solve the problem or do anything remotely, which is commonly done with most of these things, and that hence, added to, I think, the aggravation, because now you need physical manpower in spots that is spread out around the globe. And I have to give them credit. Even though there was a lot of mistakes made. They did respond rather quickly, and I think that's something to their credit, CrowdStrike, that it didn't solve all of the problems that fell out, the reputational damage, and everything else. But I think they did act quickly to restore systems. And they did come forth and apologize. And there's something to be said there. Everybody could be critical and say, How could this have happened? But some people also don't dig in and realize the complexity of this. A lot of people didn't want to hear about the complexity of solving this problem. They just said, "Hey, this is an epic. Fail, fix it! Why did it take so long? Let's go here." So it's understandable on both sides. Now, CrowdStrike themselves did take a hard hit for this. Besides the reputational damage, just looking at the stock price, I think it was about close to 40% down initially, and it rebounded slightly. It's still about 35% down. When you're talking about a company that size, a global leader, that is a tremendous hit that they took, and their customer base.
David: I'm sure the market was very quick to recognize that this would introduce, however, it was resolved in an operational sense, this introduced quite a lot of risk of litigation. And now we're seeing one of the 1st major cases to come out of this. One of the airlines, Delta Airlines, is saying you owe us. It was always going to happen. You mentioned that the airline industry was particularly hard hit. Why was that? And what is Delta actually alleging here?
Scott: Absolutely. Airline industry has always been on the edge of failing, in my belief, because oftentimes, they're using legacy systems. There's not enough manpower. There's not enough cyber efforts behind it. So they're all struggling. And that's part of the fundamental problem. So when they get hit, and usually it's weather, we hear about. A big storm comes through, tornado, hurricane, this, that, and it paralyzes them. Now, here, it's an IT situation. They can't print boarding passes. In a lot of the behind the scenes, it was really rerouting people for the planes, the pilots, and everybody else supporting it. They didn't have enough people and manpower to shift things around, in particular Delta Airlines and the total impact, they said there was more than 50,000 flights that were cancelled or delayed to the airline industry. But Delta alone had about 6,000 flights, and that means about 1.3 million passengers that were affected. Now, when you got 1.3 million passengers affected, that means they're not getting their bags off the planes. They're not getting on planes. They can't get a new flight, some of them, they had to get put up in a hotel and meal vouchers and transportation, and the list goes on and on and on. It was just a terrible thing.
Most of the other airlines were able to respond quicker and had enough manpower, were able to get back online. Delta, in particular, really suffered I think the most, and they're still facing it now. I even heard the other week somebody saying that they had a problem back in July when they booked a flight, and they're still feeling the repercussions. This was last week and had to get a different flight on a different airline, because they're that far behind trying to catch up. So we really appreciate how severely impacted Delta was in particular, and I think now they're allegedly going to be trying to seek damages for about half a billion dollars. And Microsoft quickly responded and say, "Hey, it's not CrowdStrike's problem. It's your problem, your legacy systems, your response, the way you handled it." So there's a little bit of this finger-pointing that's starting to go on now.
David: It's a really curious situation, because, as you mentioned, CrowdStrike came out very early and said, "Oops sorry about that. Our bad. Sorry about all those flights and everything else that we've interrupted around the world. We'll do better in the future." I mean, it was just so obvious what had happened that I guess they couldn't really hide from that. Now that they start to come out and actually come back and respond to something like this from Delta and say, "Well, not our fault. It's your fault." It's a bit of a surprising tactic, not because, of course, you'd think that they're going to try and find any way to reduce the liability. But to say to Delta, "We caused this problem. But it was actually your fault."
Scott: Yeah, I hate these type of situations, because I think, no matter how you slice it and dig in, there's probably some accountability on both sides. Could Delta have done a better job, had more resources, manpower? Did they have a plan? What happens if this IT outage happens in this particular region? How do we respond? So sometimes, planning for the worst allows you to have a plan to respond. Other airlines actually had a better plan in place, and they were able to stick to it and respond quickly. And there was some stats. I saw out there that CrowdStrike claimed they had 99% of the computers back online in a very short period of time, which is pretty astounding if that really could happen. That was the 19th of July, they claim. So that morning, everything failed. And they said, by 8 PM Eastern time, that evening, 99% of the affected Windows sensors were back online. Now that being said, customers were still feeling the aftermath for weeks. So they did respond quickly, doesn't solve the problems, I think. And that's the really fundamental part. And I think we're going to be hearing about this in litigations for years to come, especially with Delta and CrowdStrike
David: Most definitely. And there's going to be a lot of detail that comes out about who did what when. One of the really interesting aspects that we're hearing about this Delta case is apparently Microsoft and CrowdStrike actually approached Delta and said, Look, we see that this is a big problem for you. Can we help? And they did it on the day it happened, the 19th they did it on the 20th, they came back on the 21st, 22nd, and even the 23rd of July. So for 5 days in a row, they were offering their assistance to Delta to try and get them back online as quickly as possible. And what did Delta say?
Scott: Yeah, I think it kind of went to deaf ears, unfortunately. And maybe that's part of the fundamental problem. We may not know all the conversations that are going on in the background, but when I look at a major cyber breach, and when we've talked about many of them, oftentimes the company that suffers, it really doesn't fully know what extent it's going on. But when the Federal Government steps in, when cyber security experts step in, such as CrowdStrike, they know how to handle these things. This is how they operate. They're ready to be in emergency mode day in, day out. So when you have that offer on the table, you jump at it because that could mean the difference between success and failure. And I think maybe there was a little bit of proudness there in Delta, and maybe now they're a little embarrassed. Looking back, hindsight, saying, "Yeah, we probably should have listened and jumped at the opportunity. The 1st or second or 3rd time they offered, because it may not be as bad as it is now."
David: It's definitely going to come out in the wash, I suppose you know who said what, when and what offers were made? I guess Delta could say, well, "You guys already caused enough problems. We really don't want you going through our systems and supposedly fixing it." They could use that defense, perhaps. One of the issues that you touched on, and that seems to be one of the other problems they had, is that Delta is apparently running pretty antiquated IT. Where do you think that could have really become a problem compared to what some of the other airlines are doing?
Scott: That's fundamental. And it's true, not just in the airline industry, but many industries, especially throughout the United States critical infrastructure. Lots of legacy systems. The fundamental problem is the sheer cost to update that. Not many are willing to spend the money in the right areas. And that's true of the airline industry, and they've been slow to the party. Some of the competitive arrival airlines have spent more money upgrading some of the legacy systems, and I think they did better. They fared better in this. And it's it's a good message for the airline industry. But really, industries as a whole that are listening to this. If you take your time and do your research and spend the right money in the right areas, it will pay dividends down the road. It's an investment in your success. Sometimes it could be shocking, the cost. And when we're talking airline industry, the larger the airline industry. The more money, the more the spend is upfront. But once they put that behind them, it really is like an insurance policy because these things are going to continue to happen, hopefully not to this extent. But there will be cyber breaches. There'll be hacking. There'll be disruptions. There'll be the normal storms and other things the airline industries face, and having modern mechanisms in place to respond to this and the ability to hopefully patch this stuff quicker. And I think when you add in there the AI and the machine learning and other things, there should be provisions in there to check some of this stuff. I think that maybe more fault should be placed a little bit on CrowdStrike's side is, I sit back, and again, we're running a hardware company, but we understand software. When the 1st person puts their thumbs up and said, "Hey, we're ready to release this." Then I hand it to 2 other engineers and say, "Let's check it really careful and make sure we got this 100%. Let's exercise every possible variable that we can to make sure we don't cause a problem for our customer base." Perhaps CrowdStrike got a little complacent here and did not check things carefully enough to see about all the possible outcomes and iterations which obviously we saw manifest themselves and cause a terrible problem here.
We'll be right back after a quick word from our sponsor.
Cimcor develops innovative, next-generation file integrity monitoring software. The CimTrak Integrity Suite monitors and protects a wide range of physical, network, cloud, and virtual IT assets in real-time while providing detailed forensic information about all changes. Securing your infrastructure with CimTrak helps you get compliant and stay that way. You can find out more about Cimcor and CimTrak on the web at cimcor.com/cimtrak. That's C-I-M-C-O-R.com slash C-I-M-T-R-A-K.
And now, back to the podcast.
David: Well, this is an issue that any company that's writing software or making hardware has to deal with. I mean, it's really complicated development process. There's so many moving parts. You try to introduce new features, you know you push here and pull there, and who knows what's going to happen? And this is probably a worst-case scenario of that, not only because it happened, but because it couldn't be easily repaired without manual intervention. Normally you could just push a new update and overwrite the previous one, and you'd be fine. But that didn't work here. There really are some lessons, not only for the companies that need to manage their exposure to stuff like this but also for software providers that are trying to do this and push out updates as quickly as new security problems are being discovered, and that's always a race.
Scott: Absolutely. One article I was looking at and it kind of gave me a little chuckle, and it was talking about some of the crisis management response, you know, since CrowdStrike had mixed reviews, how they handled it, depends upon which perspective you are. But their President received an award, and they granted him at a show. It was called The Most Epic Fail at the recent industry conference, and he said, he's going to keep it and display it at the company's headquarters as a reminder of the incident. So at 1st I chuckled at that, but then I said, you know what? That's showing a bit of humility. It shows, "Hey, guys, we messed up. Let's always point to this. So we never do it again." And I think that's brave of them to do that. I always commend people when they're willing to at least admit mistakes and use it as an opportunity to learn from, and especially in the world of cyber security, that is paramount.
We learn so much from looking back at mistakes that industry leaders have done, cyber security companies, the government, consumers, you name it. We all make mistakes. Imperfection, I think, helps us improve when we reflect upon it, and strive to do better.
David: Hear hear. Particularly in cases like this, I mean this in terms of own goals in the technology world, they probably don't come much bigger than this. So he really moved quickly, said from the beginning, it's our problem, this is it. They weren't hacked. This was not an attack on the airlines, and I think there's probably a lot of recognition that he needed to settle everyone's nerves because when something this big happens, people would think something else is going on. What could it be? So he came out he said that the award you mentioned, he said. It's important that I'm there. I need to be there and own this and really get the company to take this as a lesson learned.
So that's clearly from CrowdStrike's perspective. They've done that. Microsoft has been intimately involved in this sort of response to what's happened here. But ultimately, when it comes down to it. And you hinted at this earlier, this is an industry where they have very extensive disaster response plans. They plan for everything. It's airlines. This is complicated stuff, whether in terms of just interruptions to the flight physically from weather, whether there's scheduling issues, whether there's congestion at an airport, and they have to reschedule everything. Whatever happens in the industry, it's incredible that they didn't have a better plan in place, and I suspect that if nothing else, hopefully, everybody in the airline industry and other really mission-critical industries like that will realize that they have to pull this stuff out of the closet, dust it off, and really make sure that they could deal with something like this.
Scott: Yeah, I was reflecting upon this and think that maybe everybody should step back from something like this, or also think about it. Are we too dependent upon a couple of companies? In other words, is there a monopoly? Is CrowdStrike, the cyber security go-to company in the world? Well, they were, maybe not now. But maybe when you back up and pause and reflect upon this. Maybe not. Putting all your eggs in one basket does help large companies, such as Delta or other airlines or other companies that were really affected by this, maybe spreading it around might minimize some of our exposure. Can't put away, you know the possible litigation, the financial repercussions, and all of that. That happened, but going forward, I think it's going to be a long road. But people should think about that. Not just saying, let me just invest everything in one company. It's much easier. But sometimes that makes difficulty for you to respond to an incident like this, and I think it also opened the door to some of the possible ways you handle this. The importance of transparency we saw. There's got to be a level of accountability for things like this, and having an effective communication plan when these things unfold is really important for maintaining your reputation and solving these problems going forward.
David: Absolutely no question there. And really, I think anybody who's managing risk in a company would be aware of the need for this, even if it maybe hasn't been implemented just as well there. It's an interesting development because, as you talk about CrowdStrike being so widely used, I mean, it has been a big success story. I interviewed George Kurtz, whose the CEO of CrowdStrike, probably 5-6 years ago, when they were emerging and growing very quickly, he was very excited about the application of AI to security, and some of the potential that was there. It was all growth and possibility. Really, this incident, I think, is a coming of age for the company. The way that he's managed it, we've talked about, certainly sets some examples for people, but even in terms of the way that customers really wrap their security innovations, and, you know, adopt this new technology, but wrap it into the risk management frameworks that they need to be maintaining to make sure that the business isn't left too exposed. As you said, to this dependency on the technology.
Scott: Yeah. And I think usually somebody has to be the fall guy, unfortunately. And I think he probably took the brunt of it if you had to pick anybody. But, and I had the privilege of talking to George Kurtz a number of years ago. I think it was back when he was a McAfee, if I recall, and he was a CTO back then. But he's brilliant. He knows his stuff. He's a great guy, good business leader. These things happen. I think it's more important. How do you respond to it? And I think he's doing the best he can. I mean, put a video out right away, a public statement. He's doing the things you're supposed to do as a leader of an organization. I think it's safe to say, though probably no other leader in the world has ever faced such a massive debacle as this. Very few stand out, and I think, considering all things, you got to give him credit for doing the right thing. He didn't put his head in the sand. He tried his best from an apology standpoint, from a technical standpoint, offering resources. I think he, even he appeared on The Today Show. I didn't see that clip, but he did what he could to get out there to let the public know, "Hey, we messed up. We're partly to blame. We're here to help. Let's get this solved and roll up our sleeves and work together." Got to give somebody credit for that.
David: Definitely. Any other lessons that we can take from this incident, do you think? About the way it was managed? The dangers of technical debt which has certainly been writ large, I think, through all of this, or any ideas about where it's going to go from here? Surely, Delta won't be the last company that's raising this.
Scott: I think, for in particular, the airline industry. Since they were hit the hardest, they continue to suffer. The governments have, in a sense, kind of helped them bail out from their mistakes, I would say in the past. I think they need a little bit of, unfortunately, regulation and oversight to come in and force them to do things I hate when government is stepping in to solve problems because they inherently create more problems sometimes with regulation.
However, I think the airlines continue to change the way we travel, and we suffer as passengers, and I think we've all missed flights and been disappointed when a plane's grounded, or maintenance issues or doors fall off, or whatever the case may be, and I think a little more oversight and regulation needs to kind of come into place, and this might be likened to other major things that have happened. I think back on some of the stuff like Boeing, the 737 Max defect. That was a huge corporate crisis. You look at Chipotle had a terrible food poisoning scare. All of these things are bumps in the road. Companies will recover. However, it's going to take a long time, it's going to be expensive, and there's going to be challenges, and that makes a company sometimes stronger. So I'm hoping Delta comes out as a stronger company. CrowdStrike comes out as a stronger company, Microsoft, all these companies. There's a lot of lessons that they learned, and then we, too, as consumers and business owners, have learned a lot as well. Don't put all your eggs in one basket, but hope this calms down because the damage is just staggering. It's really 10 billion dollars. That's disappointing.
David: Absolutely, absolutely. And maybe in all this introspection and overhauling, they can do something about the food as well.
Scott: Yes, that'll never change
David: If we're asking too much.
Scott: Yeah, now we're pushing it.
David: Scott, thanks again for joining me today. It was a pleasure to talk as always.
Scott: Wonderful to be with you, David
