A senior vice president for CrowdStrike today, during testimony before a Congressional committee, said the company is now managing all content updates using the same processes it uses to test code, to prevent a repeat of the outage that impacted more than 8.5 million devices last July.
Adam Meyers, senior vice president for counter adversary operations at CrowdStrike, apologized on behalf of the company before explaining how content updates to it sensor led to the outage. That event was caused by a content validator that, in the absence of a rule not being presented, led to an infamous blue screen of death being rendered on millions of Windows platforms.
In the wake of the outage, CrowdStrike is now treating content updates that reconfigure the sensor to recognize new threats. Historically, the company has been providing 10 to 12 updates per day, with most of those updates being rolled out simultaneously. Now those content updates will be tested in phases before being added to the sensor to ensure there are no similar catastrophic events, said Meyers.
Additionally, customers will have more control over how those content updates are rolled out across their IT environments, he added.
CrowdStrike, however, remains committed to continue to run its sensors at the kernel-level of the operating system, because it provides better performance, increased visibility into cybersecurity threats and ensures the kernel itself has not been tampered with by threat actors.
Congressman Morgan Luttrell (R-TX), however, noted that it’s clearly a flawed process that led to the outage – and that adversaries in multiple foreign countries, in the event of hostilities, are trying to find ways to inflict problems. In effect, the CrowdStrike update is an example of an incident where a technology update resulted in the U.S. essentially “shooting itself in the foot.”
At the same time, Congressman William Timmons (R-SC) noted that it’s estimated the outage resulted in more than $5 billion in damages that CrowdStrike customers have yet to be whole for, because of a “fat finger” mistake. Many of those damages are the subject of ongoing negotiations and potential lawsuits.
It’s not clear how many organizations are replacing CrowdStrike cybersecurity tools as a result of the outage, but it’s apparent that the processes used to update its platforms have been revamped. Each organization will need to determine how much confidence they have in those processes, assuming they are eventually content with whatever efforts that are made to ultimately make them whole. In many cases, those same organizations will need to determine to what degree they might be over reliant on a process that they should have more closely vetted themselves.
Regardless of the ultimate outcome, hard lessons have been learned, especially when it comes to deploying software that runs at the kernel level of any operating system. In the meantime, CrowdStrike rivals that might be inclined to cast proverbial stones may want to determine first to what degree they might be living in a glass house.
