A former co-worker reached out to me last Friday morning, wondering if I had heard about the widespread Windows© outages. I was still drinking my morning coffee and reading through emails, so I was blissfully unaware of the global IT outage triggered by a Crowdstrike© update. It didn’t take me long to find out that the security software behemoth had rolled out a global update that caused millions of Windows-based computers to BSOD and get caught in an endless boot cycle.
This should never have happened in the first place.
IT companies responsible for security software at the scale of Crowdstrike have responsibilities that go beyond threat-protection services. To be worthy of those responsibilities, they must have effective controls in place. Here are three extremely basic measures that Crowdstrike should have in place (or should not have skipped):
Thorough Quality Controls
Quality control is not just a checkbox; it is a comprehensive process that involves rigorous testing and validation at multiple levels. Before any update is rolled out, especially one that affects the kernel of an operating system, it should undergo extensive testing in a controlled environment that simulates various user scenarios. This includes compatibility testing with different hardware configurations, software environments, and usage patterns.
Rolling Updates
If you are pushing an update that affects the actual kernel of an operating system, then it should be done as a ‘rolling update’. By that, I mean the update should not have been pushed out globally to every computer running Crowdstrike. It should have been rolled out to a limited number of computers that have opted in to early release updates. Then, Crowdstrike could have been made aware of the issue before a global release that resulted in the largest IT outage of all time.
Rolling updates help in identifying unforeseen issues in a controlled manner, allowing the development team to address any problems without affecting the entire user base. This phased approach minimizes risk and provides a feedback loop for continuous improvement.
No Big Updates on a Friday
Releasing major updates on a Friday is risky business. Most IT professionals know that rolling out significant changes at the end of the workweek is a recipe for disaster. If something goes wrong, it leaves limited time to address the issues before the weekend, potentially leaving systems down for days. In Crowdstrike’s case, this ill-timed update caused massive disruptions that could have been avoided with better planning.
By addressing these points, we can see that while the outage was a significant failure, it also offers valuable lessons for the future. Companies like Crowdstrike must ensure that they not only provide top-tier security solutions but also maintain the highest standards of operational excellence to avoid such catastrophic events.
