By Martin Jansen, Owner of Jansen-PCINFO
Almost two weeks ago CrowdStrike, a cybersecurity firm widely used by many corporations, sent out a disastrous update to their Falcon software which sent all affected computers into a boot loop. Windows on the affected computers no longer booted to a usable desktop.
I don’t like to comment on these types of outages until all the facts are known. In this case there is plenty of blame for the outages to go around.
Not Affected
At this point I think it is important to note which computers were not affected by the outage:
- Home computers don’t use CrowdStrike software, it is made for business
- Linux and Mac computers were not affected due to a different architecture
The above points are only to say which computers were not affected, not that people in general were not affected. Everyone who made any digital transactions at the time were and still are being affected by the outage. Trillions of dollars are being lost due to this one error.
Apparent and Hidden Effects
Airlines and transportation around the world was immediately affected by the outage. Long lines formed as would be passengers had their tickets, but could not be validated. Suddenly shipping companies could no longer deliver packages effectively. Lesser known are the companies that had to send employees home because they could not work. Long term effects and inconveniences of the outage are yet to be known.
One File
It’s hard to believe, but it is one critical .SYS file that is the root of all the problems. SYS files contain configuration and setting information for the Windows operating system. Somehow, software engineers at CrowdStrike didn’t go through proper debugging checks before releasing the software update to the masses. It’s even possible that a wrong untested software update was sent instead of the true update. In this case it was a driver .sys file that was sent putting Windows into a boot loop.
Remote Fix Not Possible
Millions of computers are worked on remotely by computer technicians daily. But to access the systems the technicians must have a fully working Windows operating system. With Windows in a boot loop, a remote fix is not possible. Instead, each affected computer had to be visited by a technician with a USB Stick and recovery instructions in hand. Too many computers and too little technicians make this a time consuming process. Complicating this further, many business computers are encrypted with Bitlocker. If encrypted, a special code has to be entered to access the files on the operating system.
A CrowdStrike Fix
CrowdStrike was quick to push an online update, but it was ineffective, again because Windows was not fully working and the network card had to be active with Windows to receive the update. Some remediation instructions seemed on the hopeful side that constantly rebooting the computer would somehow allow the update to replace the errant .sys file.
Microsoft To Blame?
Microsoft wants to appear to be the good guy in this misadventure, but pundits point out that the Windows operating system is to blame for weak security features. Microsoft wanted to reserve kernel access for themselves while denying access to third party software developers like CrowdStrike. An European court rendered a verdict to level the playing field allowing access for all developers. Microsoft does have a long history of security lapses, this being the latest failure.
Conclusion
CrowdStrike certainly caused an uproar when it released an update to its Falcon software on July 19th. The introduction of one bad file to the Windows operating system sent the computers into a boot loop. Fast remediation was impossible because remote access required a fully working Windows, instead each computer had to be visited by technicians. Microsoft may also be implicated in this technological disaster. CrowdStrike is now considering releasing updates to computers in small batches, seeing if there are any negative effects, before releasing them to all computers.