CrowdStrike Windows Crashing Issue
The CrowdStrike event began just after midnight on Friday, July 19, 2024, with an errant automatic update. CrowdStrike fixed the update within 90 minutes. Unfortunately, CrowdStrike-installed Windows computers online during those minutes were affected worldwide. Symptoms included a bug check/blue screen error related to the Falcon Sensor. Linux and macOS computers were not affected.
In a Preliminary Post-Incident Review (PIR), CrowdStrike pointed to testing failures and vowed to improve software testing, resiliency, and the release process to avoid a future occurrence. CrowdStrike also promised to publish a root cause analysis when it is complete.
Throughout the outage and recovery, CrowdStrike published detailed and actionable updates in coordination with Microsoft. Carnegie Mellon’s CrowdStrike reps constantly communicated with us, providing live updates and technical support.
Impact on CMU
This issue affected Windows computers with CrowdStrike installed.
Affected by Event:
- 23% of Windows computers were affected.
Not Affected:
- macOS and Linux computers.
- Windows computers not online.
CMU’s Response and Recovery
CMU’s response was swift.
- Computing Services teams were in discussions and recovery within two hours of the faulty update.
- The Computer Crisis Management Team (CCMT) assembled multiple times for situation reports, instructions, and status briefings.
- Nearly half of all impacted computers (42%) were restored before the start of the business day, and only 25% remained by the close of business.
- The Emergency Preparedness and Response Team received an alert at 8:45 a.m.
- A web page detailing the issue and the workaround was published at 9:00 a.m.
- A campus-wide email was sent at 11:15 a.m.
Given the speedy recovery of Computing Services’ computer labs and classroom podium computers, scheduled classes for those spaces were not interrupted.
Next Steps
The Information Security Office (ISO) will continue monitoring CrowdStrike’s improvement efforts and root cause analysis, taking risk-informed actions dependent on their outcomes.
CrowdStrike remains an essential security tool for effectively preventing, detecting, and responding to cyber threats at CMU. The ISO is holding CrowdStrike accountable for addressing the failures in its testing and release processes even as it continues to strongly encourage CrowdStrike adoption.
Workaround for Individual Computers
If you have DSP or local IT support, follow their guidance before following these steps.
Restart the computer to allow it to download the new update.
- If the computer restarts normally, you're done.
- If the computer crashes again, attempt to restart 3-4 more times.
- If that doesn't work, try connecting to the wired network and restart 3-4 more times. A wired connection activates earlier in the boot process than Wi-Fi, increasing the likelihood of downloading the fix.
- If the computer is still crashing, follow CrowdStrike's video OR Microsoft's written instructions to boot into Safe Mode if possible.
If you have any questions or need additional help, please contact the Computing Services Help Center at it-help@cmu.edu or 412-268-4357 (HELP), not CrowdStrike, as listed in the article.