Urgent - CrowdStrike Outage

 Tech Alert | Windows crashes related to Falcon Sensor | 2024-07-19

printFavorite

Cloud:  

US-1EU-1US-2

Published Date: Jul 19, 2024

Summary

• CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.

Details

• Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.

• Windows hosts which have not been impacted do not require any action as the problematic channel file has been reverted.

• Windows hosts which are brought online after 0527 UTC will also not be impacted

• This issue is not impacting Mac- or Linux-based hosts

• Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.

• Channel file "C-00000291*.sys" with timestamp of 0409 UTC is the problematic version.

Current Action

• CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.

• If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:

Workaround Steps for individual hosts:

• Reboot the host to give it an opportunity to download the reverted channel file.  If the host crashes again, then:

  • Boot Windows into Safe Mode or the Windows Recovery Environment

  • Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory

  • Locate the file matching “C-00000291*.sys”, and delete it.

  • Boot the host normally.

Note:  Bitlocker-encrypted hosts may require a recovery key.

Workaround Steps for public cloud or similar environment including virtual:

Option 1:

• Detach the operating system disk volume from the impacted virtual server

• Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes

• Attach/mount the volume to to a new virtual server

• Navigate to the %WINDIR%\\System32\drivers\CrowdStrike directory

• Locate the file matching “C-00000291*.sys”, and delete it.

• Detach the volume from the new virtual server

• Reattach the fixed volume to the impacted virtual server

Option 2:

• Roll back to a snapshot before 0409 UTC. 

Workaround Steps for Azure via serial

1. Login to Azure console --> Go to Virtual Machines  --> Select the VM

2. Upper left on console --> Click : "Connect" --> Click --> Connect --> Click "More ways to Connect"  --> Click : "Serial Console"

3. Step 3 : Once SAC has loaded, type in 'cmd' and press enter.

   1. type in 'cmd' command

   2. type in : ch -si 1

4. Press any key (space bar).  Enter Administrator credentials

5. Type the following:

   1. bcdedit /set {current} safeboot minimal

   2. bcdedit /set {current} safeboot network

6. Restart VM

7. Optional: How to confirm the boot state? Run command:

   • wmic COMPUTERSYSTEM GET BootupState

To access Safe Mode, press the Function and F4 keys at the same time during the start up:

 Lenovo Safe Mode Instructions:

https://support.lenovo.com/us/en/solutions/ht116905-how-to-enter-or-boot-to-safe-mode-in-windows-7-8-81-and-10

Dell Safe Mode Instructions:

https://www.dell.com/support/kbdoc/en-us/000124344/how-to-boot-to-safe-mode-in-windows-10

Latest Updates

• 2024-07-19 05:30 AM UTC | Tech Alert Published.

• 2024-07-19 06:30 AM UTC | Updated and added workaround details.

• 2024-07-19 08:08 AM UTC | Updated

• 2024-07-19 XXXX AM UTC | Updated