Running Digital Signage Networks On Windows? Pour A Three-Finger Drink Instead Of Morning Coffee Before Reading This
July 19, 2024 by Dave Haynes
People on the America side of the Atlantic are sipping coffee and taking in news this morning that enterprise devices running Windows on networks – like digital signage media players using that OS – were blue-screened if they also were using security software from CrowdStrike
It is not a Windows error directly, but one caused by cybersecurity software developed by CrowdStrike. Ironically, software intended to keep enterprise-level networks secure and stable instead caused the much-dreaded blue screen of death, and has caused global disruptions that include airlines, banks, rail systems, broadcast networks and even health care systems.
This story is and will be a moving target, and things may well change, but one of the fixes recommended involved visiting each affected device and rebooting it. That’s a big job around a large office, but it is a nightmare, if that’s the solution, for networks that have devices dispersed all over the place – as digital signage and digital out of home networks often do
CrowdStrike CEO George Kurtz, who is having the mother of all bad days, says: “CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed.”
“We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website. We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels. Our team is fully mobilized to ensure the security and stability of CrowdStrike customers.”
Compounding things was news, also overnight, that Windows OS developer Microsoft also had a lengthy outage of its Azure cloud computing services.
“A backend cluster management workflow deployed a configuration change causing backend access to be blocked between a subset of Azure Storage clusters and compute resources in the Central US region,” the company says in an update. “This resulted in the compute resources automatically restarting when connectivity was lost to virtual disks. Mitigation has been confirmed for all Azure Storage clusters, the majority of services are now recovered. A small subset of services is still experiencing residual impact. Impacted customers will be continuing to communicate through the Azure service health portal.”
Windows is no longer the default operating system used by digital signage networks, and in many respects is now a specialty choice. A lot of networks run on Linux, Android, ChromeOS and proprietary operating systems like Samsung’s Tizen OS and LG’s webOS. There are also, now, several software and hardware firms with their own custom operating systems – like the one developed by SpinetiX, and BrightSign’s OS. TelemetryTV just announced one this week.
It will likely be a long, rough day (or days) for digital signage network operators running Windows that also had CrowdStrike software on them. I HOPE the fix is at least remote and able to be done as a batch, not one by one. If the fix really is touching each machine in person, the costs for, let’s say, a C-store network with hundreds or 1,000s of locations would be crippling. Imagine how a field tech visit, at $200/hour, would add up across a widely dispersed network.
If you are a Windows shop, you are probably too busy to let me know what’s happening. I can think of a few Windows-centric CMS shops out there and will reach out.
If you run something like Google’s ChromeOS, it’s an unmissable opportunity to communicate how this sort of thing wouldn’t happen. One user told me by e-mail he was able to easily reassure customers they were unaffected: “The nature of ChromeOS and its verified boot system simply wouldn’t allow anything like this issue to ever occur. Any changes to the root OS, which are not verified, triggers a power-wash, and a safe reboot.”
All flights grounded nationwide? After 2 hrs on the tarmac we had to deplane, this red eye flight just got a lot longer with no end in sight. #Crowdstrike pic.twitter.com/IJnTSI0J4Z
— Scott Sanders (@SeahawkSanders) July 19, 2024
Here is what Microsoft says and suggests, at least at the moment:
We have received reports of successful recovery from some customers attempting multiple Virtual Machine restart operations on affected Virtual Machines. Customers can attempt to do so as follows:
- Using the Azure Portal – attempting ‘Restart’ on affected VMs
- Using the Azure CLI or Azure Shell (https://shell.azure.com)
We’ve received feedback from customers that several reboots (as many as 15 have been reported) may be required, but overall feedback is that reboots are an effective troubleshooting step at this stage.
Additional options for recovery:
We recommend customers that are able to, to restore from a backup from before 19:00 UTC on the 18th of July.
- Customers leveraging Azure Backup can follow the following instructions:
How to restore Azure VM data in Azure portal
- Alternatively, customers can attempt to repair the OS disk offline by following these instructions:
Attach an unmanaged disk to a VM for offline repair
- Disks that are encrypted may need these additional instructions:
Unlocking an encrypted disk for offline repair
Once the disk is attached, customers can attempt to delete the following file.
Windows/System/System32/Drivers/CrowdStrike/C00000291*.sys
The disk can then be attached and re-attached to the original VM.
We can confirm the affected update has been pulled by CrowdStrike. Customers that are continuing to experience issues should reach out to CrowdStrike for additional assistance.
Additionally, we’re continuing to investigate additional mitigation options for customers and will share more information as it becomes known.
Boy am I glad I retired!
And all those highly paid windows certified techs were powerless
Running Windows IoT cuts down on most big windows update issues.
If you have the software on ALL your hardware clients automatically updating, with no way for you to approve, monitor, or pause them, then this *will* eventually happen to your network. It may not be a blue screen, but no software is immune from a bad update going out.
Don’t trust anyone else to manage YOUR computers or their updates!