See it and Sell it First at ASI Show Orlando – January 4-6, 2025.   Register Now.

Last Updated: July 22, 2024

Promo Contends With Fallout From CrowdStrike IT Outage

From disrupted travel and banking to problems with tech systems, merch industry firms were caught up in some of the global chaos resulting from impacts to Microsoft Windows caused by cybersecurity firm CrowdStrike.

By  Christopher Ruvo

UPDATE Monday Morning July 22, 2024

In a a blog post, CrowdStrike warned that hackers could be trying to exploit the outage, sending a malicious file to CrowdStrike customers that purports to be a “quick fix” to the problem. The file, “crowdstrike-hotfix.zip,” contains malware that, if activated, allows hackers to control or monitor a user’s computer. The original outage was not caused by a cyberattack but a faulty update/file channel.

ORIGINAL STORY

What’s been described as both a botched update or faulty file channel from cybersecurity firm CrowdStrike has affected Microsoft and triggered IT outages around the world, temporarily hobbling the operations of banks, airports, media, healthcare providers and other businesses across industries, while causing challenges for branded merchandise companies that include Counselor Top 40 firms.

“Any company using CrowdStrike, including those in the promotional products industry, is potentially impacted,” said Seth Kusiak, chief information security officer at ASI, which doesn’t use CrowdStrike as part of its IT stack.

computer login screen

CrowdStrike has said that half of the Fortune 500 companies in the U.S. use its software.

Issues resulting from the outages on Friday, July 19, included waves of canceled flights across the globe. Banks and financial service companies were experiencing disruptions, and stock traders were having trouble executing transactions. In the United Kingdom, booking systems used by doctors were down, and major media outlet Sky News couldn’t broadcast for a time. Supply chain snags, including several terminals being down at the Port of Houston, were also reportedly among the challenges.

Adding to the woe was the fact that Microsoft 365 services like Outlook, Teams, SharePoint and OneDrive were experiencing reported widespread disruptions due to a separate issue with a backend Azure configuration change. 

“This is a very, very uncomfortable illustration of the fragility of the world’s core Internet infrastructure,” said Ciaran Martin, professor at Oxford University's Blavatnik School of Government and former head of the UK National Cyber Security Centre.

Headaches for Promo

Promo pros were getting caught up in the mess, sharing in discussion with ASI Media and online forums that they were dealing with canceled flights, airport delays and an inability to access their bank accounts.

“I’ve been at LAX for 7 hours,” wrote one industry pro on Facebook. “Have boarded and deplaned twice, then canceled. No retail stores can process credit transactions. All ATMs are down. It’s a fun night.”

Some promo firms confirmed they were directly contending with issues from the CrowdStrike trouble.

“We, like many other users of CrowdStrike, were affected by their global outage,” Phil Gergen, chief information at Counselor Top 40 supplier Koozie Group  (asi/40480), told ASI Media. “Many of our core systems do not run on Windows, so we were able to recover quickly. At times like these, having a nimble, talented IT team and solid IT infrastructure is crucial, and the quick work done by our team kept our business operating as normal, and we do not expect any customer impact from this outage.”

Counselor Top 40 firm SanMar (asi/84863), promo’s largest supplier, was dealing with fallout, too. “We’re having some issues with our time clock for hourly employees to clock in and we do understand UPS, who’s our largest carrier, is somewhat affected,” Jeremy Lott, CEO/president of SanMar and a member of Counselor’s Power 50 list of promo’s influential people, told ASI Media. "That said, none of the core operations at SanMar are affected and we are operating normally."

To Lott’s point on deliveries: UPS and FedEx were both impacted. Delays in service are expected. UPS said its drivers are out on the roads. “We are continuing to work to resolve all issues as quickly as possible; there may be some service delays,” UPS said in a statement. FedEx said it’s working contingency plans, but noted that there are “potential delays” for package deliveries due Friday. Promo suppliers and distributors should be communicating openly with clients on time sensitive orders. 

Other promo executives who spoke of issues wished to remain anonymous. A company leader, for instance, told ASI Media that one of its ancillary systems had been affected, but the issue was being rectified and the system would be operational again quickly.

Meanwhile, an executive with Counselor Top 40 supplier alphabroder (asi/343063) said: “We are managing through some issues – but overall our order processing and fulfillment/shipping operations are running smoothly. We are not anticipating any material customer or business interruption.”

Promo executives ASI Media spoke with also acknowledged that clients/vendors being directly affected or challenges with banking could temporarily put a damper on sales but felt at this point that the issues wouldn’t be prolonged or pronounced.

“As long as the outage is fixed relatively quickly, I don’t think it will have a major impact on business,” Mike Wolfe, CEO of Counselor Top 40 distributor Zorch (asi/366078), told ASI Media. Zorch is not a CrowdStrike customer.

Other promo firms like Zorch that were also not directly struck by the CrowdStrike/Microsoft problems said they were nonetheless keeping a close eye on the situation.

“Fortunately, we do not use CrowdStrike as an application, and all of our systems are up and running and not affected,” Jo-an Lantz, CEO/president of Counselor Top 40 distributor Geiger (asi/202900) and a Counselor Power 50 member, told ASI Media. “Our U.K. and European teams were notified shortly after the outage that we continued to be operational. Likewise, U.S. and Philippines teams were notified first thing this morning that all systems are working. We are monitoring this closely.”

What Happened?

CrowdStrike unintentionally caused the chaos when it issued what a statement described as a defective update to its Falcon antivirus software, which works to protect Microsoft Windows devices from cyberattacks. One company employee later clarified it was a faulty file channel rather than a full update.

Regardless, the problem was triggering Microsoft Windows applications to crash and computers to show a blue screen – colloquially known as the “Blue Screen of Death.” In other words, companies and their people couldn’t use their computers.

The problems weren’t the consequence of a cyberattack, CrowdStrike stressed. Mac and Linux products aren’t affected. The problem was reportedly being rectified, but that wasn’t enough to prevent Friday’s IT carnage.

“The issue has been identified, isolated and a fix has been deployed,” said CrowdStrike CEO George Kurtz. “We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website. … Our team is fully mobilized to ensure the security and stability of CrowdStrike customers.”

While the situation was evolving quickly, CrowdStrike at least initially was suggesting what IT pros described as a manual and time-consuming process for computers to get back up and running. “I would not wish this on any organization,” said one leading IT professional in the promo products space.

One potential workaround, according to a CrowdStrike worker, was to: 1. Boot Windows into Safe Mode or Windows Recovery Environment; 2. Go to C:\Windows\System32\drivers\CrowdStrike; 3. Locate and delete file matching “C-00000291*.sys;” 4. Boot normally.

“That workaround won’t help everyone though and I’ve no further actionable help to provide at the minute,” the employee said on X, formerly Twitter.

CNBC reported that Troy Hunt, a cybersecurity researcher behind the password-breach monitoring service HaveIBeenPwned, called the ongoing global tech disruption as “the largest IT outage in history.”