Global Compu-Trouble

[Lawrence D'Oliveiro]

I thought this world-wide trouble might be due to some widely-used
cloud service, but it appears the common factor may be something else,
namely their dependence on a security service called “CrowdStrike”.

<https://www.nzherald.co.nz/nz/bank-problems-reports-bnz-asb-kiwibank-anz-visa-paywave-services-down/R2EY42QKQBALXNF33G5PA6U3TQ/>

[Computer Nerd Kev]

Post by Lawrence D'Oliveiro
I thought this world-wide trouble might be due to some widely-used
cloud service, but it appears the common factor may be something else,
namely their dependence on a security service called ?CrowdStrike?.

Yes, though it has in turn taken many VMs in Microsoft's Azure
cloud service offline, and amusingly Microsoft's official advice to
affected users is to turn their VMs off and on again, repeatedly:

https://www.theregister.com/2024/07/19/azure_vms_ruined_by_crowdstrike/

[candycanearter07]

Post by Computer Nerd Kev

Post by Lawrence D'Oliveiro
I thought this world-wide trouble might be due to some widely-used
cloud service, but it appears the common factor may be something else,
namely their dependence on a security service called ?CrowdStrike?.

Yes, though it has in turn taken many VMs in Microsoft's Azure
cloud service offline, and amusingly Microsoft's official advice to
https://www.theregister.com/2024/07/19/azure_vms_ruined_by_crowdstrike/

So what was actually affected?

[Scott Alfter]

Post by candycanearter07
So what was actually affected?

I first heard about it here:

https://acecomments.mu.nu/?post=410680

which in turn contains these links:

https://www.abc.net.au/news/2024-07-19/global-it-outage-crowdstrike-microsoft-banks-airlines-australia/104119960
https://x.com/_JohnHammond/status/1814178288220479565
https://x.com/disclosetv/status/1814192537348833699
https://x.com/senadaruc/status/1814182862146429367

Where I work, the cloudy time-and-attendance software we use (which lives on
Azure IIRC) is affected, so people can't clock in and out and HR and
accounting can't do their usual things within it.

[mm0fmf]

Post by candycanearter07

Post by Computer Nerd Kev

Post by Lawrence D'Oliveiro
I thought this world-wide trouble might be due to some widely-used
cloud service, but it appears the common factor may be something else,
namely their dependence on a security service called ?CrowdStrike?.

Yes, though it has in turn taken many VMs in Microsoft's Azure
cloud service offline, and amusingly Microsoft's official advice to
https://www.theregister.com/2024/07/19/azure_vms_ruined_by_crowdstrike/

So what was actually affected?

Crowdstrike is an endpoint monitoring system that is meant to detect bad
stuff on your computers and stop intrusions and shit. We have it on
Windows laptops and Windows servers. Never seems to do anything apart
from once when it decided the software my employer pays me to write was
a threat when I was testing it ;-)

There was an update today that like many security updates, can get
rolled out automatically. My employers don't do this... they do do some
testing of all updates. The update caused Windows (maybe just Win10)
machines to BSOD when booting.

There is a manual fix, do a safe boot (doesn't load lots of extra
drivers and utilities), go and find the erroneous file in one of
Windows' folders and delete it. Reboot, job done. Repeat on all affected
machines.

My place has about 17000 Windows laptops and servers (and almost as much
Linux stuff) so I think IT will be quite pleased they didn't push the
update out automatically.

[BungleBob]

Post by mm0fmf

Post by candycanearter07

Post by Computer Nerd Kev

Post by Lawrence D'Oliveiro
I thought this world-wide trouble might be due to some widely-used
cloud service, but it appears the common factor may be something else,
namely their dependence on a security service called ?CrowdStrike?.

Yes, though it has in turn taken many VMs in Microsoft's Azure
cloud service offline, and amusingly Microsoft's official advice to
https://www.theregister.com/2024/07/19/azure_vms_ruined_by_crowdstrike/

So what was actually affected?

Crowdstrike is an endpoint monitoring system that is meant to detect
bad stuff on your computers and stop intrusions and shit. We have it on
Windows laptops and Windows servers. Never seems to do anything apart
from once when it decided the software my employer pays me to write was
a threat when I was testing it ;-)
There was an update today that like many security updates, can get
rolled out automatically. My employers don't do this... they do do some
testing of all updates. The update caused Windows (maybe just Win10)
machines to BSOD when booting.
There is a manual fix, do a safe boot (doesn't load lots of extra
drivers and utilities), go and find the erroneous file in one of
Windows' folders and delete it. Reboot, job done. Repeat on all
affected machines.
My place has about 17000 Windows laptops and servers (and almost as
much Linux stuff) so I think IT will be quite pleased they didn't push
the update out automatically.

An issue caused by a combination of two or three IT stupidities:

1. Using and relying on silly "cloud" services for everything.

2. Using and relying on Microsloth Windoze.
(Linux and MacOS computers were not affected)

3. Having your computer system / software set to automatically
install updates.


The only way to fix it is to manually boot into Safe Mode and delete /
rename the updated file from computers. A long and labourious process
for many IT departments and a lot of money for businesses. (No doubt
Crowdstrike is going to be facing a lot of lawsuits, especially in
America, land of the "sue 'em all" mentality and little taking
responsibility for your own decisions!)

[Computer Nerd Kev]

Post by BungleBob
The only way to fix it is to manually boot into Safe Mode and delete /
rename the updated file from computers.

It seems the anti-virus software checks for updates when it starts
up, so if the internet connection is available early enough after
booting it may fix itself automatically before it has a chance to
break the system:

https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/

[BungleBob]

Post by Computer Nerd Kev

Post by BungleBob
The only way to fix it is to manually boot into Safe Mode and delete /
rename the updated file from computers.

It seems the anti-virus software checks for updates when it starts
up, so if the internet connection is available early enough after
booting it may fix itself automatically before it has a chance to
https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/

Possibly, *if* a lot of things happen perfectly, but even that link
still says to do it manually if the computer keeps crashing.

[John McCue]

trimmed followups to comp.misc
<snip>

Post by BungleBob
1. Using and relying on silly "cloud" services for everything.

We are going back to pre 199x days.

Post by BungleBob
2. Using and relying on Microsloth Windoze.
(Linux and MacOS computers were not affected)

Same is true for the BSDs :)

Post by BungleBob
3. Having your computer system / software set to automatically
install updates.

Yes, I was surprised this was still a thing in Windows.
I would think people should be prompted say once per day,
after 5 declines, the update(s) are forced in.

<snip>

[BungleBob]

Now Crowdstrike are offering their client companies a bribe ...
insanely it is a measley US$10 Uber Eats gift card!!
<https://futurism.com/the-byte/crowdstrike-10-gift-card-apology>

Crowdstrike is a company that is about to be made bankrupt by numerous
court claims for millions, if not billions, of dollars in compensation.
(US$5 billion is esitmated to have been lost by the US Fortune 500
companies alone.)

[Lawrence D'Oliveiro]

Post by mm0fmf
My place has about 17000 Windows laptops and servers (and almost as much
Linux stuff) ...

The problem was only on the Windows machines.

[mm0fmf]

Post by Lawrence D'Oliveiro

Post by mm0fmf
My place has about 17000 Windows laptops and servers (and almost as much
Linux stuff) ...

The problem was only on the Windows machines.

I am aware. The comment was to give a scale of the number of machines
and hence likely size of IT dept.

[Lawrence D'Oliveiro]

Post by mm0fmf

Post by Lawrence D'Oliveiro

Post by mm0fmf
My place has about 17000 Windows laptops and servers (and almost as
much Linux stuff) ...

The problem was only on the Windows machines.

I am aware. The comment was to give a scale of the number of machines
and hence likely size of IT dept.

Interesting, though that you have so much Linux-based gear. Could that be
typical nowadays?

[BungleBob]

Post by Lawrence D'Oliveiro
I thought this world-wide trouble might be due to some widely-used
cloud service, but it appears the common factor may be something else,
namely their dependence on a security service called ?CrowdStrike?.

Microsloth as usual starts to blame everyone else rather than it's own
craptastic bug-ridden software.


Microsoft Blames European Commission for Major Worldwide Outage
<https://www.macrumors.com/2024/07/22/microsoft-blames-european-commission-for-outage/>

[D]

Post by BungleBob

Post by Lawrence D'Oliveiro
I thought this world-wide trouble might be due to some widely-used
cloud service, but it appears the common factor may be something else,
namely their dependence on a security service called ?CrowdStrike?.

Microsloth as usual starts to blame everyone else rather than it's own
craptastic bug-ridden software.
Microsoft Blames European Commission for Major Worldwide Outage
<https://www.macrumors.com/2024/07/22/microsoft-blames-european-commission-for-outage/>

Wow! But I shouldn't be surprised. All people I know run linux and were
completely unaffected. There is a link floating around that crowdstrike
did crash debian and rocky linux, but on linux there is no need for
crowdstrikes products in the first place so I imagine that there were very
few debian and rocky installations that were affected.