ADVERTISEMENT

Wow, global tech system crash

The Spin Meister

Well-Known Member
Gold Member
Nov 27, 2012
37,531
48,890
1
An altered state
Airlines shut down, computers crashing. Banks, healthcare affected.

Seems that Crowdstrike installed updates over night with a serious glitch that spread throughout the world.

Really shows just how vulnerable we all are.

PS: affected gambling, NIL payments, travel so football related!🥴
 
Seriously affecting MS Azure.

QSQxao.gif
 
Seriously affecting MS Azure.

QSQxao.gif
Said to be isolated to Microsoft systems. Not affecting Apple or Linux.

Said most systems are cured by a simple reboot but some actually require a manual fix. Meaning a tech from Crowdstrike had to show up and manually fix it. 😮

In the discussion they talked about an AT&T failure a week ago wiped out car dealers. Said some are still using paper for transactions a week later. Never heard anything about that.
 
As a tech provider, our system was out for three hours. We were able to correct our CrowdStrike issues and get back into production before 7am. We are still experiencing Microsoft problems on individual laptops. I am an Apple user and am going to buy stock today.

The problem with the airlines is being down for just a few hours totally screws them. Their planes are now out of position, in the wrong airports and don't have the gates or personnel to handle the additional load. Just about all of the planes were on the ground whereas over half are in the air at any time. it will take at least a day to recover. This looks like Southwest's meltdown at Xmas time.
 
As a tech provider, out system was out for three hours. We were able to correct our CrowdStrike issues and get back into production before 7am. We are still experiencing Microsoft problems on individual laptops. I am an Apple user and am going to buy stock today.

The problem with the airlines is being down for just a few hours totally screws them. Their planes are now out of position, in the wrong airports and don't have the gates or personnel to handle the additional load. Just about all of the planes were on the ground whereas over half are in the air at any time. it will take at least a day to recover. This looks like Southwest's meltdown at Xmas time.
Lucky you.
 
Please also be aware that phishers are taking advantage of the CrowdStrike situation to push malware via malicious links that are supposedly “CrowdStrike” updates or “OS Update.”
 
  • Like
Reactions: The Spin Meister
Airlines shut down, computers crashing. Banks, healthcare affected.

Seems that Crowdstrike installed updates over night with a serious glitch that spread throughout the world.

Really shows just how vulnerable we all are.

PS: affected gambling, NIL payments, travel so football related!🥴
If there is ever a war, Russian and Chinese hackers will show just how vulnerable we are. We will also learn who is actually in this country. The whole thing will be beyond devastating.
 
If there is ever a war, Russian and Chinese hackers will show just how vulnerable we are. We will also learn who is actually in this country. The whole thing will be beyond devastating.
Yep, all this chaos by just one company by an accidental update.

Imagine how much stuff is already imbedded waiting to be activated. Or how many bad actors have back doors propped open for future attacks.
 
Yep, all this chaos by just one company by an accidental update.

Imagine how much stuff is already imbedded waiting to be activated. Or how many bad actors have back doors propped open for future attacks.
there are a lot of tools to find this stuff. That isn't to say this can't happen but most state-of-the-art companies are pretty safe. In this case, most companies recovered quickly (mine was minimally affected) but the airlines suffer a physical problem in addition to the technical one. The airlines have planes all over the place now and not enough people to handle them. There aren't even enough gates to house them while grounded.

The big problem has been ransomware. Local govts, with antiquated systems, are the most vulnerable. The city of CLE was out for a week two weeks ago. AT&T got hit last week. With it, they simply shut down your system until you give them money. If you are in the cloud and backup your systems periodically (every night), your exposure is very low.

In this case, MS pushed out an update that was wrong. My guess is they picked up the wrong file. But I don't know. It is hard to believe that this wasn't tested by thousands of people before being pushed into production. I once had a client deconverting over a weekend to another provider. I got a panicked call on Friday night that the files were corrupted. I excused myself from dinner and after a dozen calls, realized my team had picked up a deconversion file from a different client. A was sent to B and B was sent to A. This was supposed to happen after 9pm on a Friday. We had an 18-year-old kid in his first few months do the work. It was just a dumb mistake.
 
  • Like
Reactions: AvgUser
there are a lot of tools to find this stuff. That isn't to say this can't happen but most state-of-the-art companies are pretty safe. In this case, most companies recovered quickly (mine was minimally affected) but the airlines suffer a physical problem in addition to the technical one. The airlines have planes all over the place now and not enough people to handle them. There aren't even enough gates to house them while grounded.

The big problem has been ransomware. Local govts, with antiquated systems, are the most vulnerable. The city of CLE was out for a week two weeks ago. AT&T got hit last week. With it, they simply shut down your system until you give them money. If you are in the cloud and backup your systems periodically (every night), your exposure is very low.

In this case, MS pushed out an update that was wrong. My guess is they picked up the wrong file. But I don't know. It is hard to believe that this wasn't tested by thousands of people before being pushed into production. I once had a client deconverting over a weekend to another provider. I got a panicked call on Friday night that the files were corrupted. I excused myself from dinner and after a dozen calls, realized my team had picked up a deconversion file from a different client. A was sent to B and B was sent to A. This was supposed to happen after 9pm on a Friday. We had an 18-year-old kid in his first few months do the work. It was just a dumb mistake.
Wasn’t MS, it was Crowd Strike. It affected MS systems only. CEO said it wasn’t a coding issue but a bad content file.

He was CNBC for a long interview around 9:00 AM so probably on their website now.

 
Wasn’t MS, it was Crowd Strike. It affected MS systems only. CEO said it wasn’t a coding issue but a bad content file.

He was CNBC for a long interview around 9:00 AM so probably on their website now.

I get it but Crowdstrike is most prominent in MS systems. Regardless, sounds like they did pick up the wrong file to push. That is what I figured. Just a simple stupid mistake. There are checks and balances all along the way to avoid these outages. I had a consultant do a 50 page report on my company's systems after sending a team of 8 people to our offices for a week. Yet, an intern picks up the wrong file to push. Crazy. Just like the Butler PA shooting. You put in contingencies for a million things and some crazy kid sneaks in and shoots people.

One of my favorite movie scenes is from the movie "Body Heat". Ex-con Mickey Rourke is preparing a bomb for his attorney friend who is played by William Hurt. Mickey warns Hurt not to do the crime, whatever it is. He goes on to say "When you do a crime, fifty things can go wrong. If you can think of 25 of them, you are a genius. Conselor, you ain't no genius."
 
My wife and daughter both work for UPMC. Tons of stuff down there. A bunch of people didn't get paid. Computers down. A few couldn't buy lunch or gas with their bank cards. What a mess.
 
My wife and daughter both work for UPMC. Tons of stuff down there. A bunch of people didn't get paid. Computers down. A few couldn't buy lunch or gas with their bank cards. What a mess.
911 systems down. Pharmacies down. Just crazy one little file did this by accident. Yet we are assured all is well and no need to worry about an intentional cyber attack by some of the smartest computer hackers on the planet.
 
911 systems down. Pharmacies down. Just crazy one little file did this by accident. Yet we are assured all is well and no need to worry about an intentional cyber attack by some of the smartest computer hackers on the planet.
No kidding. This much chaos was caused by an accident. Imagine what someone could do with bad intentions. But hey, they said we can all feel free to quit carrying cash.
 
  • Like
Reactions: The Spin Meister
Crowdstrike CEO is on CNBC explaining what happened. Taking serious heat but is accepting it was their update.
They’ll be bankrupt after all of the lawsuits.

Quite the cluster trying to fly today. Hotel I was at couldn’t check me out. Starbucks was cash only until noon. Might be wise not to have so many large entities on the same platform.
 
They’ll be bankrupt after all of the lawsuits.

Quite the cluster trying to fly today. Hotel I was at couldn’t check me out. Starbucks was cash only until noon. Might be wise not to have so many large entities on the same platform.
And this was done by a cyber security specialist firm🥴

It was a crowd strike …. Fer sure
 
  • Like
Reactions: bison13
Being an IT guy who doesn’t have any clients using Crowdstrike is like watching a train derailment today.

It is also educational as to who is using Windows in a critical or public facing role. Windows has its place and it’s not in mission critical applications.
 
Being an IT guy who doesn’t have any clients using Crowdstrike is like watching a train derailment today.

It is also educational as to who is using Windows in a critical or public facing role. Windows has its place and it’s not in mission critical applications.
UPMC Altoona definitely uses Windows. We were just there for a minor outpatient procedure. I remember the computer in the room had a screensaver noting an upgrade to whatever version of Windows.
 
Airlines shut down, computers crashing. Banks, healthcare affected.

Seems that Crowdstrike installed updates over night with a serious glitch that spread throughout the world.

Really shows just how vulnerable we all are.

PS: affected gambling, NIL payments, travel so football related!🥴
Woke up this morning in Sardinia, Italy, concluding a work trip. Took a very early flight on ITA to Rome to catch a United flight to Chicago. 4 hour delay, missed connection, and a "bonus" night in Schaumburg, IL after being awake for 22hours....divesting from Crowdstrike stock as I type.
 
Woke up this morning in Sardinia, Italy, concluding a work trip. Took a very early flight on ITA to Rome to catch a United flight to Chicago. 4 hour delay, missed connection, and a "bonus" night in Schaumburg, IL after being awake for 22hours....divesting from Crowdstrike stock as I type.
Too late. Stock crashed as fast as the global network !
 
Computers where I work (County Government) were down when I came into work at 8:30 AM. IT staff had to do manual fixes, but we were all up and running by 11:30 AM. Took the IT tech 10 minutes to fix mine. Why are they saying this will take 2-3 weeks to fix on some systems?

P.S. Why wasn't my home computer impacted? Was this an update that IT staff at these various companies had to push out? I also noticed that my coworkers who shut down their computers were fine, people like me who just restarted (or left their computers on) were all hit.
 
Last edited:
No kidding. This much chaos was caused by an accident. Imagine what someone could do with bad intentions. But hey, they said we can all feel free to quit carrying cash.
I only use credit cards for gas (although often cash), hotels, airline reservations, and game tickets. Pretty much cash for everything else.
 
  • Like
Reactions: PAgeologist
UPMC Altoona definitely uses Windows. We were just there for a minor outpatient procedure. I remember the computer in the room had a screensaver noting an upgrade to whatever version of Windows.

When I arrived at O'Hare airport last night, about 16 hours after things first crashed, there were monitors everywhere still showing the blue screen of death.
 
  • Like
Reactions: PAgeologist
Computers where I work (County Government) were down when I came into work at 8:30 AM. IT staff had to do manual fixes, but we were all up and running by 11:30 AM. Took the IT tech 10 minutes to fix mine. Why are they saying this will take 2-3 weeks to fix on some systems?

P.S. Why wasn't my home computer impacted? Was this an update that IT staff at these various companies had to push out? I also noticed that my coworkers who shut down their computers were fine, people like me who just restarted (or left their computers on) were all hit.
The fix wasn’t difficult as long as BitLocker (disk encryption) wasn’t set up. If it was then it needed to be set up properly with the keys in Active Directory and the tech would use the 48 character key to unlock the disk before the fix.

Your home PC wasn’t affected because it wasn’t a Microsoft update, it was Crowdstrike. Crowdstrike is a managed security platform that is paid for by endpoint by customers.
 
The fix wasn’t difficult as long as BitLocker (disk encryption) wasn’t set up. If it was then it needed to be set up properly with the keys in Active Directory and the tech would use the 48 character key to unlock the disk before the fix.

Your home PC wasn’t affected because it wasn’t a Microsoft update, it was Crowdstrike. Crowdstrike is a managed security platform that is paid for by endpoint by customers.
Wonder what these customers paid to get their massive IT outage?
 
Airlines shut down, computers crashing. Banks, healthcare affected.

Seems that Crowdstrike installed updates over night with a serious glitch that spread throughout the world.

Really shows just how vulnerable we all are.

PS: affected gambling, NIL payments, travel so football related!🥴
It was a test of things to come.
 
Woke up this morning in Sardinia, Italy, concluding a work trip. Took a very early flight on ITA to Rome to catch a United flight to Chicago. 4 hour delay, missed connection, and a "bonus" night in Schaumburg, IL after being awake for 22hours....divesting from Crowdstrike stock as I type.
My sister was supposed to fly from Amsterdam to Philly (w some connections) arriving Friday night. Luckily she was still 3rd in line for check in when systems went haywire. So she was able to have her luggage- spend another day and a half in a city which is not at all comparable to Schaumburg😀

Supposed to arrive late tonight. She works for Peace Corps and is a global traveller with appreciation for "first world problems"....but even she was flustered.

A real mess!
 
Does Crowdstrike have financial liability for this?
My guess is that in all of their contracts, their duties are phrased in a way that excludes liability. Otherwise, in an analogous situation, Microsoft would be sued all of the time.
 
I don't like the way that software has been sold recently. The way I understand it Microsoft 365 is sold as part of a renewable license. (per year I believe) Effectively, you don't own what you have paid for and your work (say for instance Word docs) is at the mercy of Microsoft whims. I use Windows 7 (have about 5 backup discs in case of security problems) and I believe Microsoft 2012 for Word and Excel. Haven't had any of the current problems.
 
My guess is that in all of their contracts, their duties are phrased in a way that excludes liability. Otherwise, in an analogous situation, Microsoft would be sued all of the time.
No matter how they word their contracts they still have an expectation to perform. Contract language can protect from some issues but this was an extremely egregious failure.

Hard to believe they don’t have major liability here. The lawsuits will be massive and they eventually negotiate a class action settlement. Only questions are how soon and how big?🤷🏻‍♀️
 
  • Like
Reactions: EdwardoCarrachio
ADVERTISEMENT
ADVERTISEMENT