Saturday, 20 July 2024

The great Crowdstrike debacle ...

 Strangely, it wasn't much of a problem for us.

I was taking J to the airport to catch a rescheduled early morning flight (Thanks Virgin!) the next day to Launceston. It was actually our wedding anniversary so the idea was to have lunch in a cafe we like in Euroa, continue down to an airport hotel and have a not particularly romantic dinner - our preferred airport hotel, the Mantra, does a perfectly competent if unimaginative dinner menu, and I would see J onto the airport shuttle bus at stupid o'clock before driving back.

So, off we went. We had an enjoyable lunch, and then about ten to three, we pulled into Wallan services on the freeway to fill the car. This was of course about ten minutes before the planet started going offline, so as far as we were concerned it was just a normal Friday afternoon drive through horrible Melbourne traffic.

When we got to the hotel, I noticed that the pair of public computers they have in the lobby for people to print boarding passes or confirm their check in had blue screened, but that's not exactly unusual.

The hotel could check us in, take money off us for the room, and book us a table for dinner, and the hotel wifi worked and we could login.

It was only when we logged in to check our email that we discovered that the world had gone to shit while we were crawling along the M31. The ABC news channel was full of stories about cash machines not working, supermarket self service checkouts going off line, and more concerningly that airport checkins were generally down, and trauma of traumas, McDonalds was offline!

Strangely, I wasn't concerned. I've seen my share of IT cockups, and while one could ask very pointed questions about contingency planning and roll back strategies, and perhaps contemplate taking the team responsible out to the car park and lining them up against a wall a la Ceausescu, I was fairly certain that people would be working on a way to undo the autodeployed patch.

And so it was, by the time we went to dinner there was a fix, it was being tested and looked to be working.

The ABC had a confusing interview with a security geek who was struggling to explain that because reboots were required some services might take longer than others to come back, and some poor engineers were going to spend Friday evening driving round to remote locations to do server restarts.

By the time we'd finished dinner, Virgin was happily claiming the J's flight would depart as scheduled. 

Just to be on the safe side we set our alarm for 0500 to give her plenty of time in case they were having to resort to old school check ins and bag drops, but actually everything was more or less working.

Due to the early start we'd skipped breakfast. I'd originally planned to have breakfast at one of the cafes in Euroa, but due to my super early start, none of them were open, well except the McDonalds on the way out of town, and they managed to produce  both a ham and cheese toastie, and what they claimed was coffee, but was more like hot brown water than anything else.

I did stop at the big Coles supermarket on the edge of Wangaratta to pick up some fresh vegetables, bread and salad, and their self service checkouts were still down, but they'd solved the problem by throwing people at it - every old school checkout was manned with the result that everything flowed really smoothly and it was quicker than a usual Saturday morning.

So, even though the world might have seemed to be ending, it wasn't really.

There are however serious questions about corporate reliance on (a) Crowdstrike and (b) Microsoft products.

Crowdstrike themselves put our a statement to say that anything reliant on Linux or Mac backends should be unaffected. What was worrying was the sheer number of critical systems that went off line and the extent to which corporate IT has embraced a single point of failure ... 

[Update]

And for those of you who are concerned Virgin did take off more or less on time and deliver J to Launceston only twenty minutes later than claimed, mostly due to a pretty dramatic storm coming out of the Antarctic this morning ...

No comments:

Post a Comment