Just watching the proverbial bodies stack up
I hate you. Today I discovered a server that appeared to be in working order was actually just hanging on by a thread. The crash when the crowdstrike driver caused the blue screen of death must have corrupted some of the network configuration in the registry. When I couldnt make the failover cluster live migrate to or from that server I ejected it from the cluster and rebooted it, only it never came back up 😥
Now all my servers are suspect until I can verify that they are healthy.
In general, I don’t think the concept of a configuration database is bad - at least not inherently moreso than every application putting its own configuration file(s) in whichever format it wants somewhere on the filesystem.
Whether the window registry is a good implementation of this concept is surely debatable.
This latest debacle is making my department move from windows to Linux. We were already planning it very slowly but then everything crashed at the same time…and all our other services worked except the ones on windows boxes. We can’t afford downtime so it was decided.
It’s not a good reason but I’ll take it haha.
I don’t know how CrowdStrike works on Linux, but it’s worth remembering that if it’s a kernel level driver like it is on windows, and they release a driver that crashes the Linux kernel, there’s a chance for the same thing to happen.
Thank you for mentioning that. I really hate how people on here think Linux is some panacea that will magically solve everything. It too it just another tool that depends on how it’s used. CrowdStrike exists for Linux, and it was crashing systems a few months ago.
The bigger issue is most people who use Linux know what they’re doing. There are a lot of competent Windows Administrators too who didn’t have issues or were able to recover them in a timely manner. What happens though is you have a very large set of people who just need a computer, need is secured and don’t know how to administer or manage it. Doesn’t matter if they’re running Linux, or Windows, they’ll always have the greatest problems. They just happen to use Windows because it offers better Enterprise support options and usability. One day, Linux may be that, but I guarantee it won’t fix all of those pebkac issues.
It did happen already https://www.theregister.com/2024/07/21/crowdstrike_linux_crashes_restoration_tools/
I personally get nervous when any software wants to mess with drivers unless it’s graphic drivers.
For work we don’t plan on using cloud strike. We needed to get everything up asap and the os allowed us to do so quickly. Seemingly unrelated systems and Azure was all down for quite some time.
I was on vacation. They laid off my boss 2 weeks ago. They laid off my bosses boss 3 weeks ago. Guess who no one could contact?
I found out about all of this 2 days after the fact. Best vacation ever.
Wtf did they honestly expect from you in that situation anyway? Do they really think you would come back from vacation to deal with that? From what I can tell, fixing it is barely beyond a T1 hekpdesk amount of knowledge, just an annoying process. Unless there’s more to it, I haven’t looked heard at the details but that’s the gist I’m hearing.
But unless they’re going to triple my vacation length (in terms of paid days off in other words) and maxing out all my upgrade options for the next one I’ma have to say they can get bent.
Basically they gotta pay for the best version possible of that trip (assuming there was a trip) and give me enough time to take 2 more.
I ain’t taking a work call on vacation.
Funny story. My boss vetoed work phones as a cost savings and at first there was a squawking fest but honestly he handled all communication to us and protected us from so much bullshit. Now he’s gone and no one could figure out how to get in contact with us aside from email/teams.
The Feels Good guy has a stick figure hand sticking up out of his arm. I cannot unsee it, and I suspect now neither can you.
Man, I quit my IT job 2 years ago and I haven’t looked back. So glad I got out when I did.
