To add to that, I very much doubt any big company tests and verifies anything anymore.
Boeing ships planes with missing bolts and proper software, Crowdstrike pushes updates with no testing, we’ve all seen Microsoft push updates that break stuff because there’s no testing, and that’s just what comes to mind.
That’s how they maximize profits - get rid of testing environments, do minimal checks, and have the one guy doing 3 jobs at once just push it to production.
I’ve been in IT for the banking industry for over a decade and I promise you, we’re all a missed cup of coffee or a comma away from another massive outage due to a program or network misconfig.
As long as business culture is set to maximize profits for one quarter, I wouldn’t trust a sales website about “verification” or “disaster recovery backups” any more than I trust a used car salesman.
That goes for Crowdstrike, but also all of their competitors.
I’ve got friends at Boeing on DoD contracts. Not only is it waterfall, it gets tested hardcore. My experience in private industry is the exact opposite. A consultancy I know of just lost (pretty sure) a state contract because they opened shit up to the public because, surprise surprise, they didn’t test their infra changes.
Now I will say that when I have had to manage client SLAs and there is a cost to post-release defects and change requests, testing increases. Not to the level I’m super comfortable with (which is well below perfect, mind you; I like shipping more than once in a lifetime), but a bit more.
The CTO of a competitor, Sentinel one, was just on the security podcast Risky buisness. He went deep into how his company does this.
Apprently, their client touches the kernel much less, so it is less likely to cause issues. They also have a large internal test bed that updates have to pass to go out at all, and then if they have a 2% failure rate during the wide deployment, the update is automatically stopped.
Crowdstrike does almost none of this. There core client is deep in the kernel, making it powerful and dangerous. They apprently do test on their local machines, but the company is an apple shop, so none of the windows updates was tested locally. The updates pushed out started crashing computers immediately, but weren’t stopped for 78 minutes by manual intervention. That was long enough to crash 8 million computers across the world.
