The problem of getting power to startup equipment is one thing, but there’s another cool problem that this article covers but I don’t think explained very well.
A power grid is a massive distributed physical system. The energy input must exactly match the energy output at all times.
But what happens when the energy ins and outs are not balanced? The answer is that a balance is found somehow. Physics demands it.
If there is excess power on the grid and no electrical load, that power comes back to the generator (s). The turbines or whatever driving the generators produce more torque than the retarding torque from the generator coils, so they speed up. The AC grid frequency is mostly maintained by the rotating generator speed (3000 rpm for 50 Hz), so that goes up too.
Conversely, if there is excess load and not enough power, electrical drag from the generator coils exceeds the torque from the turbine (or whatever), and the generator slows down. The operator has to burn more fuel, or pull out control rods, or open more water gates, to get the speed back up.
So what is the black start challenge here? You have to go from 0 W to whatever GW the grid normally runs at. Normally when a generator plant is switched onto the grid, that gen represents a small fraction of the total grid power, so the disturbance to the grid is small. But coming back from a black start that’s not true. Going from 1 plant online to 2: you could be doubling the power level. This means you have to switch on loads (possibly many km away) at the exact same time you switch in the power. If the disturbance is too much, various equipment will trip off the grid as the AC frequency careens out of control.
it’s also made worse by the fact that many appliances are turned on in blacked out areas, so when these are brought online, there’s spike in load power that then tapers off pretty quickly. this can be managed by powering on areas on in smaller chunks, as small as single blocks
this is PSA to turn off appliances off during blackout (biggest ones are things that deal with heat: mostly air conditioning/heat pump and all kinds of heaters)
also in Spain specifically some capacity was met by spinning generators from nuclear or hydro or gas powerplants, but also there’s a lot of photovoltaic generation, which doesn’t vary frequency with imbalances in load. after loss of power nuclear reactors that were running at that time are out for a day because of xenon poisoning, and it looks like first nuclear powerplant went online again only yesterday
maybe they’ll update inverters in at least some of generating stations so that blackstart with PV assistance would be possible in future, because from what i understand most of these are grid-following inverters. this might require policy changes and tighter control of PV powerplant by grid operator. hydropower is most useful in starting from complete blackout condition because all power that is needed is just what it takes to turn valves + some communications and remote switching to make sure it goes to other powerplants
for now no one knows what really happened, but i do hope that investigation will allow for figuring out what went wrong and preventing similar failure in the future
it looks like Portugal had it even worse - their power generation dropped to zero (until 15:00), disconnected from Spain, started hydro and pumped hydro to bring up gas and wind power (22:00 to midnight), then solar, then when they figured their shit out connected to Spain again and lent them a couple gigawatts (bigger grids are more stable so it’s good for both)
It’s interesting, it’s kind of like starting a fire the old fashioned way, to get a big fire you start with embers, and in this case, it’s the smaller power plants getting operational, temporarily electrically powered by diesel generators, that supplies those embers in order for the bigger power plants to start.
Just because it’s only mentioned in some reposts, they had the grid back up and running at 100% in like six hours after this was published, so about 24 hours after the event itself (and it was at about 30%-50% when this was published already).
I mean, I still think it’s interesting to learn about this, but I do think it’s fun how much more pessimistic the article’s estimate is than how this actually went down. It’d be interesting to see a follow up covering how they went about pulling that off.
Major power facilities require power to operate, and there’s lots of unmet demand.
Since the Iberian Peninsula lost power in a massive blackout, grid operators are in the process of trying to restore power to millions of customers and businesses. As you might imagine, the process—termed a “black start”—is quite a bit more challenging than flicking on a switch. However, the challenge is made considerably more difficult because nearly everything about the system—from the management hardware that remotely controls the performance of the grid to the power plants themselves—needs power to operate.
You’d assume the critical “energy grid management systems” to have battery backups at a minimum, but also solar or some other independent power supply.
Grid management is currently my business specialisation. I can assure you that maintaining a grid operational is a lot more complex than keeping backup generators at a few key facilities. For example, the smart meters collect data all the time. Even with a power and communication loss, they will continue to collect for a while, storing the data internally. When communication is restored, they will send all their stored data at once. Now multiply that by millions of meters and days of data. The servers are not going to be able to ingest that torrent information. Now, what if the outage was extra long like this one, and the meters ran out of internal storage or charge? How are you going to manage the data gaps?
That’s only meter data. There’s a ton of other systems in play. Very few people realize how delicate and complex power grids are. I can’t speak for Spain, but the US grid is held up by spit, duct tape, brute force and prayers to the old gods. May Cthulhu have mercy on us.
EDIT: Typos
I’m not doubting the complexity. I’m doubting capitalisms ability to properly fund and plan the necessary redundancy of critical infra.
The problem you describe is not the same as mine. That’s an IT/DevOps problem I am extremely familiar with, and planning for spikes in usage is much simpler than battery backup and independent power. Internet services can scale extremely well these days. That’s a solved problem.
The point is that critical infra should be architected to handle this type of scenario.
You are telling me that smart meters, of all things, don’t have API circuit breakers to prevent them from crashing the entire system? Talk about irony.
