Sounds like double ungood day for Linode

Posted Jul 28, 2025 23:05 UTC (Mon) by bracher (subscriber, #4039)
In reply to: Sounds like double ungood day for Linode by mote
Parent article: LWN is back

I'm reminded of an incident from early in my career. We were co-located in a DC that sat on two theoretically-independent power grids, plus diesel generators to backup in the unlikely event that both grids went down. It seemed as if these folks had thought of everything, they had really detailed procedures for just about everything.

Naturally both grids eventually _did_ go down, and the generators kicked in, all according to plan. And a few minutes later all of the generators died. It turns out that they procedures for _many_ things, including testing the failover to the generators on a monthly basis. The one thing they did _not_ have a procedure for was the refueling of the generators. So when they were eventually needed they had only a few minutes of fuel left.

Sounds like double ungood day for Linode

Posted Jul 29, 2025 6:42 UTC (Tue) by MortenSickel (subscriber, #3238) [Link] (2 responses)

We are running a small on-site DC at work. There we have a diesel generator that can power the entire building, it is tested (and refuelled) each 3rd month running the entire building off the generator. We also have it serviced once a year. So when we had a power outage about a year ago, the battery powered UPSs kicked in as they should to keep the servers running while the generator starts up, the generator started up, ran for about five minutes, overheated and shut down. (the power outage lasted for a couple of hours).

Just a couple of weeks before, we had done a system test and had the yearly service. It turned out there was some contamination in the cooling liquid that clogged the radiatior which was not detected in the service... The company doing the service got a few questions to answer, but as mentioned elsewhere here, the good thing is that we learned how to get the DC up from nothing.

Sounds like double ungood day for Linode

Posted Jul 29, 2025 7:56 UTC (Tue) by Wol (subscriber, #4433) [Link] (1 responses)

I think I got this from Risks ...

The generator in this incident had a five minute header tank, which was re-filled by a pump from the main tank. Guess what the pump was plugged into ... the mains ...

Cheers,
Wol

Sounds like double ungood day for Linode

Posted Jul 29, 2025 14:08 UTC (Tue) by archaic (subscriber, #111970) [Link]

During hurricane Sandy, I (and many others) found ourselves carrying 5 gallon buckets of diesel up 31 flights of stairs due to the pumps being offline. Since this was Manhattan, the combined inability of being able to buy that fuel, plus the ever increasing delays in leaving the island to get it and then trying to get back into Manhattan with it resulted in a human pipeline of nearly 100 people. Just to keep the 10 or so sad sacks from having nothing to carry up those stairs. Ahhh, better days.... :)