Sounds like double ungood day for Linode

Posted Jul 29, 2025 6:42 UTC (Tue) by MortenSickel (subscriber, #3238)
In reply to: Sounds like double ungood day for Linode by bracher
Parent article: LWN is back

We are running a small on-site DC at work. There we have a diesel generator that can power the entire building, it is tested (and refuelled) each 3rd month running the entire building off the generator. We also have it serviced once a year. So when we had a power outage about a year ago, the battery powered UPSs kicked in as they should to keep the servers running while the generator starts up, the generator started up, ran for about five minutes, overheated and shut down. (the power outage lasted for a couple of hours).

Just a couple of weeks before, we had done a system test and had the yearly service. It turned out there was some contamination in the cooling liquid that clogged the radiatior which was not detected in the service... The company doing the service got a few questions to answer, but as mentioned elsewhere here, the good thing is that we learned how to get the DC up from nothing.

Sounds like double ungood day for Linode

Posted Jul 29, 2025 7:56 UTC (Tue) by Wol (subscriber, #4433) [Link] (1 responses)

I think I got this from Risks ...

The generator in this incident had a five minute header tank, which was re-filled by a pump from the main tank. Guess what the pump was plugged into ... the mains ...

Cheers,
Wol

Sounds like double ungood day for Linode

Posted Jul 29, 2025 14:08 UTC (Tue) by archaic (subscriber, #111970) [Link]

During hurricane Sandy, I (and many others) found ourselves carrying 5 gallon buckets of diesel up 31 flights of stairs due to the pumps being offline. Since this was Manhattan, the combined inability of being able to buy that fuel, plus the ever increasing delays in leaving the island to get it and then trying to get back into Manhattan with it resulted in a human pipeline of nearly 100 people. Just to keep the 10 or so sad sacks from having nothing to carry up those stairs. Ahhh, better days.... :)