LWN is back

During the Akamai acquisition it was reported Linode (like many others) does/did not own the datacenters, but lease from wholesale providers like Databank (Atlanta), Pacnet (Singapore) and Telecity (Frankfurt). The ongoing incident report reads that the DC they lease from in Newark had a catastrophic failure in power/cooling leading to total power outage across the shared facility (or a segment, depending on physical design).

Linode employees are attempting to recover the entire DC runtime from a hard cold machine start, no small feat - I've worked at a company like this that had a similar outage in the late 2000s due to a truck hitting the building (yeah); once the cooling goes, you *have* to shut down servers or electronic components start to melt themselves.

I think the train issue I had a few Christmas's ago was worse.

I was at a complex London terminus. They told us all trains from one set of platforms aren't running due to a tree falling in the storm - which makes sense, tree on line, can't move trains, no service, checks out.

But then every few minutes they would announce "one" exception which was running, the next train for those platforms, and they continued doing this for over an hour. "No trains" "Except the 1145" "No trains" "Except the 1152" "No trains" "Except the 1156" and so on. Rather than say OK, the trains are running normally but..., or actually not having trains because there's a tree in the way they had decided to gaslight their customers by insisting there wouldn't be any trains, then nonetheless running every single train on schedule anyway.

I assume there's some mismatch between strategy to run a day's trains (tree blocks line, OK no trains) and tactics to run individual services (we can just avoid the closed section, for every single train) and it wasn't anybody's job to ensure the outcome makes coherent sense.

The result however was the normal departure boards don't work - there aren't any trains, they've just said so - so you can't tell when your train will run or where from. But if you just give up and leave the station your train does run as scheduled anyway, and you've missed it, you need to know where it ought to be and then get aboard when inevitably it does run after all. Very silly.

I've been with Hetzner for over 20 years - first dedicated servers, now cloud instances. The experience has been flawless, the few times I needed support it was quick and knowledgeable, and the pricing has always been very reasonable.

Lots of good experiences with Hetzner here too. About the only complaint is that sometimes our customers would like to get a bit more detail than Hetzner provides after short (rare) outages. But personally I think the amount of detail is pretty reasonable most of the time.

I especially like their APIs which allow automating a lot of things.

Key is to have a full backup not just in another region, but at your own home or at a completely different company. Catastrophic failures like the OVH fire are one thing - and if you didn't pay for geographic redundancy, the provider won't proactively do backups of your data. Another worry is the sudden termination of your account because "algorithm said so".

My two points for hosting are: look at classic dedicated servers, not "Cloud" if you don't need any Cloud feature. If you don't need modern hardware, go for the old stuff at OVH's "eco" servers or Hetzner's "auctions". (Hyperscalers happily sell you old hardware for very modern prices.) And look for providers where all traffic is included. Sure, at some point speed is capped, but it's a much better feeling that you can't have any bad billing surprise when someone popular links you - or some botfarm, or some scrapers. Especially for community projects.

I'd argue that unless you're paying for something like SaaS, PaaS, or some other cloud operations, the onus of backups should be on the renter. There's a difference in simply paying for a VM/server hardware and paying for more expensive cloud oriented services. One of those differences is who's responsible for data backup. If I'm paying for basic virtual space or basic server hardware, then the main virtual image is on my own hardware along with any backups. The image being hosted is the copy. If I'm hosting mixed data in the cloud then I want to make sure the hosting provider is providing backup copy services. I'm unfamiliar with OVH's services, but if they weren't providing backups nor did they claim to provide them, then the fault is not OVH. Anyway, point is, buyer beware and read the fine print.