LWN: Comments on "LWN is back"

Corporate separation

corbet — Mon, 11 Aug 2025 14:03:20 +0000

That separation also insulates one company from a failure of the other; if the company using the building goes bankrupt, the building itself is unaffected (beyond needing to find a new tenant).

Sounds like double ungood day for Linode

farnz — Mon, 11 Aug 2025 10:02:24 +0000

I've noted a common pattern across the world (including the EU) where a publicly traded holding company owns two sub-companies: one owns capital assets like land and buildings, which it leases to the other sub-company that operates them.

The purpose is not taxation-related in the cases I've seen; rather, it's that the sub-companies have separately audited accounts, and thus you can be confident (as an investor in the holding company) that you're not seeing results "goosed" by accounting tricks to move losses between capital and operational sides. Instead, you've got an audited set of accounts for the capital side, and a separate set for the operational side, which ensures that you cannot be tricked by accounting games.

Sounds like double ungood day for Linode

bartoc — Thu, 07 Aug 2025 22:36:24 +0000

I'm not sure how it works in the EU, but in the USA these sorts of leases are really, really common for all types of real estate.

I think it's because C-corporation taxation is quite punishing compared to other ownership structures, particularly for capital gains realization.

Sounds like double ungood day for Linode

anselm — Wed, 30 Jul 2025 09:19:34 +0000

once the cooling goes, you *have* to shut down servers or electronic components start to melt themselves.

Way back at university we once had a week or so of unscheduled downtime because legionella bacteria had been discovered in the main campus server room AC and they had to shut everything down in order to get the cooling system disinfected. This was at the height of summer, of course, with temperatures in the 30°C+ range outside.

It sucked especially because apart from the NFS servers everybody's files were sitting on even if they had their own workstation on their desk, it included the machine everyone was reading their e-mail on (this was in the early 1990s when e-mail was still much more of a thing than it is today, but POP3/IMAP hadn't caught on, so you would telnet to the e-mail server and use the mail command there, or MH if you were a serious e-mail nerd), and some sysadmins spent sleepless nights jury-rigging a replacement system in some cool and airy location elsewhere on the premises just so people could get at their e-mail.

Backup providers

meyert — Tue, 29 Jul 2025 21:45:43 +0000

Of course you want to host your website on a multi regional kubernetes cluster for high availability!

Backup providers

mbunkus — Tue, 29 Jul 2025 19:27:56 +0000

Again, I totally agree. What I was talking about, though, is that there were customers that _did pay_ for the optional backup addon/option, and those backups were lost, too, due to the backup servers being located in the same DC that burned down.

You can argue that only having one backup location is not enough, and I also agree, but still — the setup chosen by OVH is downright incompetent.

Backup providers

Cyberax — Tue, 29 Jul 2025 17:54:12 +0000

Typically cloud providers don't provide backups for servers. In fact, they explicitly tell you that you can lose the data if the DC is destroyed.

It's still your responsibility to mirror the data into another region (datacenter) and/or use storage that is guaranteed to have durability guarantees (like S3 in AWS).

HostHatch

yodermk — Tue, 29 Jul 2025 17:53:03 +0000

Curious if anyone has experience with https://hosthatch.com/
I'm leaning towards putting some personal stuff on that. Looks like a really good value.

Sounds like double ungood day for Linode

yodermk — Tue, 29 Jul 2025 17:44:47 +0000

> I've worked at a company like this that had a similar outage in the late 2000s due to a truck hitting the building (yeah)

Haha, I was there working support that night. Fun times!

Backup providers

mbunkus — Tue, 29 Jul 2025 14:21:18 +0000

I totally agree with you wrt. backup responsibilities. The thing is that in the OVH case not only were bare-bones VMs affected, even customers who did pay for backups were affected as OVH had located the backup targets in the same data centers as the machines that were backed up, and therefore the backups burned down, too. I even think they lied to customers about where backups were stored, but don't quote me on that.

That part's definitely on OVH.

Sounds like double ungood day for Linode

archaic — Tue, 29 Jul 2025 14:08:21 +0000

During hurricane Sandy, I (and many others) found ourselves carrying 5 gallon buckets of diesel up 31 flights of stairs due to the pumps being offline. Since this was Manhattan, the combined inability of being able to buy that fuel, plus the ever increasing delays in leaving the island to get it and then trying to get back into Manhattan with it resulted in a human pipeline of nearly 100 people. Just to keep the 10 or so sad sacks from having nothing to carry up those stairs. Ahhh, better days.... :)

LLMs looking over your recovery plan

farnz — Tue, 29 Jul 2025 11:29:59 +0000

It can also do things like decide that certain job titles imply particular people (not reliably, but often enough to be useful), and then highlight that your recovery plans depend on Chris being present at site, because Chris is the Site Manager, the Health and Safety Lead, and the Chief Electrician, and your plans depend on the Site Manager, the Health and Safety Lead, the Chief Electrician, or the Deputy Chief Electrician being on site.

While the LLM made a mistake here (since there's two people you rely on, not one), it's still highlighted that a document saying that 1 of 4 roles is needed to recover has, in fact, become 1 of 2 people.

Sounds like double ungood day for Linode

mathstuf — Tue, 29 Jul 2025 11:02:04 +0000

I'm not saying to just blindly hand it to an LLM and call it a day on its first response. But you can use it to ask if it notices anything like a dependency cycle, implicit knowledge, or steps lacking clarification.

It's obviously not going to know about problems like "Bob has the key, but it is lost on his 100-key keyring", but it (hopefully) can notice things like "it says to get the key, but not where the key lives" that might be implicit knowledge in the author's eyes but these instructions are those that really should have a thorough once-over by someone *not* intimately familiar with the process to help rid it of such implicit assumptions.

Sounds like double ungood day for Linode

tialaramex — Tue, 29 Jul 2025 10:35:44 +0000

I think the train issue I had a few Christmas's ago was worse.

I was at a complex London terminus. They told us all trains from one set of platforms aren't running due to a tree falling in the storm - which makes sense, tree on line, can't move trains, no service, checks out.

But then every few minutes they would announce "one" exception which was running, the next train for those platforms, and they continued doing this for over an hour. "No trains" "Except the 1145" "No trains" "Except the 1152" "No trains" "Except the 1156" and so on. Rather than say OK, the trains are running normally but..., or actually not having trains because there's a tree in the way they had decided to gaslight their customers by insisting there wouldn't be any trains, then nonetheless running every single train on schedule anyway.

I assume there's some mismatch between strategy to run a day's trains (tree blocks line, OK no trains) and tactics to run individual services (we can just avoid the closed section, for every single train) and it wasn't anybody's job to ensure the outcome makes coherent sense.

The result however was the normal departure boards don't work - there aren't any trains, they've just said so - so you can't tell when your train will run or where from. But if you just give up and leave the station your train does run as scheduled anyway, and you've missed it, you need to know where it ought to be and then get aboard when inevitably it does run after all. Very silly.

Sounds like double ungood day for Linode

Wol — Tue, 29 Jul 2025 07:56:04 +0000

I think I got this from Risks ...

The generator in this incident had a five minute header tank, which was re-filled by a pump from the main tank. Guess what the pump was plugged into ... the mains ...

Cheers,
Wol

Backup providers

taladar — Tue, 29 Jul 2025 07:40:40 +0000

APIs are very useful to have, but you don't need them for all kinds of detailed services. APIs for creating and deleting servers and rebooting them and similar tasks are quite useful to automate installations though (e.g. the Hetzner API) and should be relatively trivial to replace if you need to move somewhere else.

Backup providers

taladar — Tue, 29 Jul 2025 07:38:13 +0000

Lots of good experiences with Hetzner here too. About the only complaint is that sometimes our customers would like to get a bit more detail than Hetzner provides after short (rare) outages. But personally I think the amount of detail is pretty reasonable most of the time.

I especially like their APIs which allow automating a lot of things.

Sounds like double ungood day for Linode

taladar — Tue, 29 Jul 2025 07:33:15 +0000

Why would you ever entrust something like that to an LLM? Here you want reliable answers and algorithms exist to get reliable answers.

Sounds like double ungood day for Linode

MortenSickel — Tue, 29 Jul 2025 06:42:29 +0000

We are running a small on-site DC at work. There we have a diesel generator that can power the entire building, it is tested (and refuelled) each 3rd month running the entire building off the generator. We also have it serviced once a year. So when we had a power outage about a year ago, the battery powered UPSs kicked in as they should to keep the servers running while the generator starts up, the generator started up, ran for about five minutes, overheated and shut down. (the power outage lasted for a couple of hours).

Just a couple of weeks before, we had done a system test and had the yearly service. It turned out there was some contamination in the cooling liquid that clogged the radiatior which was not detected in the service... The company doing the service got a few questions to answer, but as mentioned elsewhere here, the good thing is that we learned how to get the DC up from nothing.

Sounds like double ungood day for Linode

mathstuf — Tue, 29 Jul 2025 00:08:41 +0000

Something that LLMs might actually be useful at helping to spot assuming you have things written down in a LLM-consumable medium (i.e., not only stored in meatspace).

Sounds like double ungood day for Linode

bracher — Mon, 28 Jul 2025 23:05:05 +0000

I'm reminded of an incident from early in my career. We were co-located in a DC that sat on two theoretically-independent power grids, plus diesel generators to backup in the unlikely event that both grids went down. It seemed as if these folks had thought of everything, they had really detailed procedures for just about everything.

Naturally both grids eventually _did_ go down, and the generators kicked in, all according to plan. And a few minutes later all of the generators died. It turns out that they procedures for _many_ things, including testing the failover to the generators on a monthly basis. The one thing they did _not_ have a procedure for was the refueling of the generators. So when they were eventually needed they had only a few minutes of fuel left.

Backup providers

mrkiko — Mon, 28 Jul 2025 21:22:20 +0000

I am having a good experience with them as well. Hoping they can implement a blind users friendly way to access system console for their Virtual Private server Instances. When networking does not come up for any reason, finding out what failed during boot might be tricky. Last time I Had to start a qemu VM from a live ISO environment.

Sounds like double ungood day for Linode

smurf — Mon, 28 Jul 2025 21:02:59 +0000

This is why reasonable badge systems have an offline mode which knows the cards of a few senior DC operators, along with the card that's in the locker which the fire brigade has access to.

The interesting part about cyclic dependencies is that you have customers, which need to be notified when their servers go belly-up for some reason (including when the router they're behind blows a fuse). Thus you need dependency tracking anyway. Which presumably should alert you when there's any cycles in that graph. Which should prevent this from happening. Famous last words …

Backup providers

chmod — Mon, 28 Jul 2025 20:35:07 +0000

Arch Linux uses Hetzner for a huge part of its infrastructure, too. archlinux/infrastructure repository
In the end, it's important to have two independent providers, in case of a disaster (like DC on fire).

Backup providers

dskoll — Mon, 28 Jul 2025 19:33:48 +0000

I agree with this. And it's another reason why I prefer to get just generic Linux machines or VMs rather than paying for fancy cloud stuff. There's much less vendor lock-in if all you need is root on your own Linux machine; it's cheaper; and you can manage your own backups.

Don't fall for the siren song of fancy cloud APIs that tie you to a specific vendor. They're convenient, but they hook you in.

Backup providers

Heretic_Blacksheep — Mon, 28 Jul 2025 18:36:55 +0000

I'd argue that unless you're paying for something like SaaS, PaaS, or some other cloud operations, the onus of backups should be on the renter. There's a difference in simply paying for a VM/server hardware and paying for more expensive cloud oriented services. One of those differences is who's responsible for data backup. If I'm paying for basic virtual space or basic server hardware, then the main virtual image is on my own hardware along with any backups. The image being hosted is the copy. If I'm hosting mixed data in the cloud then I want to make sure the hosting provider is providing backup copy services. I'm unfamiliar with OVH's services, but if they weren't providing backups nor did they claim to provide them, then the fault is not OVH. Anyway, point is, buyer beware and read the fine print.

Backup providers

seneca6 — Mon, 28 Jul 2025 16:42:47 +0000

Key is to have a full backup not just in another region, but at your own home or at a completely different company. Catastrophic failures like the OVH fire are one thing - and if you didn't pay for geographic redundancy, the provider won't proactively do backups of your data. Another worry is the sudden termination of your account because "algorithm said so".

My two points for hosting are: look at classic dedicated servers, not "Cloud" if you don't need any Cloud feature. If you don't need modern hardware, go for the old stuff at OVH's "eco" servers or Hetzner's "auctions". (Hyperscalers happily sell you old hardware for very modern prices.) And look for providers where all traffic is included. Sure, at some point speed is capped, but it's a much better feeling that you can't have any bad billing surprise when someone popular links you - or some botfarm, or some scrapers. Especially for community projects.

Re: LWN is back

sionescu — Mon, 28 Jul 2025 16:41:33 +0000

Something like a dedicated status.lwn.net would be good (and not too expensive either).

Backup providers

dskoll — Mon, 28 Jul 2025 16:29:47 +0000

I remember the OVH fire; that was in France, I believe.

And that's one reason I have VMs in two different data centres in cities about 550km apart. 🙂

Sounds like double ungood day for Linode

farnz — Mon, 28 Jul 2025 16:05:32 +0000

And I suspect that if you could review the chain of decisions that led to "door badge system depends on DC being up and running", every single one would make sense in isolation; it's only when you test "what happens when we take the DC down remotely? Can we get in?" that you discover that these decisions have chained together to imply "when the DC is down, door badge system is also down".

For example, the door badge system might be configured to automatically sync access permissions from internal directories (so that you don't have to manually update DC access permissions as people join and leave), and fail closed if internal directories including all mirrors in other DCs are unavailable (on the basis that you don't want an attacker to be able to cut the OOB Internet link and get in), then you allow the badge system to use DC Internet (because it's more reliable than the OOB link, so you get fewer problems where it can't sync) and then you terminate the OOB Internet link for the badge system (on the basis that the DC has redundant fibres and routers, so won't go down, whereas the OOB link is a consumer link without redundancy). And then you update the config management system so that, for all but legacy systems (like the DC routers), if you don't confirm that a change is good to test within 5 minutes, it autoreverts, so people develop a habit of testing rather than carefully confirming that they've not made a mistake.

All these decisions sound reasonable in isolation, but they chain together to a world where a mistake with the DC's router configurations result in the door badge system locking you out.

Fediverse notifications

jzb — Mon, 28 Jul 2025 15:51:32 +0000

Yes; we have realized that replies are not visible from the profile page unless one clicks on "Posts and Replies". We'll be working on more redundancy and better notifications in the event this happens again.

Fediverse notifications

mote — Mon, 28 Jul 2025 15:51:17 +0000

> posted as replies rather than a new post, Mastodon seems to like to hide those

Hopefully constructive feedback from a very lightweight Masto user (lurker/reader) - the problem is incoming information overload for the amount of time I want to devote to the platform; I quickly found that I needed to disable all replies and boosts in my main feed. Reply content is never seen unless I intentionally click-to-open a post and read the replies. My personal usage mirrors what you've felt as a content publisher. $0.02 hth

Backup providers

jorgegv — Mon, 28 Jul 2025 15:48:37 +0000

+1 for Hetzner.

Pretty much the same good experience here, for about 15v years now and counting.

Dedicated servers are top-notch and quite cheap.

And great support.

Fediverse notifications

zdzichu — Mon, 28 Jul 2025 15:43:35 +0000

This post is not visible on https://fedi.lwn.net/@lwn . Visible posts are, at this moment: Security updates for Monday; LWN is back; 6.16 is out; Rethinking confidential VMs. Last post, about rethiking, is from "3d ago".

Backup providers

imcdnzl — Mon, 28 Jul 2025 15:32:02 +0000

OVH had a massive fire a couple of years back (just search for OVH fire) and they didn't have backups and a lot of customers lost all their data - servers you can replace more easily, but not data.

They're probably more safer now I guess though as they would have learnt from that.

I think key is to have backup of all data / server config in another region whether you use a small provider or one of the big hyperscalers.

Backup providers

burki99 — Mon, 28 Jul 2025 14:46:01 +0000

I've been with Hetzner for over 20 years - first dedicated servers, now cloud instances. The experience has been flawless, the few times I needed support it was quick and knowledgeable, and the pricing has always been very reasonable.

Backup providers

koverstreet — Mon, 28 Jul 2025 14:23:28 +0000

I shifted all of my stuff to Hetzner awhile back. Data point of 1, and I'd love to hear from others, but so far I've been impressed.

Sounds like double ungood day for Linode

paulj — Mon, 28 Jul 2025 14:19:23 +0000

It's like the time Facebook had a DC go down. Someone had to physically go from the nearest Prod. Eng. office to the DC to start the initial steps for bring-back. When they arrived they found they couldn't get in - the door badge-entry system was out, cause it depended on the DC. They + DC engineers had to resort to some good old physical mallet based engineering to hack their way in.

Backup providers

lyda — Mon, 28 Jul 2025 14:16:51 +0000

Living in the EU, I agree. I'm looking at OVH and scaleways. Both are French.

Backup providers

dskoll — Mon, 28 Jul 2025 14:09:51 +0000

I don't know if you're looking to diversify providers, but these are two that I use that I've had good luck with:

OVHCloud. I have a virtual machine in one of their Montreal data centres.
Luna Node. I have a VM in one of their Toronto data centres. (Unlike OVH, Luna Node doesn't own the data centre; they lease space from the owner.)

I'm in Canada, so having my data in Canada was important to me. If, for similar reasons, you want to say in the US, I'm pretty sure OVH has US data centres.