Freedesktop looking for new home for its GitLab instance [LWN.net]

Visitors to the freedesktop.org GitLab instance are currently being greeted with a message noting that the company who has been hosting it for free for nearly five years, Equinix, has asked that it be moved (or start being paid for) by the end of April. The issue ticket opened by Benjamin Tissoires in order to track the planning of a move is clear that the project is grateful for the gift: "First, I'd like to thank Equinix Metal for the years of support they gave us. They were very kind and generous with us and even if it's a shame we have to move out on a short notice, all things come to an end."

The current cost for the services, much of which is for 50TB of ~~bandwidth~~ data transfer per month and a half-dozen beefy servers for running continuous-integration (CI) jobs, comes to around $24,000 per month. Tissoires believes that the project should start paying for service somewhere, in order to avoid upheaval of this sort, sometimes on short or no notice. "I personally think we better have fd.o pay for its own servers, and then have sponsors chip in. This way, when a sponsor goes away, it's technically much simpler to just replace the money than change datacenter." Various options are being discussed there, but any move is likely to disrupt normal services for a week or more.

to post comments

50TB of bandwidth?

Posted Jan 30, 2025 17:25 UTC (Thu) by rds (subscriber, #19403) [Link] (11 responses)

Did that really use 50TB of bandwidth or did they transfer 50TB of data each month? Because if its the former that's impressive. If its the latter then my car used 35mpg of gas last month.

50TB of bandwidth?

Posted Feb 4, 2025 15:40 UTC (Tue) by mbunkus (subscriber, #87248) [Link] (10 responses)

I've just read the same verbiage in Alpine Linux's announcement[1], too:

> The T1 mirrors require 5TB of disk space each and currently use approximately 800TB of bandwidth per month.

Same provider for them. So I wonder if this is something that Equinix themselves used in their communication with the projects they're now canceling.

I fear this is a fight we might end up losing, similar to how it's basically a crapshoot what a person means when you hear them say "literally".

[1] https://www.alpinelinux.org/posts/Seeking-Support-After-E...

50TB of bandwidth?

Posted Feb 4, 2025 16:10 UTC (Tue) by Wol (subscriber, #4433) [Link] (7 responses)

If you read the Alpine post, it sounds like Equinix are pulling the plug on their data centres and going "virtual". So the physical location where they were hosting everyone will soon no longer exist ...

Cheers,
Wol

50TB of bandwidth?

Posted Feb 4, 2025 18:52 UTC (Tue) by zdzichu (subscriber, #17118) [Link] (6 responses)

"Bandwidth" is expressed in units of data per units of time. Like megabits per second. What both announcements have in common is that they use word "bandwidth" to mean "transfer". "Transfer" or "quota" is expressed in units of data (no time component).

So the suspicion is that hosting provider termination letters used "bandwidth" incorrectly and it got copy-pasted into projects announcements.

50TB of bandwidth?

Posted Feb 6, 2025 14:31 UTC (Thu) by mgedmin (guest, #34497) [Link] (5 responses)

80 TB per month sounds like a unit of data (80 terabytes) per a unit of time (month). What's the issue?

50TB of bandwidth?

Posted Feb 6, 2025 14:50 UTC (Thu) by Wol (subscriber, #4433) [Link]

When you leave out the unit of time. At which point you fall foul of the advice I was always given in Physics Multiple Choice exams - the FIRST thing you do is check the units. If they don't match either side of the equals, the answer has to be wrong. If you're lucky, there's only one answer left so you don't have to bother checking the numbers.

It's the same thing as people assuming a lightyear is a measure of time, because a year is a measure of time. If the units don't match, the answer is garbage.

Cheers,
Wol

50TB of bandwidth?

Posted Feb 6, 2025 17:20 UTC (Thu) by mbunkus (subscriber, #87248) [Link] (3 responses)

The text didn't say "50 TB per month", it said "50 TB of bandwidth per month", which is like saying "I drive 300 miles of speed per month". It's "50 TB of amount of data per month per month". No, you use an _amount_ of data per month, and you drive an _amount_ of miles per month, not a throughput (bandwidth = amount of data per time) or a speed (distance per time). The Alpine Linux post used the same verbiage:

> currently use approximately 800TB of bandwidth per month

Like Wol said, if the left & right parts of a statement use different units (left part: amount, right part: amount per time) they cannot go together. It simply makes no sense.

50TB of bandwidth?

Posted Feb 6, 2025 18:48 UTC (Thu) by zdzichu (subscriber, #17118) [Link] (2 responses)

Exactly. If not for surplus "bandwidth", those notes would be sensible although unusual. 50TB / month is unwieldy, but corresponds to about 150 Mbps.

50TB of bandwidth?

Posted Feb 6, 2025 20:06 UTC (Thu) by intelfx (subscriber, #130118) [Link] (1 responses)

> 50TB / month is unwieldy, but corresponds to about 150 Mbps

Nit: in context of hosting costs, you're probably paying for _traffic_ rather than bandwidth, so a "50 TB/mo" figure is likely more of a "preferred unit" than a 150 Mbps average.

50TB of bandwidth?

Posted Feb 6, 2025 21:39 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

That depends on the hosting. With AWS you're paying for TB, but with classic colocation you typically pay for megabits per second. Either just for the static allocation, or for the average use.

50TB of bandwidth?

Posted Feb 26, 2025 8:16 UTC (Wed) by daenzer (subscriber, #7050) [Link] (1 responses)

While the wording might be sloppy, "TB per month" is technically a unit of bandwidth, and "x TB/<unit of time> per month" wouldn't make sense, so it seems pretty clear what's meant.

50TB of bandwidth?

Posted Feb 26, 2025 10:14 UTC (Wed) by mbunkus (subscriber, #87248) [Link]

In hosting there are usually two billing models, one based on sustained or peak bandwidth usage & one based on amount of (most often outgoing) traffic. I consider it to be important to be unambiguous & correct when looking to host things elsewhere in order to estimate the cost correctly.

$24,000 per month?

Posted Jan 30, 2025 18:46 UTC (Thu) by dskoll (subscriber, #1630) [Link] (23 responses)

$24K per month seems very expensive, unless the beefy servers are really and truly beefy.

You can get some pretty decent dedicated servers from OVH, for example, with no monthly data transfer limits, for around $350 - $500/month and that's $CAD.

$24,000 per month?

Posted Jan 30, 2025 18:59 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (2 responses)

That feels…very expensive to me too. Pegging a 250W PSU 24/7 at 24¢ / kWhr (seems to be about the NY annual average) is $43 / month. Buy 10 $2000 machines for the current monthly cost and you're left with 2 years of electricity costs left over. Of course, that is without a GPU which some projects fd.o probably want, but that seems…within that cost envelope.

$24,000 per month?

Posted Jan 30, 2025 19:22 UTC (Thu) by dskoll (subscriber, #1630) [Link] (1 responses)

Yeah, buying machines and then just paying for rack space, power and connectivity might be quite a bit cheaper. But then if there's a hardware problem, you have to deal with it. Might be worth renting dedicated servers to offload the hardware maintenance responsibility.

$24,000 per month?

Posted Feb 1, 2025 17:28 UTC (Sat) by ibukanov (subscriber, #3942) [Link]

Typically servers for development tooling can afford an outages for a day or two, sufficient to replace the faulty hardware. One can also have separated much smaller/cheaper server to use when the main server crash with some downgrade in functionality, like no CI jobs and read-only Git access.

On my previous work one can clear see how having own hardware and just renting the space in a data center allowed to save a lot of money. The company also had couple of IP guys who lived within couple of hours of driving from that place.

$24,000 per month?

Posted Jan 30, 2025 20:55 UTC (Thu) by MarkVandenBorre (subscriber, #26071) [Link] (4 responses)

I assume freedesktop.org know what they need. I don't.

What I do know is I won't touch OVH with a ten foot pole for critical applications. Been there, done that.

$24,000 per month?

Posted Feb 2, 2025 2:16 UTC (Sun) by marcH (subscriber, #57642) [Link] (3 responses)

Can you elaborate, some figures maybe?

As others have mentioned, most open-source projects can tolerate outages as long as they are limited in time (frequency and duration); "five 9s" not required.

$24,000 per month?

Posted Feb 2, 2025 5:48 UTC (Sun) by MarkVandenBorre (subscriber, #26071) [Link] (2 responses)

Not sure how relevant my anecdote is relevant in an article about freedesktop's choices, but since you ask for more detail, here is one personal OVH anecdote.

I was running a small business and had to have a less common domain name renewed (small EU country tld). OVH had charged my renewal to my credit card successfully, but the domain name never got renewed. Warnings started spewing directly from the tld registry about expiry approaching. Naive me lost a week proactively communicating with OVH in every possible way. That failed.

In the end, I solved things by physically sending a local acquaintance to the tld registry with cash for moving my tld directly to them.

That's just a personal anecdote. I'm sure many seriously considering them have read about OVH's Strasbourg DC fire and more. I wouldn't wish my worst enemy to have to deal with subpar support like this, let alone a FOSS project in good standing.

OVH (was $24,000 per month?)

Posted Feb 2, 2025 15:02 UTC (Sun) by dskoll (subscriber, #1630) [Link] (1 responses)

I've had a VPS in OVH's Montreal data centre for going on 9 years and I've found their support to be pretty good and the service trouble-free. I think customer experiences with most providers can be all over the map, depending on how lucky or unlucky you are.

OVH (was $24,000 per month?)

Posted Feb 2, 2025 15:48 UTC (Sun) by pizza (subscriber, #46) [Link]

After about 3.5 years as an OVH customer, the the few times I've needed OVH's support (~once per year) it's been both timely and excellent.

$24,000 per month?

Posted Jan 30, 2025 22:38 UTC (Thu) by lunaryorn (guest, #111088) [Link] (9 responses)

I'm not sure if I read the ticket correctly but as far as I can see FDO runs two 80 core ARM runners and three 32 core x86 runners, all under heavy use apparently.

I don't think you can get that for 500$; you probably can't even easily rent that much ARM power at any of the usual cloud providers.

$24,000 per month?

Posted Jan 30, 2025 23:32 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

You can build this kind of hardware for the cost of 1-2 months!

$24,000 per month?

Posted Feb 4, 2025 10:02 UTC (Tue) by geuder (subscriber, #62854) [Link] (1 responses)

Not "you", but "very few people with the right skillset".

Most of those people don't have a lot of freetime and/or do not want to commit to such project for free.

That's why "you" often pay a premium to a company for doing it.

$24,000 per month?

Posted Feb 4, 2025 10:44 UTC (Tue) by intelfx (subscriber, #130118) [Link]

You can also pay a fee to a "person with the right skillset" (i.e., hire one) to do the building for you. And a (likely smaller) fee to another person when the need for maintenance arises. It'll still be much less than paying $24k/mo to a cloud behemoth...

$24,000 per month?

Posted Jan 31, 2025 3:39 UTC (Fri) by WolfWings (subscriber, #56790) [Link] (2 responses)

I mean... it depends on a lot.

A 56-core R730 w/ 128GB of RAM and no drives is about $500 used at this point.

Buy four, N+1 redundancy on parts, buying drives new and you're only in for about $5K even if you opt for SSDs.

ARM is the sticker TBH, there's basically nothing besides literally RPis and the variations on the theme unless you drop six digits for the hardware, it's bonkers, and mostly focused at generative-AI stuff at this point from NVidia.

$24,000 per month?

Posted Jan 31, 2025 7:42 UTC (Fri) by hrw (subscriber, #44826) [Link]

Ampere Altra with 64 cores is 1500 USD at Newegg for motherboard and cpu.

Then DDR4 ram, storage, case etc.

$24,000 per month?

Posted Feb 5, 2025 20:19 UTC (Wed) by DemiMarie (subscriber, #164188) [Link]

I suspect FDO cannot use used hardware for security reasons. Someone who compromised it would be able to inject a backdoor into Mesa and libwayland, which would be an absolute disaster.

$24,000 per month?

Posted Jan 31, 2025 7:35 UTC (Fri) by tlamp (subscriber, #108540) [Link]

At work, we bought a new, beefy build and CI server just this month for about €12 000,00

It consists of an Epyc 9475F (48 CPU cores, 96 threads, 3.65 GHz base clock, 4.8 GHz boost), 384 GiB DDR5 RAM, all flash storage Kioxia CM7-V (2.45 million IOPS, 14 GB/s read, 6.75 GB/s write bandwidth) U.2/NVMe attached SSDs.

FD.o might need a bit more memory or the like, but IMO such a server can handle most build and CI jobs quite fine and in a timely manner.

FD.o could get about two of them for a single monthly Equinix rate, with three monthly rates they could easily cover the rest of the HW and some more, and that's a one time cost (well, won't last forever, but 5+ years are very likely and 10+ are also likely, might need to replace the power supply once then, maybe the disks depending on how much they get trashed).

The perpetual cost is then renting a 1/3 of a rack at some reputable hosting company/datacenter to host these serve, which might be around for a 100 to 500 €/month depending on location and guaranteed uplink speed (1 Gbps is standard nowadays, more becomes expensive fast though).

The underlying story here is that server rental, be it bare-metal or virtual machine, is squeezing out quite a bit of buck, albeit that still is orders of magnitude lower compared to the SaaS offerings; which is basically only cheap as long as you programs/infrastructure have almost nothing to do anyway, otherwise some owned (even rented) servers and hiring an admin comes always out cheaper.

$24,000 per month?

Posted Jan 31, 2025 10:15 UTC (Fri) by daniels (subscriber, #16193) [Link]

We also run a GitLab instance with 100k active users, hosting all the development of GStreamer, libinput, Mesa, NetworkManager, PipeWire, PulseAudio, Wayland (including Weston and wlroots), X.Org, plus the DRM and V4L2 kernel subsystems.

$24,000 per month?

Posted Feb 3, 2025 13:59 UTC (Mon) by greggyb (subscriber, #173803) [Link]

Hetzner offers 80-core ARM w/ 256GiB RAM and 2x3.84 TiB NVMe starting at 244/mo. RAM and storage are expandable.

The only 32-core x86 offerings are specialized (maybe in ways FDO wants, but I can't say), but you could match the cores with 2x 48-core servers.

48-core x86 w/ 256 GiB RAM and 2x1.92 TiB starts at 221/mo.

If remaining hosted, Hetzner is probably the way to go for any chunky compute.

$24,000 per month?

Posted Jan 31, 2025 10:13 UTC (Fri) by daniels (subscriber, #16193) [Link] (4 responses)

It's actually very cheap, because it cost us $0.

The Equinix Metal rack rate isn't particularly relevant, because for five years they gave us a great service for free. We could probably halve our costs by moving to a different provider, sure. But without that sponsorship, even a cheaper provider would've cost roughly $600k over the five years. Instead we were able to use that money on things like XDC, sponsoring travel to bring more people into the community, internship programs, etc, which seems like a good outcome to me.

They've also given us nearly six months of notice to be able to find an alternate provider and move our services across, which is completely reasonable.

$24,000 per month?

Posted Jan 31, 2025 14:50 UTC (Fri) by ATLief (subscriber, #166135) [Link] (3 responses)

I think the impression of their generosity depends on how much of that cost they were able to write off as a charitable donation. If they’re counting it by the retail price and it has an 85% markup, then they’d still save more money than they’d lose with that arrangement.

If they were just completely eating the cost with no write-offs, then I agree it was very generous.

$24,000 per month?

Posted Feb 1, 2025 0:34 UTC (Sat) by daniels (subscriber, #16193) [Link] (2 responses)

I don’t know anything about their tax arrangements. There is certainly a gap between cost and price of their service, because otherwise it wouldn’t be a viable business. And there is a marketing benefit to them, because they are visibly supporting open-source services.

So clearly it worked for both, but from the point of view of fd.o, it has saved us hundreds of thousands of dollars (assuming a much more economical operation on a much cheaper provider), which has allowed us to provide a great service whilst spending the money we do have on other things. That’s why I’m grateful to them.

$24,000 per month?

Posted Feb 1, 2025 19:02 UTC (Sat) by anholt (subscriber, #52292) [Link] (1 responses)

It was wild being in the org and seeing Equinix just keep loaning us an incredible amount of hardware. Tremendous thanks to them for their years of sponsorship.

I see a lot of people here speculating about "you could just buy some hardware and stick it on a network!" and it's so tempting, until you actually face the issues of hardware reliability and reprovisioning. With a service we can just nuke an instance and spin up a new one, compared to getting someone to go down to our host and pull a machine out of the rack and try to debug it and replace parts and put it back and see if it wrecks our ability to merge code again. Having run 1 to 3 of the bare-metal hardware farms we used for GPU testing, the admin overhead of maintaining hardware and the risk to the productivity of all the developers is so much more than it feels like it should be.

If someone still thinks it's easy, I've got an idle rack of hardware I'd love to see maintained. ;)

$24,000 per month?

Posted Feb 1, 2025 19:45 UTC (Sat) by dskoll (subscriber, #1630) [Link]

I started this thread, and my intention was certainly not to minimize the valuable contribution made by Equinix.

That said, I did run an email security service for quite a few years on 13 servers that my company owned, colocated in a nearby data center. In my experience, keeping the hardware going was not that onerous. Our hardware contained a lot of spinning disks, so we'd have disk drive die on us every month or two; I'd drive to the data center and replace it. The disks were all hot-swappable, so actually replacing a disk took about a minute.

Maintaining hardware in a data center that's not an easy drive away is indeed a pain. We used to have a few servers in a data center 200km away and decided that the geographic redundancy was not worth the hassle, unless you can contract with the data center to do "remote hands" routine maintenance like disk drive replacement.

Alpine Linux

Posted Feb 4, 2025 12:45 UTC (Tue) by pabs (subscriber, #43278) [Link]

Alpine Linux is in a similar position:

https://www.alpinelinux.org/posts/Seeking-Support-After-E...

patchwork.freedesktop.org?

Posted Feb 4, 2025 18:28 UTC (Tue) by geert (subscriber, #98403) [Link]

Does this include patchwork? I always had mixed feelings about DRM using in their git commits Link:-tags pointing to https://patchwork.freedesktop.org/ instead of https://lore.kernel.org/, like most other subsystems use...

P.S. https://patchwork.freedesktop.org/ lists several instances (most tagged "DEPRECATED - USE GITLAB", oh the irony).

24k/mo for 6 CI servers and 50TB data transfer is crazy

Posted Feb 13, 2025 4:19 UTC (Thu) by hexhu (guest, #127703) [Link]

It only costs $400-$800 in cheaper providers like Hurricane Electric (400-600 for a full rack w/ 1gig port) if they bring their own servers, and peering at FCIX.net is free