The road to Zettalinux [LWN.net]

The road to Zettalinux

Posted Sep 16, 2022 13:20 UTC (Fri) by ianmcc (subscriber, #88379)
Parent article: The road to Zettalinux

Is it April First already? 2^64 is a significant fraction of Avagadro's number - it will never be possible to build a device that stores that much data in a small enough package as to make it worthwhile to directly address by a CPU. If you wanted to virtualize the entire internet and map it into a single address space then you need more than 64 bits. But needing more than 64 bits for the locally addressable memory for a CPU is going to run into physical limitations WELL before we get to 64 bits.

to post comments

The road to Zettalinux

Posted Sep 16, 2022 13:38 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (4 responses)

Isn't the problem that, although the pointer data size is 64 bits, only 48 bits are actually used for addressing today? The rest either are already spoken for or are being given over to userspace[1]?

[1]https://lwn.net/Articles/902094/

The road to Zettalinux

Posted Sep 16, 2022 14:13 UTC (Fri) by ianmcc (subscriber, #88379) [Link] (3 responses)

Right, so what is wanted is a decorated or tagged pointer, and hardware support for that requires that the addressable space is smaller than the pointer size. Don't pitch it as needing more than 64 bit address sizes, pitch it as needing hardware support for decorated pointers.

The road to Zettalinux

Posted Sep 17, 2022 21:41 UTC (Sat) by willy (subscriber, #9762) [Link] (2 responses)

The presentation calls for 128-bit registers in CPUs with fairly limited ALU support (addition, subtraction, shifts). That's what we need to avoid LFS2 and HIGHMEM

The road to Zettalinux

Posted Sep 19, 2022 9:35 UTC (Mon) by arnd (subscriber, #8866) [Link] (1 responses)

I think there is a conflict between the ideas of making 'long' 128-bit wide, using the same C calling conventions as 128-bit user space, and not requiring 128-bit multiplication and division to be fast.

We'll probably keep having this discussion for a while before even the basic type model is decided, including input from more than just kernel developers as well as actual benchmarking of user space software. My guess is that we'll end up with an LLP128 model (128 bit long long, 64 bit long) in both user space and kernel space, as is proposed in the draft riscv128 specs, which in turn means more invasive kernel changes but keeps the types compatible and does not require expensive 128-bit ALUs.

The road to Zettalinux

Posted Sep 19, 2022 16:49 UTC (Mon) by willy (subscriber, #9762) [Link]

I do think we'll end up with different conventions in userspace and kernel space. That's OK, as long as we have an API that can describe what's going on.

You make a good point that 128b x 128b is going to have to work. However, there are algorithms for this kind of multiplication & division that allow for early termination if the top bits are zero.

The road to Zettalinux

Posted Sep 16, 2022 14:16 UTC (Fri) by atnot (guest, #124910) [Link] (9 responses)

I don't think memory with more than 2^64 locations will exist any time soon either. But that does not mean that 2^64 bits is enough.

The major improvement between 32 and 64 bit was not just allowing more memory to be addressed, but that addresses ceased being a scarce resource. This means that today, you can do things like mmap nearly unlimited files regardless of their size, with ASLR, store additional metadata in your pointers and *still* have more than enough bits to address everything you could ever want without even having to think about fragmentation.

That's where the true crunch will be felt IMO, and it's going to be far sooner than the availability of hypothetical 2^64 or even 2^48 byte memory sticks. Terrabyte file sizes are already common, exabytes are a meaningful unit and things like nvmeof or cxl are likely set to allow a single computer to access far more storage than it's ever been able to. A world where virtual addresses might become scarce again is definitely closer than one would think.

The road to Zettalinux

Posted Sep 16, 2022 15:22 UTC (Fri) by linuxrocks123 (subscriber, #34648) [Link] (6 responses)

Is that a problem worth solving, though? Do we really want every pointer to be four times as big as it needs to be just so someone can mmap a whole hard drive? I don't think so.

"Four times as big" because x32 demonstrates that, if we try to squeeze, 32 bits is still enough for a userspace address space. The fact that x32 didn't take off demonstrates that it's not worth the effort to bother saving that memory, but I'd still rather not see us make pointers even fatter.

The road to Zettalinux

Posted Sep 16, 2022 16:03 UTC (Fri) by epa (subscriber, #39769) [Link] (5 responses)

Perhaps 'near' and 'far' pointers will make a comeback?

The road to Zettalinux

Posted Sep 16, 2022 18:41 UTC (Fri) by Wol (subscriber, #4433) [Link] (4 responses)

Segmented memory? I was thinking that ...

Cheers,
Wol

The road to Zettalinux

Posted Sep 18, 2022 9:43 UTC (Sun) by jengelh (subscriber, #33263) [Link] (3 responses)

Segmented memory sucked. You had two 16-bit registers, but rather than allowing to reach 32 bits worth of address space, the x86 implementation of it only gave you 20 bits. What a waste. Wouldn't want to repeat that.

The road to Zettalinux

Posted Sep 18, 2022 10:45 UTC (Sun) by Wol (subscriber, #4433) [Link]

That was Intel. I used 50-series which also had segmented memory, and while I know far less about the silicon internals, I gather it worked very well.

It didn't migrate very well to Intel, and quite possibly because their segmentation was (compared to 50-series) broken.

I know it was used for security, for separating the OS from user-space, and from separating users/processes from each other.

Cheers,
Wol

The road to Zettalinux

Posted Sep 19, 2022 12:51 UTC (Mon) by eru (subscriber, #2753) [Link] (1 responses)

You are thinking of real-mode segmentation in 16-bit x86. In 386 and later protected mode, segments can be up to 4G, you have two additional segment registers (FS and GS), and addressing happens via a descriptor table, where you can set protections and the size of the segment, getting an exception if a pointer accesses outside it. I have worked with an inhouse OS that actually used these features.

The road to Zettalinux

Posted Nov 20, 2022 20:43 UTC (Sun) by nix (subscriber, #2304) [Link]

386 and later protected mode, segments can be up to 4G, you have two additional segment registers (FS and GS), and addressing happens via a descriptor table, where you can set protections and the size of the segment, getting an exception if a pointer accesses outside it. I have worked with an inhouse OS that actually used these features.

This was true on 386 and later in real mode too: you just had to set things up right in protected mode first (basically setting up the selector bounds right before transitioning back). Of course, this 'unreal mode' wasn't too useful, given that unless you were stuck in DOS you might as well just use protected mode, but it was there. There were really very few differences between real and protected mode on the 386: real mode was protected mode in a very, very thin disguise. You could use the full register file (%eax et al) and everything. Long mode on x86-64, now, *that's* different.

The road to Zettalinux

Posted Sep 16, 2022 22:59 UTC (Fri) by bartoc (guest, #124262) [Link] (1 responses)

Similar to the rationale for why ipv6 picked 128bit addresses instead of like 48bit.

The road to Zettalinux

Posted Sep 17, 2022 6:56 UTC (Sat) by gdt (guest, #6284) [Link]

IPv6 was designed for 80 bits for network addressing (with 16 bits expected to be used for addressing within a site) and 48 bits for the autoconfiguration feature (basically, the ethernet MAC address, the longest station address of the era). More than a third of the network addressing was expected to be used for better network aggregation than IPv4; that is, to trade off the number of addressable networks for improved stability and small global routing table sizes.

In practice, much of that design had to be altered in practice.

Starting with IEEE pointing out their next-generation station addressing for their future ethernet follow-on protocol. This 'EUI' used 64 bits for station addressing. So barely was the design finalised before the intended addressing had to be altered to the (64, 64) split between network and station addressing which we have now.

I can't think of a single ISP which has used IPv6's in-service network prefix change feature, which was meant to allow even huge ISPs to advertise just one prefix. So plans for route aggregation haven't been as successful as hoped.

To reduce configuration error, routers moved away from being able to identify single bits in addresses (eg, moving from the 'subnet mask' notation to the 'prefix length' notation). This means that it's hard for sites to use their subnetting allocation to carry meaning (such as a particular bit-pattern meaning "no egress"). Whereas it's still possible to pick out selected bits in CPU's virtual addresses.

The road to Zettalinux

Posted Sep 16, 2022 17:47 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (16 responses)

> Is it April First already? 2^64 is a significant fraction of Avagadro's number - it will never be possible to build a device that stores that much data in a small enough package as to make it worthwhile to directly address by a CPU.

You can rent a computer with 128TB of RAM right now if you ask the right folks nicely. And you can get a 25TB computer on AWS with a couple of clicks and a deep enough credit card. That's 2^47 bytes already directly adressable, and in a single system. This very moment, no magical technology needed (well, not more magical than modern semiconductor production).

And since we need at least 1 bit for user/kernel separation, this leaves us with just 16 bits in reserve. And this is for RAM, not persistent storage. There are already storage arrays that have almost as much space, so you can conceivably exceed the 2^64 limit by memory-mapping a huge file.

The road to Zettalinux

Posted Sep 16, 2022 19:23 UTC (Fri) by mb (subscriber, #50428) [Link] (15 responses)

> this leaves us with just 16 bits in reserve

This is over 65000 times of what you currently have.

The road to Zettalinux

Posted Sep 16, 2022 19:41 UTC (Fri) by Wol (subscriber, #4433) [Link] (2 responses)

Not if those bits get used for flags ...

Cheers,
Wol

The road to Zettalinux

Posted Sep 17, 2022 11:53 UTC (Sat) by milesrout (guest, #126894) [Link] (1 responses)

Memory addresses are for addressing memory. If you want space for extra flags, create a struct with a 'flags' member.

The road to Zettalinux

Posted Sep 17, 2022 21:21 UTC (Sat) by colejohnson66 (subscriber, #134046) [Link]

Tell that to Intel and AMD who are proposing to let programs use those bits for whatever they want. As if they completely forgot the entire purpose of "canonical" addresses and "32 bit clean" programs.

The road to Zettalinux

Posted Sep 16, 2022 20:24 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (11 responses)

The 128TB was for a "mass market" machine. I looked up supercomputers and apparently we are already at around 4PB size (HPE Frontier). So we're at 2^52 already, giving us just 11 bits of leeway.

If we double the capacity every 2 years, that's 22 years from now. And we'll be getting uncomfortable with the address space exhaustation long before we reach the actual limit.

And its storage is at 250PB, so that's 2^58 if you want direct RAM mapping of it.

So yep, we should start to plan for 128 bit now. It's not a pressing concern by any means, but it's something that we should start taking into account for future long-term designs.

The road to Zettalinux

Posted Sep 17, 2022 4:23 UTC (Sat) by linuxrocks123 (subscriber, #34648) [Link] (10 responses)

Moore's Law is dead and buried, so it'll probably be more than 22 years. That said, if you're insane enough and have enough money, you can of course wire together as much RAM as you want into a single machine. So, eventually, perhaps someone will want a 128-bit Linux distro.

I just hope the rest of us can keep using 64-bit. As someone who very rarely has a need to run programs that take more than 4GB of memory on his personal machines, I am miffed that I need to pay the cost of 64-bit pointers everywhere. As the author of a malloc() implementation that would work extremely well if x64 pointer tagging were a thing, though, I think I'm going to be glad 64-bit pointers are the standard quite soon.

The road to Zettalinux

Posted Sep 17, 2022 4:35 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (6 responses)

> Moore's Law is dead and buried

It's not. It just became more step-like, rather than gradual.

EUV litography took a lot of time to be developed, but now it can work almost all the way to atomic level.

The road to Zettalinux

Posted Sep 19, 2022 17:34 UTC (Mon) by linuxrocks123 (subscriber, #34648) [Link] (5 responses)

https://en.wikipedia.org/wiki/Transistor_count

Go back to 2010. Most dense released processor had a transistor density of 4,875,000 transistors per square millimeter. According to Moore's Law, processors in 2022 should have pow(2,6)=64 times as many transistors per square millimeter today. That would be 312,000,000 transistors per square millimeter. Reality? 139,300,000 per square millimeter.

It's dead, Jim.

The road to Zettalinux

Posted Sep 19, 2022 18:30 UTC (Mon) by malmedal (subscriber, #56172) [Link]

Hmm, originally Moore's law said doubling every 12 months, then it was revised plus six to 18 months, then another six more to 24 months and by your numbers we are apparently now at 30 months doubling time.

The road to Zettalinux

Posted Sep 19, 2022 19:51 UTC (Mon) by Wol (subscriber, #4433) [Link]

So we've missed it by about ONE cycle in TEN years.

If we've missed it, it's "close, have a cigar".

Cheers,
Wol

The road to Zettalinux

Posted Sep 20, 2022 9:59 UTC (Tue) by paulj (subscriber, #341) [Link] (2 responses)

Moore's "law" was about # transistors on a device though, not density per se. If a company can get more transistors on a device with improved processes that deliver higher yield, and so can produce a device with more transistors by virtue of being able to produce a larger device for essentially the same cost, then that contributes to Moore's "law", regardless of density.

The road to Zettalinux

Posted Sep 20, 2022 10:02 UTC (Tue) by paulj (subscriber, #341) [Link]

Subject for discussion: Does 'chiplet' technology count towards Moore's "law"? Should a set of chiplets, bound together on a single silicon substrate signalling and power/clock, be considered one 'device' for the purposes of Moore's law?

Cause that technology has enabled manufacturers to produce GPUs and CPUs with far greater numbers of transistors than would have been (economically) possible before...

The road to Zettalinux

Posted Sep 20, 2022 11:05 UTC (Tue) by farnz (subscriber, #17727) [Link]

It wasn't even about # transistors on a device - it was about # transistors you could affordably put on a single device. In its original form, Moore speculated that the number of transistors you could get on one device at the cheapest cost-per-transistor would double every 18 months - you can get this by doubling the number of transistors you put on a device while holding the price fixed, or you can halve the cost per transistor and still meet the original formulation - indeed, if you look at the original article by Dr Moore, the graph on page 2 suggests that even when he came up with the rule, it was already the case that you got both reduced cost-per-transistor and increased density working together to make the rule true.

People forget that Moore's Law is purely about the economics of producing more complex devices - for as long as it holds, it's a good engineering tradeoff to assume that, if you can only afford $50 per IC manufactured, you'll be able to buy the equivalent of today's $100 one in 18 months (or whatever the current Moore's Law timeframe is) and thus can plan for being able to afford that complexity, rather than trying to make your design fit within the $50 choice.

But with Dennard scaling coming to an end (because leakage currents can no longer be neglected), this doesn't necessarily translate into improved performance, or improved energy consumption - just into reduced price for devices of a given complexity. This has been most evident in games console pricing - at launch, they're using devices that are slightly too expensive for the product, preventing the console maker from profiting on the hardware, but within 5 years, you're getting cost-reduced versions where the console maker is making a profit on the reduced price version.

The road to Zettalinux

Posted Sep 22, 2022 0:44 UTC (Thu) by mirabilos (subscriber, #84359) [Link] (2 responses)

Yeah, x32 (amd64ilp32) was a nice thing, especially to reduce web browser memory consumption by naturally giving it only 4 GiB minus a few KiB to work in, while at the same time letting it keep the wider and more registers amd64 has.

Wish it had taken off better. I did run it on a machine until I switched all my Debian systems from sid to bullseye to avoid the UsrMove fiasco.

The road to Zettalinux

Posted Sep 22, 2022 1:41 UTC (Thu) by pabs (subscriber, #43278) [Link] (1 responses)

Avoiding usrmerge is fairly easy using equivs and or the usrmerge config file.

https://github.com/jwilk/usr-is-not-merged/
https://sources.debian.org/src/usrmerge/31/debian/README....

The road to Zettalinux

Posted Sep 22, 2022 15:12 UTC (Thu) by mirabilos (subscriber, #84359) [Link]

Yeah, but it’s unsupported now, which is a PITA. Plus, it will not work right, but kilobyte already wrote about that enough and is… suppressed.

I even consider the forced UsrMove a violation of SC§4, but what do I know…

The road to Zettalinux

Posted Sep 19, 2022 0:57 UTC (Mon) by developer122 (guest, #152928) [Link]

2^64 bytes is only 18,000 petabytes. We've been able to fit 1PB in 1U for many years now.

Have a few thousand 1PB storage servers in your datacenter and want to address them all? You have a problem.

The road to Zettalinux

Posted Sep 19, 2022 19:27 UTC (Mon) by wvaske (guest, #114127) [Link]

CLX 3.0 (switched fabrics) get us to datacenter-scale memory and storage accessible by a single 'host'. The architecture changes we'll see with CXL are likely to drive us to zettabyte scale memory sooner than we all expect.