LWN: Comments on "An end to high memory?"

An end to high memory?

ringerc — Mon, 30 Jan 2023 01:57:11 +0000

Beware of removing high memory.

The 16 Exabyte boundary will be a problem before we know it 😀

An end to high memory?

neilbrown — Sun, 08 Mar 2020 23:46:59 +0000

> Any idea what devices are actually affected by this?

I know about precisely one MIPS board - the MT7621A that you have already mentioned. So you already know more than me :-(

Maybe ask on linux-mips ??

An end to high memory?

arnd — Fri, 06 Mar 2020 14:19:28 +0000

Any idea what devices are actually affected by this?

As far as I can tell, most MIPS32r2 based chips are limited to less than 512MB anyway by having only a single DDR2 channel, your MT7621A being a notable exception that has up to 512MB DDR3.

Out of the chips that support more than 512MB, the ones based on MIPS32r3 or higher (Baikal, Intel/Lantiq, ...), may be able to use th eextended virtual addressing to allow larger direct lowmem mappings with a bit of work.

The Creator CI20 is one that has 1GB of RAM on MIPS32r2, and I can see that the bmips defconfigs enable highmem support, so I would guess they also need it, but I could not find any actual products. Are there others?

An end to high memory?

khim — Fri, 06 Mar 2020 11:32:04 +0000

Indeed IBM was never concerned about memory limits and haven't planned for that infamous 640KiB barrier to ever exist.

You opponent says IBM didn't have to reserve ⅜ of the (20-bit) address space for the UMA, 256 KiB or even only 128 KiB would have been possible too… but IBM haven't reserved ⅜ of the address space for that! Look on the System Memory Map in the manual. It places "128KB RESERVED GRAPHIC/DISPLAY BUFFER" at 256K position!

XT acknowledged the fact that you may actually add more RAM, but even AT manual says it's not a standard use but option which requires an addon card!

In fact that's how IBM Itself perceived it till PS/2: that's why they were so happy to allow IBM PC DOS 4 to become so much larger than IBM PC DOS 3. The idea was: "hey, 512K is standard now, we could offer option card we talked around long ago and people would get bigger DOS, yet more space for programs, too.

Only by that time “an option” have already become de-facto “standard” and many packages needed 640KiB with DOS 3…

P.S. And after all that IBM decided that it's doesn't matter how much BIOS takes since everyone would soon use OS/2 anyway… and introduced ABIOS… but that's another story…

An end to high memory?

kpfleming — Fri, 06 Mar 2020 11:09:10 +0000

Sorry, that wasn't quite what I meant to say. I follow the stream of kernel updates provided by the Raspberry Pi foundation, so there are frequent kernel updates. If they can deliver an 'update' from a 32-bit kernel to a 64-bit kernel without requiring significant changes in the userspace tools installed on the machine, then users could be upgraded in place.

An end to high memory?

geert — Fri, 06 Mar 2020 11:04:19 +0000

So they keep on using their current kernel, which includes highmem support.

An end to high memory?

kpfleming — Fri, 06 Mar 2020 10:59:07 +0000

It can be done today, if you're willing to give up support for some of the onboard peripherals. Even when there is full hardware support, it's likely that most users won't convert their existing systems.

An end to high memory?

geert — Fri, 06 Mar 2020 09:39:02 +0000

But one day the Raspberry Pi 4B will run a 64-bit kernel, solving the problem, right?

An end to high memory?

kpfleming — Thu, 05 Mar 2020 18:52:30 +0000

Pretty much every Raspberry Pi 4B sold with 2GB or 4GB of RAM is in this situation, because the default (and still the only well-supported) kernel is a 32-bit kernel. There are large numbers of these, and they are being sold and installed today, so can be expected to be in use for quite some time.

An end to high memory?

neilbrown — Mon, 02 Mar 2020 19:31:51 +0000

32bit MIPS needs high memory just to make use of 512MB of RAM. The first 448MB are directly mapped, then there is IO space, then there is the rest of RAM as high-mem.

Maybe this "interesting" address space layout could be managed differently to avoid the dependency on highmem - I don't know.

[    0.000000] MIPS: machine is GB-PC1
[    0.000000] Determined physical RAM map:
[    0.000000]  memory: 1c000000 @ 00000000 (usable)
[    0.000000]  memory: 04000000 @ 20000000 (usable)

...
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x000000001fffffff]
[    0.000000]   HighMem  [mem 0x0000000020000000-0x0000000023ffffff]
...

An end to high memory?

dave4444 — Mon, 02 Mar 2020 01:05:32 +0000

Having worked on many 32bit CPUs with large memory, this is somewhat nistalgic, but it's time is probabaly near the end. And yes, there were many 32bit ARM CPUs with large amounts of memory (such as some NPUs!). Intel also had some SKUs with 32bit cores long after consumer and server chips supported 64bit, notibly some SoCs and Atom CPUs. Deprecating is probably due (especially for some archs).

Lets also not forget about highmem's ugly step child, the bounce buffer for IO devices that DMA to/from high memory. Oh the headaches from that.

An end to high memory?

ldearquer — Sun, 01 Mar 2020 12:32:47 +0000

I'd be sorry to see highmem going, now that I *finally* got a decent understanding of what it is and where it came from :) Thanks for a great article!

An end to high memory?

willy — Sat, 29 Feb 2020 19:29:02 +0000

Ah, I see the ambiguity in what flussence said now.

Yes, you're correct, only 896MB of physical RAM is usable unless you enable at least HIGHMEM4G.

An end to high memory?

nivedita76 — Sat, 29 Feb 2020 17:56:45 +0000

Userspace gets 3Gb address space by default whether HIGHMEM is enabled or not, but I thought you can only access 896Mb of physical memory if NOHIGHMEM, regardless of how much you have installed -- HIGHMEM4G is what will let you access the rest of RAM, no?

An end to high memory?

willy — Sat, 29 Feb 2020 12:25:55 +0000

With HIGHMEM entirely disabled, on x86, userspace gets 3GB of address space, the kernel gets 896MB of physical memory, and the remainder is used for PCI BARs, vmalloc space and random other gunk.

Updated list of machines with 4GB or more

arnd — Sat, 29 Feb 2020 10:48:04 +0000

> How can be posible that a compiler produces different results depending on the kernel running the compiler (instead of depending on the kernel targeted by the compiler)?

This is about building entire packages, not just a single source file, so a lot of differences between native 32-bit kernels and compat mode can be relevant, though it would usually be a bug in a package if they are:

* The contents of /proc/cpuinfo, uname or the cpuid registers are different, which can confuse configure scripts in all kinds of ways, such as thinking they are cross-compiling when they should be building natively, or being unable to parse the incompatible /proc/cpuinfo format. Simply changing these to look like an ARMv7 CPU would be bad for applications that have a legitimate interest in finding out the actual CPU type on a 64-bit kernel.

* For doing local builds, upstream maintainers may default to using -march=native, but when building a distro package you normally want to target the minimum supported CPU instead. This bug can also hit when building an ARMv6 package on an ARMv7VE host just as it can when building an i586 package on Skylake x86-64 host.

* The compat mode in the kernel is a pretty good approximation of the native interfaces, but there are always a few differences -- usually those are bugs in compat handling, but there is also sysvipc being broken on native sparc32 kernels or such as arm32 kernels traditionally not having NUMA syscalls, both of which worked fine on 64-bit compat mode.

* On 64-bit kernels, you usually have 4GB of virtual address space for user space, while native kernels have only 3GB or less. This is great for most applications, but it can cause problems in a runtime environment that treats any high pointer value as a special cookie.

Some of these can be worked around using the 'personality' interfaces, others should probably be configurable the same way, and some should just be considered bugs that need to be fixed in upstream user space packages. I personally think the build systems should use 64-bit kernels with 32-bit user space and fix the resulting bugs, but it's not my decision.

> How crosscompilers even work then?

You can probably cross-compile a majority of all packages in a distro these days, but there are enough exceptions that Debian or OBS decide to only build packages natively.

Updated list of machines with 4GB or more

rossburton — Sat, 29 Feb 2020 10:12:47 +0000

I’m guessing the point was to do native and not cross compilation.

An end to high memory?

mwsealey — Sat, 29 Feb 2020 02:57:03 +0000

I didn’t consider the security of mapping the entire physical address space into kernel virtual space but that’s another good reason not to do it.

“Low” or pinned memory for kernel allocations needs to be there, I don’t deny that. However as a way of providing watermarks for particular kinds of allocations is it *really* “faster” to not have to map it?

For 64-bit kernels not having the linear map of an arbitrary amount of physical memory and just having those watermarks will still divide throwaway pages from fixed ones. For 32-bit kernels they hit HIGHMEM pressure so quickly it doesn’t makes sense to me “reserve” the memory used for non-HIGHMEM regions.

In any case by removing the linear map no physical address is mapped >=twice to virtual addresses without there being an extremely specific purpose for it. Userspace pages will never “turn up” in the middle of kernel space, or alias with other memory types.

For Arm the linear map is just trouble - I spend a good deal of time running training courses on Arm architecture and it is always disappointing to have to point out that a particular architectural feature like fine grained memory type specification is just a pain in the backside in Linux.

If you have a need to mark something as only outer cacheable or not-outer-shareable (and your HW supports that) the possibility that it also exists in the linear map as inner cacheable and outer shareable too just defeats any efficiencies you could gain from it. It serves to promote all cache activity to “L1 vs. DRAM” and all DVM transactions towards all other devices with nothing lighter in between. The x86-ism of poor TLB functionality and limited memory types and coherency traffic spamming leaks to other architectures like a plague.

Getting rid of HIGHMEM and making everything a linear mapped address range just codifies that x86-ism for 64-bit, while crippling 32-bit. I can’t say I like either idea. I’d rather see HIGHMEM live on and we reduce our requirement (and subsequently watermarks) for NORMAL. Spectre/Meltdown and the rise of virtualization have helped now x86 cores are gaining more efficient TLB flush operations, working ASIDs et al.

I reckon it’s worth investigating, for all we know it could surprisingly end up more performant and with less side effects and give Linux a few extra features on top of a more fine-grained memory management subsystem. I’m failing to find significant parts of mm that are actually hardcore enough to rely on the linear mapping and don’t have a “slow path” through using high memory (what requires it will allocate it, and then it’s doing the job it is meant to do), and from what I’ve seen the machinations around contiguous/large pages and other zone allocators like CMA already implement all you’d need to efficiently divorce the kernel from having to keep the linear map around.

Updated list of machines with 4GB or more

clopez — Sat, 29 Feb 2020 01:23:41 +0000

> Calxeda Midway servers used in build farms for native 32-bit builds to avoid differences in compilation results between running on 32-bit and 64-bit kernels, usually with 16GB of RAM.

How can be posible that a compiler produces different results depending on the kernel running the compiler (instead of depending on the kernel targeted by the compiler)?
How crosscompilers even work then?

Updated list of machines with 4GB or more

arnd — Fri, 28 Feb 2020 16:05:40 +0000

I talked to a number of people about this at EmbeddedWorld and elswhere to figure out who is affected. There were many more examples of 2 GB systems with 32-bit ARM processors, which in theory could use a modified VMSPLIT_2G_OPT mapping. Systems with more than that, in no particular order, include:

- Calxeda Midway servers used in build farms for native 32-bit builds to avoid differences in compilation results between running on 32-bit and 64-bit kernels, usually with 16GB of RAM.
- Most TI keystone-2 systems (sometimes up to 8GB): https://lwn.net/ml/linux-kernel/7c4c1459-60d5-24c8-6eb9-d...
- Baikal T1 (MIPS, not ARM) with up to 8GB: https://www.t-platforms.ru/production/products-on-baikal/
- Dragonbox Pyra game consoles with TI OMAP5: https://pyra-handheld.com/boards/pages/pyratech/
- Novena Laptop: https://www.crowdsupply.com/sutajio-kosagi/novena
- SolidRun CuBox Pro i4x4 (early models only) https://www.solid-run.com/solidrun-introduces-4gb-mini-co...
- Tegra K1 and Rockchips RK3288 based Chromebooks: https://www.chromium.org/chromium-os/developer-informatio...
- Very rare industrial NXP i.MX6 and Renesas RZ/G1 systems (most board manufacturers I talked to said they never sold 4GB options, or never even offered them because of 8Gbit DDR3 availability and cost reasons)

An end to high memory?

cpitrat — Fri, 28 Feb 2020 15:26:37 +0000

https://en.wikipedia.org/wiki/Time_flies_like_an_arrow;_f...

An end to high memory?

Deleted user 129183 — Fri, 28 Feb 2020 15:19:26 +0000

> Fruit flies like a banana.

I’m pretty sure that round fruit fly like an apple, not like a banana. In fact, are there any other fruit shaped like a banana besides bananas? Because flight of bananas may be actually _sui generis_.

An end to high memory?

arnd — Fri, 28 Feb 2020 14:56:36 +0000

I looked this up a few days ago, and the original patch set was from 2003 and only posted for review [https://lore.kernel.org/lkml/Pine.LNX.4.44.0307082332450....] briefly then. It notably made it into the RHEL4 distro, but never into mainline kernels. Back then the main downside was that x86 CPUs of the time lacked a address space identifiers, so changing between user space and kernel always required flushing all TLBs, which has an enormous performance impact for system calls and page faults.

However, it does seem possible to use a 4G vmsplit on 32-bit ARM (probably also MIPS or others) ASIDs to avoid the TLB flush and keep the performance impact way down. On ARM with LPAE, we could e.g. use the split page tables to have the top 256MB (or another power-of-two size) reserved for vmalloc pages, modules, MMIO and the kernel image, while the lower 3.75GB are mapped to either user space or a large linear map. There is some extra overhead for copy_to_user() etc, but also extra isolation between user and kernel addresses that may provide security benefits.

It's also interesting that the original VMPLIT_4G_4G patches were not even intended to replace highmem, but to allow configurations with 32GB or 64GB of physical RAM that would otherwise fill a lot of the 896MB lowmem area with 'struct page' structures even before you start allocating inodes, dentries or page tables.

An end to high memory?

farnz — Fri, 28 Feb 2020 14:18:51 +0000

In the original PC, they didn't set a 640 KiB limit (that comes in with the EGA card. Original IBM PCs with MDA displays have a 704 KiB limit, CGA raises that to 736 KiB, and a theoretical video adapter using I/O ports instead of a memory mapped buffer could raise it to 768 KiB.

Honestly, it looks like IBM never really thought about the real mode address limits; EGA lowered it to 640 KiB, but that comes in with the 80286 in the PC/AT, which in theory could be run in protected mode and thus not have issues around the 1 MiB limit. OS/2 1.x could thus have ensured we never needed to know about HMA, UMA, XMS etc, had IBM's vision been successful, and hence the "640 KiB" limit of their 8088 products would never have mattered. It's just that IBM failed in delivering its vision, and thus we kept treating x86 systems as ways to run DOS for far longer than intended.

An end to high memory?

leromarinvit — Fri, 28 Feb 2020 13:12:46 +0000

That wouldn't really be a "split" though - it would mean that all mappings have to be changed when entering and exiting the kernel, exactly what the split tried to prevent.

I wonder what the performance impact of just getting rid of the split in this manner would be. Isn't the performance advantage already rendered moot by KPTI?

An end to high memory?

ecm — Fri, 28 Feb 2020 11:48:03 +0000

That is all true. However, IBM didn't have to reserve 3/8 of the (20-bit) address space for the UMA, 256 KiB or even only 128 KiB would have been possible too. So the specific 640 KiB limit wasn't really due to the CPU (nor the OS).

An end to high memory?

eru — Fri, 28 Feb 2020 10:04:43 +0000

What about a 4g/4g split? I seem to recall some distributions had this kind of feature to allow more memory for userspace.

An end to high memory?

cpitrat — Fri, 28 Feb 2020 07:50:42 +0000

Welcome in the group of "the oldest". Time flies like an arrow ...

... Fruit flies like a banana.

An end to high memory?

farnz — Thu, 27 Feb 2020 21:20:16 +0000

Note, though, that the IBM PC has that memory layout because the BIOS ROM has to be mapped at 0xFFFF0 on the 8086/8088 - CS:IP of 0xFFFF:0x0000 - because that's where the reset vector is, and you also want 0x00000 to 0x00400 to be in RAM because that's where the 8086/88 put their Interrupt Vector Table.

The Model 5150 was being built to a price point, and was in any case limited to 256 KiB RAM, so it wasn't worth the extra ICs needed to allow you to remap all but the first 1 KiB RAM and the BIOS ROM after reset, when you could leave them in place.

Arguably, things would have been different if IBM had used the Motorola 68k instead of the 8088; the 68k has its reset vector at address 0, and thus you'd naturally put your ROM at the beginning of address space.

An end to high memory?

flussence — Thu, 27 Feb 2020 20:52:15 +0000

On x86 machines with highmem disabled it's even worse: the default for a long time has been to give userspace access to only the first 896MB(!) out of 4GB.

There are hidden kconfig options to make that 2 or 3GB, and there are third-party patches to make the default 1GB. It's probably for compatibility with some ancient blob, because I can't imagine that being a good default to keep otherwise; there are quite a few 32-bit systems out there sitting between ⅞ and 4GB of RAM installed.

An end to high memory?

willy — Thu, 27 Feb 2020 20:41:47 +0000

The problem is that everything you allocate inside the kernel has to either be in ZONE_NORMAL or be in ZONE_HIGHMEM and be explicitly mapped. So, eg, every time you call kmalloc(), that page has to be mapped in the 1GB of kernel space _somehow_. If we got rid of ZONE_NORMAL and made everything part of HIGHMEM and kmalloc mapped the page for you, it'd still have just as tight a fit. And worse performance due to every mapping being dynamic.

As for undergraduates being able to figure out how the Linux MM works ... I don't think HIGHMEM is the problem here.

We may end up with an option to map everything. Some of the security people are this as a solution to the various cache information leakage problems like meltdown and spectre. But it's likely to have a negative performance impact.

An end to high memory?

luto — Thu, 27 Feb 2020 20:37:12 +0000

> at the cost of a tiny bit of extra cache and TLB maintenance in certain cases

On an architecture with lousy TLB maintenance facilities like x86, the expense would be *huge*.

An end to high memory?

Deleted user 129183 — Thu, 27 Feb 2020 20:35:44 +0000

> We'll start by noting, for the oldest among our readers, that it has nothing to do with the "high memory" concept found on early personal computers.

Oldest? I was born in 1991 and still remember it. In fact I was really confused when I have read the title of this article, thinking that it refers to the HMA.

An end to high memory?

mwsealey — Thu, 27 Feb 2020 19:28:18 +0000

I dare say another valid concern here is that the linear map consumes precious virtual address space that would otherwise be able to be used by active highmem pages. The linear map itself will exacerbate any use of memory outside the ~512MB of the low end of the physical address map on 32-bit, since it sacrifices about 60% of it to eke out performance for using that low region. Getting rid of the linear map and making EVERYTHING highmem might even out performance, as long as you could ensure that you could always pin those important pages into the virtual address space. There's no functional difference here in running out of linear mapped low mem space and running out of actual virtual address space. And there should be a way to prevent allocated regions of memory from being migrated 'out' of the virtual address space to fault that doesn't rely on the linear mapping already, making them stick around in 'low memory' by default.

Just get rid of the linear map, and at the cost of a tiny bit of extra cache and TLB maintenance in certain cases, and not being able to cheap out on pointer math (not that this isn't being tracked many times over anyway) I think everyone's life gets easier and more computer science students would understand how the hell the kernel mm works! Also, no address aliasing..

An end to high memory?

ecm — Thu, 27 Feb 2020 18:21:09 +0000

> We'll start by noting, for the oldest among our readers, that it has nothing to do with the "high memory" concept found on early personal computers. That, of course, was memory above the hardware-implemented hole at 640KB — memory that was, according to a famous quote often attributed to Bill Gates, surplus to the requirements of any reasonable user.

This does not seem entirely clear.

Actually, there is an area on the IBM PC and 86-DOS platform that is called the Upper Memory Area (UMA), which can hold things such as ROMs, video memory, EMS frames, and Upper Memory Blocks (UMBs). It usually lies between 640 KiB (linear A_0000h) up to 1024 KiB (10_0000h). It wasn't due to the CPU however, but due to the IBM PC's memory layout.

The High Memory Area starts at 1024 KiB and reaches up to 1088 KiB minus 16 Bytes. This was introduced with the 286 when the physical memory was expanded to 24 address lines. This meant that even in Real 86 Mode (and on the 386 likewise in Virtual 86 Mode), the segmented addresses between 0FFFFh:0010h and 0FFFFh:0FFFFh would no longer wrap around to the linear addresses within the first 64 KiB minus 16 Bytes, and would instead form a 21-bit linear address pointing into the HMA. (This is also what A20 was about.)

I call the memory area from zero up to below the UMA the Lower Memory Area, in symmetry with the UMA and HMA. It is also called base memory or conventional memory.

To add to the confusion, in German MS-DOS localisations, the UMA was called hoher Speicherbereich ("high Memory-area") and then the HMA was called oberer Speicherbereich ("upper Memory-area"). That is, the terms were swapped as compared to the English terms. This I believe was fixed starting in MS-DOS 7.00 (bundled with MS Windows 95).

An end to high memory?

willy — Thu, 27 Feb 2020 18:11:53 +0000

> the kernel cannot access it without creating a temporary, single-page mapping, which is expensive

As always with caches, it's not creating the mapping that's expensive. It's removing it afterwards that's the expensive part!