LWN: Comments on "Sharing page tables with msharefs"

Sharing page tables with msharefs

mpr22 — Wed, 22 Feb 2023 23:46:15 +0000

When someone who cares does the work.

Sharing page tables with msharefs

smooth1x — Wed, 22 Feb 2023 22:35:17 +0000

"those who do not understand unix are condemned to reinvent it poorly"

Solaris already had this since the 1990s, Intimate Shared Memory (ISM) - shmat with the SHM_SHARE_MMU flag.

Solaris even now has Dynamic Intimate Shared Memory - https://docs.oracle.com/cd/E19683-01/817-3801/whatsnew-s9...

When will Linux get with the program and get this added, only 30 years later?!!

Sharing page tables with msharefs

flussence — Fri, 22 Jul 2022 10:13:00 +0000

It sounds like there's a lot of overlap here with Transparent Hugepages, which had to deal with similar merging and splitting.

Sharing page tables with msharefs

rwmj — Thu, 21 Jul 2022 14:45:54 +0000

> Automatic sharing would need to have different semantics, though; otherwise applications will be surprised when an mprotect() or mremap() call in another process changes their mappings.

Wouldn't any automatic sharing solution have to spot this case and "unshare" the PTEs?

Sharing page tables with msharefs

corbet — Mon, 18 Jul 2022 13:52:55 +0000

Pathological cases can always be constructed. But if you're trying to map GB of memory scattered through memory as individual pages, you're going to have other problems as well, and PTE overhead is still unlikely to be the constraining factor.

Sharing page tables with msharefs

ddevault — Mon, 18 Jul 2022 11:54:44 +0000

>In the pathological case, the addresses are distributed throughout memory, in which case you might need 512*3 = 1536 page tables = 6 MiB to map all of them (*3 because you only ever need one PML4 per address space).

One other note: this situation calls for 6 MiB of page tables to map 2 MiB worth of actual pages.

Sharing page tables with msharefs

ddevault — Mon, 18 Jul 2022 11:53:28 +0000

>A page-table entry (PTE) is relatively small, requiring just eight bytes to refer to a 4096-byte page on most systems.

This is a little bit misleading. On x86_64, there are four levels of page tables: PML4, PDPT, PD, and PT; the latter contains PTEs (you also have PDE, PDPTE, and PML4E's going up the chain). A virtual address essentially contains four indexes into these tables, and the CPU follows the tables to resolve the physical address for a given virtual address, such that address 0x111222333444 will look up index 0x111 from the PML4 and so on (simplified representation) to find the appropriate physical memory address. Each page table is 4 KiB each and holds 512 of these 8-byte entries, be it PDEs or otherwise. If you map fewer than 512 pages, you still need to use at least 4K*4 = 16K bytes for the four page tables, each of which takes up a single page.

In the simple case, you can map up to 512 pages with just the four page tables -- assuming the addresses are contiguous. In the pathological case, the addresses are distributed throughout memory, in which case you might need 512*3 = 1536 page tables = 6 MiB to map all of them (*3 because you only ever need one PML4 per address space).

Sharing page tables with msharefs

butlerm — Sat, 16 Jul 2022 18:32:26 +0000

If your database server can handle the load in terms of number of connections, and your application can afford to be tightly coupled to the database design, and the connection latency is typical of a local area network, it is probably almost always faster to connect directly rather than running through extra tiers that mostly shuffle things around en route. In so many cases that is not possible any more though, even old school web applications multiplex database connections across concurrent users (and often do a outstanding job of it, but that is another story).

Sharing page tables with msharefs

flussence — Sat, 16 Jul 2022 09:58:03 +0000

I've seen articles about database designs that, instead of routing things through a frontend web API or whatever, expose Postgres users and use its security rules model directly. I imagine that'd cause some scaling headaches if done to a medium-large website...

Sharing page tables with msharefs

edeloget — Sat, 16 Jul 2022 09:44:04 +0000

> Doesn't PostgreSQL also use fork per client model?

Yes, but then, correct system architecture tells you to limit the number of clients to something that the machine can handle :)

If you cannot have 1500 clients on a signe machine, then maybe you shouldn't have 1500 clients on the same machine :)

Sharing page tables with msharefs

mokki — Sat, 16 Jul 2022 06:20:32 +0000

Doesn't PostgreSQL also use fork per client model?

I was under the impression that thread per client model is faster, but less safe. In process per client model a bug in one client cannot corrupt memory in the process, outside the explicitly shared memory area.

Sharing page tables with msharefs

nickodell — Fri, 15 Jul 2022 22:59:16 +0000

Yes, exactly. From the May 17 article about this patchset:

> That raises an obvious question, he said: why not just use threads? The answer was that "mmap_lock sucks". It is also not possible to change the existing behavior of MAP_SHARED, since that would break programs, so there would need to be, at a minimum, a new mmap() flag if not a new system call. Aziz said that the separate system call makes the page-table sharing explicit rather than it just being a side effect. That makes the decision to opt into this behavior explicit as well.

https://lwn.net/Articles/895217/

Sharing page tables with msharefs

bartoc — Fri, 15 Jul 2022 20:52:07 +0000

Perhaps they want to COW some regions of memory while sharing others. This is quite difficult to do without fork/clone because as far as I can tell there's no other good way to create a COW mapping of another COW mapping. (On windows this is completely impossible, which is why you can't easily do things like forking game saves without implementing COW as part of your application's data structures). I'm not 100% sure how to do this on linux (without using fork/clone), maybe you can reflink tempfs files together or something.

Sharing page tables with msharefs

malmedal — Fri, 15 Jul 2022 20:20:08 +0000

I'm totally onboard with the no-one should be running Oracle sentiment :)

Several years out of the loop now, but I remember that Oracle, at least used to be, up to twice as fast as
PostgreSQL and MySQL on the same hardware and also the Magic Money Tree would provide far more money for
Oracle hardware since the license was so expensive :(

Sharing page tables with msharefs

josh — Fri, 15 Jul 2022 19:55:00 +0000

What makes it lower performance when using threads? Lock contention on the process mm? Something else?

Sharing page tables with msharefs

k8to — Fri, 15 Jul 2022 18:11:00 +0000

But also, if you do care about performance you really shouldn't be running Oracle.

(Sort of a joke, because no one should be. But sort of true because they tend only win in the case of very fragile tuning that cannot be done in a timely way. And because of course if you spent that same money on hardware, you'd usually be better off with postgresql.)

Sharing page tables with msharefs

malmedal — Fri, 15 Jul 2022 17:11:24 +0000

If you don't care about performance you really shouldn't be running Oracle.

Sharing page tables with msharefs

clugstj — Fri, 15 Jul 2022 15:28:05 +0000

Well, when given the choice between a performance hit and it crashing, I'd say you should pick performance hit.

Sharing page tables with msharefs

willy — Fri, 15 Jul 2022 14:38:15 +0000

The database can work in a threaded mode, but performance on Linux is worse than the fork model.

Sharing page tables with msharefs

clugstj — Fri, 15 Jul 2022 14:32:28 +0000

Maybe Oracle should not spawn a process for each client.