| From: |
| Ankur Arora <ankur.a.arora-AT-oracle.com> |
| To: |
| linux-kernel-AT-vger.kernel.org, linux-mm-AT-kvack.org, x86-AT-kernel.org |
| Subject: |
| [PATCH v7 00/16] mm: folio_zero_user: clear contiguous pages |
| Date: |
| Wed, 17 Sep 2025 08:24:02 -0700 |
| Message-ID: |
| <20250917152418.4077386-1-ankur.a.arora@oracle.com> |
| Cc: |
| akpm-AT-linux-foundation.org, david-AT-redhat.com, bp-AT-alien8.de, dave.hansen-AT-linux.intel.com, hpa-AT-zytor.com, mingo-AT-redhat.com, mjguzik-AT-gmail.com, luto-AT-kernel.org, peterz-AT-infradead.org, acme-AT-kernel.org, namhyung-AT-kernel.org, tglx-AT-linutronix.de, willy-AT-infradead.org, raghavendra.kt-AT-amd.com, boris.ostrovsky-AT-oracle.com, konrad.wilk-AT-oracle.com, ankur.a.arora-AT-oracle.com |
| Archive-link: |
| Article |
This series adds clearing of contiguous page ranges for hugepages,
improving on the current page-at-a-time approach in two ways:
- amortizes the per-page setup cost over a larger extent
- when using string instructions, exposes the real region size
to the processor.
A processor could use a knowledge of the extent to optimize the
clearing. AMD Zen uarchs, as an example, elide allocation of
cachelines for regions larger than L3-size.
Demand faulting a 64GB region shows performance improvements:
$ perf bench mem map -p $pg-sz -f demand -s 64GB -l 5
mm/folio_zero_user x86/folio_zero_user change
(GB/s +- %stdev) (GB/s +- %stdev)
pg-sz=2MB 11.82 +- 0.67% 16.48 +- 0.30% + 39.4% preempt=*
pg-sz=1GB 17.14 +- 1.39% 17.42 +- 0.98% [#] + 1.6% preempt=none|voluntary
pg-sz=1GB 17.51 +- 1.19% 43.23 +- 5.22% +146.8% preempt=full|lazy
[#] Milan uses a threshold of LLC-size (~32MB) for eliding cacheline
allocation, which is higher than the maximum extent used on x86
(ARCH_CONTIG_PAGE_NR=8MB), so preempt=none|voluntary sees no improvement
with pg-sz=1GB.
Raghavendra also tested v3/v4 on AMD Genoa and sees similar improvements [1].
Changelog:
v7:
- interface cleanups, comments for clear_user_highpages(), clear_user_pages(),
clear_pages().
- fixed build errors flagged by kernel test robot
- move all x86 patches to the tail end
v6:
- perf bench mem: update man pages and other cleanups (Namhyung Kim)
- unify folio_zero_user() for HIGHMEM, !HIGHMEM options instead of
working through a new config option (David Hildenbrand).
- cleanups and simlification around that.
(https://lore.kernel.org/lkml/20250902080816.3715913-1-ank...)
v5:
- move the non HIGHMEM implementation of folio_zero_user() from x86
to common code (Dave Hansen)
- Minor naming cleanups, commit messages etc
(https://lore.kernel.org/lkml/20250710005926.1159009-1-ank...)
v4:
- adds perf bench workloads to exercise mmap() populate/demand-fault (Ingo)
- inline stosb etc (PeterZ)
- handle cooperative preemption models (Ingo)
- interface and other cleanups all over (Ingo)
(https://lore.kernel.org/lkml/20250616052223.723982-1-anku...)
v3:
- get rid of preemption dependency (TIF_ALLOW_RESCHED); this version
was limited to preempt=full|lazy.
- override folio_zero_user() (Linus)
(https://lore.kernel.org/lkml/20250414034607.762653-1-anku...)
v2:
- addressed review comments from peterz, tglx.
- Removed clear_user_pages(), and CONFIG_X86_32:clear_pages()
- General code cleanup
(https://lore.kernel.org/lkml/20230830184958.2333078-1-ank...)
Comments appreciated!
Also at:
github.com/terminus/linux clear-pages.v7
[1] https://lore.kernel.org/lkml/fffd4dad-2cb9-4bc9-8a80-a70b...
Ankur Arora (16):
perf bench mem: Remove repetition around time measurement
perf bench mem: Defer type munging of size to float
perf bench mem: Move mem op parameters into a structure
perf bench mem: Pull out init/fini logic
perf bench mem: Switch from zalloc() to mmap()
perf bench mem: Allow mapping of hugepages
perf bench mem: Allow chunking on a memory region
perf bench mem: Refactor mem_options
perf bench mem: Add mmap() workloads
mm: define clear_pages(), clear_user_pages()
mm/highmem: introduce clear_user_highpages()
arm: mm: define clear_user_highpages()
mm: memory: support clearing page ranges
x86/mm: Simplify clear_page_*
x86/clear_page: Introduce clear_pages()
x86/clear_pages: Support clearing of page-extents
arch/arm/include/asm/page.h | 7 +
arch/x86/include/asm/page_32.h | 6 +
arch/x86/include/asm/page_64.h | 72 +++-
arch/x86/lib/clear_page_64.S | 39 +-
include/linux/highmem.h | 18 +
include/linux/mm.h | 44 +++
mm/memory.c | 82 +++-
tools/perf/Documentation/perf-bench.txt | 58 ++-
tools/perf/bench/bench.h | 1 +
tools/perf/bench/mem-functions.c | 390 ++++++++++++++-----
tools/perf/bench/mem-memcpy-arch.h | 2 +-
tools/perf/bench/mem-memcpy-x86-64-asm-def.h | 4 +
tools/perf/bench/mem-memset-arch.h | 2 +-
tools/perf/bench/mem-memset-x86-64-asm-def.h | 4 +
tools/perf/builtin-bench.c | 1 +
15 files changed, 560 insertions(+), 170 deletions(-)
--
2.43.5