Weekly Edition
Return to the Kernel page| Weekly edition | Kernel | Security | Distributions | Search |
| Archives | Calendar | Subscribe | Write for LWN | LWN.net FAQ |
[RFC]Super Page for Alpha,Sparc64,i386
| From: | Naohiko Shimizu <pshimizu@fa2.so-net.ne.jp> | |
| To: | linux-kernel@vger.kernel.org | |
| Subject: | [PATCH] [RFC]Super Page for Alpha,Sparc64,i386 | |
| Date: | Mon, 12 Aug 2002 17:45:22 +0900 |
This mail was bounced, and I resent it from my home. The original mail attached both 2.4.18 and 2.4.19 patch but it was too long for the list, and I deleted 2.4.18 patch. ------------------------------------------------------------------ Before shutting down my server, I decide to post the patch itself to this list. Our university will stop the electoricity, and my server will get back in 8/14. The url of the original patch is: http://shimizu-lab.dt.u-tokai.ac.jp/lsp.html Any comment will be welcome, but I cannot answer before 8/14. I attached 2.4.19 version. For i386, we sould turn on vm_align feature to boost the performance: sysctl -w super_page.vm_align=1 This feature makes the anonymous mmap requests align to the super page boundary if the requested length is larger than a super page size. The i386 port is efficient only with the pentium4 class machine with a program which uses a lot of memory(such as 64MB or more). The Alpha and Sparc64 port is efficient with much smaller size of the programs. Because they support not only 4MB but also 64KB and 512KB. The attached benchmark program (transm.c) will test the memory throughput performance with large strides with matrices transpose. It outputs the dimension, store stride performance (MB/S), load stride performance(MB/S) respectively. The Linux Super Page does not have any special system calls. It automatically turns on the super page feature with little excessive working set. The feature is controlable through sysctl. You can turn off it with sysctl -w super_page.nr=1, and turn on again with sysctl -w super_page.nr=2 (for i386) or sysctl -w super_page.nr=4 (for Alpha and Sparc64). I also provide /proc interface to monitor the reservation, allocation and fail of the allocation for the super pages. Plans for future enhancements: 1. simplify the set_pte macro 2. Support shared memory 3. Support COW 4. IA64 port (two plans 1: use long pte, 2: use region) But I myself is too busy these days for summer/fall course lectures and their preparations. Then do not ask me for the enhancements. -- Naohiko Shimizu Department of Communications Engineering, School of Information Technology and Electronics, Tokai University 1117 Kitakaname Hiratsuka 259-1292 Japan TEL.+81-463-58-1211(ext. 4084) FAX.+81-463-58-8320 diff -urN linux-2.4.19/Documentation/super_page.txt linux/Documentation/super_page.txt --- linux-2.4.19/Documentation/super_page.txt Thu Jan 1 09:00:00 1970 +++ linux/Documentation/super_page.txt Wed Aug 7 10:41:58 2002 @@ -0,0 +1,69 @@ +The super page is a feature which extend the TLB coverage much. +Some modern processors have this feature, and Linux Super Page +will activate this feature and result in getting more performance +for the extensively TLB missing programs. + +The target architecture of this patch is +1) Alpha +2) Sparc64 +3) i386 + +But I don't have any Sparc64 machine then it is not tested by myself. +The i386 port is under the development. The current version can not +activate the super page for i386 but only provide the framework. +I have a working patch for i386 by my student but it is a dirty hack +on my Alpha port and I will not take it. + +The super page patch provides some convenient features. + +/proc/super_page: + +current nr: The maximum super page index. Alpha,Sparc64 support 1 through 4 +and i386 support 1 or 2. +current vm_align: The current super page align flag. 1:0 means that anonymous +mmap request will be aligned to the appropriate super page boundary. +1:1 means that in addition to the start alignment, the length will be aligned +to the super page boundary. +current bitmask: The value of the super page order mask. +order reserve allocate fail: The super page order, reservation count, + allocation count and allocation fail count, respectively. + +sysctl: Provide following variables. + +nr: The maximum super page index. +vm_align: The super page align flag. +tail_align: The super page align flag for the end of allocation. Only valid when the vm_align==1. +bitmask: The super page order mask. The super page index corresponding to +the 0 value of the mask will not be used. It will fall into the atomic page. +logreset: The counter will be reset by reading it. + +The Alpha architecture defines Granularity Hint(GH) bits in the +Page Table Entry(PTE). If these bits are set to non-zero value, +it supply a hint to translation buffer implementations that +a block of pages can be treated as a single larger page. +The sparc64 has almost same feature in the PTE that defines the +page size which is mapped by the PTE. The restriction to use +and the page size variation is also the same as Alpha. + +It means that even if we don't have variable length page mechanism, +we will have the opportunity to reduce translation misses. +For the large working set HPC applications the performance +degradation caused by the translation misses should be avoided. +Then if we can use this feature, many HPC applications will be +appreciated. + +In this release there is a configuration option to support this +feature for Alpha architecture. You can turn on this feature by assert +the CONFIG_SUPER_PAGE variable. With this release, you can set GH bits in +your page table when you call brk system call or anonymous mmap system call +with MAP_PRIVATE flag on. Then if you want to run your program with +enough speed, you should allocate memory with these call (or more +general you can use malloc library call). +The GH bits only set on the heap allocation, after that if any process +changes the table it will drop the bits for safety. You may want to +stop swapping for continuous high performance operation. + + +Naohiko Shimizu<nshimizu@keyaki.cc.u-tokai.ac.jp> +URL: http://shimizu-lab.dt.u-tokai.ac.jp/lsp.html + diff -urN linux-2.4.19/arch/alpha/config.in linux/arch/alpha/config.in --- linux-2.4.19/arch/alpha/config.in Sat Aug 3 09:39:42 2002 +++ linux/arch/alpha/config.in Thu Aug 8 11:11:34 2002 @@ -259,6 +259,11 @@ # LARGE_VMALLOC is racy, if you *really* need it then fix it first define_bool CONFIG_ALPHA_LARGE_VMALLOC n +bool 'SuperPage Support' CONFIG_SUPER_PAGE +if [ "$CONFIG_SUPER_PAGE" = "y" ]; then + define_int CONFIG_FORCE_MAX_ZONEORDER 11 +fi + source drivers/pci/Config.in bool 'Support for hot-pluggable devices' CONFIG_HOTPLUG diff -urN linux-2.4.19/arch/alpha/kernel/osf_sys.c linux/arch/alpha/kernel/osf_sys.c --- linux-2.4.19/arch/alpha/kernel/osf_sys.c Sat Aug 3 09:39:42 2002 +++ linux/arch/alpha/kernel/osf_sys.c Wed Aug 7 20:31:54 2002 @@ -1321,6 +1321,9 @@ return ret; } +#if CONFIG_SUPER_PAGE +extern int super_page_vm_align; +#endif /* Get an address range which is currently unmapped. Similar to the generic version except that we know how to honor ADDR_LIMIT_32BIT. */ @@ -1329,6 +1332,18 @@ unsigned long limit) { struct vm_area_struct *vma = find_vma(current->mm, addr); + unsigned long super_page_mask=0; +#if CONFIG_SUPER_PAGE + if(super_page_vm_align&&addr==PAGE_ALIGN(TASK_UNMAPPED_BASE)) { + int i; + for(i=super_page_nr-1; i>0; i--) { + if(len>(PAGE_SIZE << super_page_order[i])) { + super_page_mask = (PAGE_SIZE << super_page_order[i]) -1; + break; + } + } + } +#endif while (1) { /* At this point: (!vma || addr < vma->vm_end). */ @@ -1338,6 +1353,7 @@ return addr; addr = vma->vm_end; vma = vma->vm_next; + addr = (addr + super_page_mask)&~super_page_mask; } } diff -urN linux-2.4.19/arch/alpha/mm/init.c linux/arch/alpha/mm/init.c --- linux-2.4.19/arch/alpha/mm/init.c Sat Aug 3 09:39:42 2002 +++ linux/arch/alpha/mm/init.c Tue Aug 6 19:52:31 2002 @@ -5,6 +5,7 @@ */ /* 2.3.x zone allocator, 1999 Andrea Arcangeli <andrea@suse.de> */ +/* SUPER_PAGE support, 2000 Naohiko Shimizu <nshimizu@keyaki.cc.u-tokai.ac.jp> */ #include <linux/config.h> #include <linux/signal.h> @@ -46,6 +47,12 @@ struct pgtable_cache_struct quicklists; #endif +#ifdef CONFIG_SUPER_PAGE +int super_page_order[SUPER_PAGE_NR] = {0,3,6,9}; +pgprot_t super_page_prot[SUPER_PAGE_NR] = + {__pgprot(0x0000),__pgprot(0x0020),__pgprot(0x0040),__pgprot(0x0060)}; +#endif + pgd_t * get_pgd_slow(void) { diff -urN linux-2.4.19/arch/i386/config.in linux/arch/i386/config.in --- linux-2.4.19/arch/i386/config.in Sat Aug 3 09:39:42 2002 +++ linux/arch/i386/config.in Sat Aug 10 18:27:54 2002 @@ -185,6 +185,11 @@ define_bool CONFIG_X86_PAE y fi +bool 'Super Page Support' CONFIG_SUPER_PAGE +if [ "$CONFIG_SUPER_PAGE" = "y" ]; then + define_int CONFIG_FORCE_MAX_ZONEORDER 12 +fi + bool 'Math emulation' CONFIG_MATH_EMULATION bool 'MTRR (Memory Type Range Register) support' CONFIG_MTRR bool 'Symmetric multi-processing support' CONFIG_SMP diff -urN linux-2.4.19/arch/i386/kernel/head.S linux/arch/i386/kernel/head.S --- linux-2.4.19/arch/i386/kernel/head.S Sat Aug 3 09:39:42 2002 +++ linux/arch/i386/kernel/head.S Tue Aug 6 19:52:31 2002 @@ -93,7 +93,11 @@ * Enable paging */ 3: +#if CONFIG_SUPER_PAGE + movl $swapper_tlb_dir-__PAGE_OFFSET,%eax +#else movl $swapper_pg_dir-__PAGE_OFFSET,%eax +#endif movl %eax,%cr3 /* set the page table pointer.. */ movl %cr0,%eax orl $0x80000000,%eax @@ -379,6 +383,28 @@ */ .org 0x1000 ENTRY(swapper_pg_dir) +#if CONFIG_SUPER_PAGE + .long 0x00103007 + .long 0x00104007 + .fill BOOT_USER_PGD_PTRS-2,4,0 + /* default: 766 entries */ + .long 0x00103007 + .long 0x00104007 + /* default: 254 entries */ + .fill BOOT_KERNEL_PGD_PTRS-2,4,0 +/* we copy the table for the tlb */ +swapper_tlb_dir: + .long 0x00103007 + .long 0x00104007 + .fill BOOT_USER_PGD_PTRS-2,4,0 + /* default: 766 entries */ + .long 0x00103007 + .long 0x00104007 + /* default: 254 entries */ + .fill BOOT_KERNEL_PGD_PTRS-2,4,0 + +#define SUPER_OFFSET +0x1000 +#else .long 0x00102007 .long 0x00103007 .fill BOOT_USER_PGD_PTRS-2,4,0 @@ -387,15 +413,17 @@ .long 0x00103007 /* default: 254 entries */ .fill BOOT_KERNEL_PGD_PTRS-2,4,0 +#define SUPER_OFFSET +#endif /* * The page tables are initialized to only 8MB here - the final page * tables are set up later depending on memory size. */ -.org 0x2000 +.org 0x2000 SUPER_OFFSET ENTRY(pg0) -.org 0x3000 +.org 0x3000 SUPER_OFFSET ENTRY(pg1) /* @@ -403,10 +431,10 @@ * initialization loop counts until empty_zero_page) */ -.org 0x4000 +.org 0x4000 SUPER_OFFSET ENTRY(empty_zero_page) -.org 0x5000 +.org 0x5000 SUPER_OFFSET /* * Real beginning of normal "text" segment diff -urN linux-2.4.19/arch/i386/mm/fault.c linux/arch/i386/mm/fault.c --- linux-2.4.19/arch/i386/mm/fault.c Sat Aug 3 09:39:42 2002 +++ linux/arch/i386/mm/fault.c Tue Aug 6 19:52:31 2002 @@ -383,6 +383,9 @@ asm("movl %%cr3,%0":"=r" (pgd)); pgd = offset + (pgd_t *)__va(pgd); +#if CONFIG_SUPER_PAGE + pgd -= PTRS_PER_PGD; +#endif pgd_k = init_mm.pgd + offset; if (!pgd_present(*pgd_k)) diff -urN linux-2.4.19/arch/i386/mm/init.c linux/arch/i386/mm/init.c --- linux-2.4.19/arch/i386/mm/init.c Sat Aug 3 09:39:42 2002 +++ linux/arch/i386/mm/init.c Tue Aug 6 19:57:44 2002 @@ -65,6 +65,17 @@ return freed; } +#if CONFIG_SUPER_PAGE +#if CONFIG_X86_PAE +int super_page_order[SUPER_PAGE_NR] = {0,9}; +#else +int super_page_order[SUPER_PAGE_NR] = {0,10}; +#endif +/* We use the unused bit of the pte. N. Shimizu */ +pgprot_t super_page_prot[SUPER_PAGE_NR] = {__pgprot(0),__pgprot(_PAGE_SUPER)}; +#endif + + /* * NOTE: pagetable_init alloc all the fixmap pagetables contiguous on the * physical space so we can cache the place of the first one and move @@ -331,7 +342,11 @@ { pagetable_init(); +#if CONFIG_SUPER_PAGE + load_cr3(swapper_pg_dir+PTRS_PER_PGD); +#else load_cr3(swapper_pg_dir); +#endif #if CONFIG_X86_PAE /* @@ -603,8 +618,13 @@ /* * PAE pgds must be 16-byte aligned: */ +#if CONFIG_SUPER_PAGE + pae_pgd_cachep = kmem_cache_create("pae_pgd", PTR_PER_PGD*sizeof(pgd_t)*2, 0, + SLAB_HWCACHE_ALIGN | SLAB_MUST_HWCACHE_ALIGN, NULL, NULL); +#else pae_pgd_cachep = kmem_cache_create("pae_pgd", 32, 0, SLAB_HWCACHE_ALIGN | SLAB_MUST_HWCACHE_ALIGN, NULL, NULL); +#endif if (!pae_pgd_cachep) panic("init_pae(): Cannot alloc pae_pgd SLAB cache"); } diff -urN linux-2.4.19/arch/sparc64/config.in linux/arch/sparc64/config.in --- linux-2.4.19/arch/sparc64/config.in Sat Aug 3 09:39:43 2002 +++ linux/arch/sparc64/config.in Thu Aug 8 11:11:56 2002 @@ -31,6 +31,11 @@ # Identify this as a Sparc64 build define_bool CONFIG_SPARC64 y +bool 'SuperPage Support' CONFIG_SUPER_PAGE +if [ "$CONFIG_SUPER_PAGE" = "y" ]; then + define_int CONFIG_FORCE_MAX_ZONEORDER 11 +fi + bool 'Support for hot-pluggable devices' CONFIG_HOTPLUG # Global things across all Sun machines. diff -urN linux-2.4.19/arch/sparc64/kernel/sys_sparc.c linux/arch/sparc64/kernel/sys_sparc.c --- linux-2.4.19/arch/sparc64/kernel/sys_sparc.c Sat Aug 3 09:39:43 2002 +++ linux/arch/sparc64/kernel/sys_sparc.c Wed Aug 7 20:31:30 2002 @@ -44,11 +44,15 @@ ((((addr)+SHMLBA-1)&~(SHMLBA-1)) + \ (((pgoff)<<PAGE_SHIFT) & (SHMLBA-1))) +#if CONFIG_SUPER_PAGE +extern int super_page_vm_align; +#endif unsigned long arch_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) { struct vm_area_struct * vmm; unsigned long task_size = TASK_SIZE; int do_color_align; + unsigned long super_page_mask=0; if (flags & MAP_FIXED) { /* We do not accept a shared mapping if it would violate @@ -63,8 +67,20 @@ task_size = 0xf0000000UL; if (len > task_size || len > -PAGE_OFFSET) return -ENOMEM; - if (!addr) + if (!addr) { addr = TASK_UNMAPPED_BASE; +#if CONFIG_SUPER_PAGE + if(super_page_vm_align) { + int i; + for(i=super_page_nr-1; i>0; i--) { + if(len>(PAGE_SIZE << super_page_order[i])) { + super_page_mask = (PAGE_SIZE << super_page_order[i])-1; + break; + } + } + } +#endif + } do_color_align = 0; if (filp || (flags & MAP_SHARED)) @@ -74,6 +90,7 @@ addr = COLOUR_ALIGN(addr, pgoff); else addr = PAGE_ALIGN(addr); + task_size -= len; for (vmm = find_vma(current->mm, addr); ; vmm = vmm->vm_next) { @@ -89,6 +106,10 @@ addr = vmm->vm_end; if (do_color_align) addr = COLOUR_ALIGN(addr, pgoff); +#if CONFIG_SUPER_PAGE + else + addr = (addr + super_page_mask)&~super_page_mask; +#endif } } diff -urN linux-2.4.19/arch/sparc64/mm/init.c linux/arch/sparc64/mm/init.c --- linux-2.4.19/arch/sparc64/mm/init.c Sat Aug 3 09:39:43 2002 +++ linux/arch/sparc64/mm/init.c Tue Aug 6 19:52:31 2002 @@ -1038,6 +1038,13 @@ struct pgtable_cache_struct pgt_quicklists; #endif +#ifdef CONFIG_SUPER_PAGE +int super_page_order[SUPER_PAGE_NR] = {0,3,6,9}; +pgprot_t super_page_prot[SUPER_PAGE_NR] = + {__pgprot(_PAGE_SZ8K),__pgprot(_PAGE_SZ64K), + __pgprot(_PAGE_SZ512K),__pgprot(_PAGE_SZ4MB)}; +#endif + /* OK, we have to color these pages. The page tables are accessed * by non-Dcache enabled mapping in the VPTE area by the dtlb_backend.S * code, as well as by PAGE_OFFSET range direct-mapped addresses by diff -urN linux-2.4.19/include/asm-alpha/pgtable.h linux/include/asm-alpha/pgtable.h --- linux-2.4.19/include/asm-alpha/pgtable.h Sat Aug 3 09:39:45 2002 +++ linux/include/asm-alpha/pgtable.h Tue Aug 6 19:52:31 2002 @@ -19,7 +19,10 @@ * within a page table are directly modified. Thus, the following * hook is made available. */ -#define set_pte(pteptr, pteval) ((*(pteptr)) = (pteval)) +/* + * SUPER_PAGE support added by Naohiko Shimizu + <nshimizu@keyaki.cc.u-tokai.ac.jp> + */ /* PMD_SHIFT determines the size of the area a second-level page table can map */ #define PMD_SHIFT (PAGE_SHIFT + (PAGE_SHIFT-3)) @@ -254,9 +257,7 @@ extern inline unsigned long pgd_page(pgd_t pgd) { return PAGE_OFFSET + ((pgd_val(pgd) & _PFN_MASK) >> (32-PAGE_SHIFT)); } -extern inline int pte_none(pte_t pte) { return !pte_val(pte); } extern inline int pte_present(pte_t pte) { return pte_val(pte) & _PAGE_VALID; } -extern inline void pte_clear(pte_t *ptep) { pte_val(*ptep) = 0; } extern inline int pmd_none(pmd_t pmd) { return !pmd_val(pmd); } extern inline int pmd_bad(pmd_t pmd) { return (pmd_val(pmd) & ~_PFN_MASK) != _PAGE_TABLE; } @@ -268,6 +269,59 @@ extern inline int pgd_present(pgd_t pgd) { return pgd_val(pgd) & _PAGE_VALID; } extern inline void pgd_clear(pgd_t * pgdp) { pgd_val(*pgdp) = 0; } +#define clear_pmd_sp(pmd) do {} while(0) +#define super_page_populate(adr, page, prot, index) do {} while (0) + +#ifndef CONFIG_SUPER_PAGE +#define SUPER_PAGE_NR 1 +#define SUPER_PAGE_MASK 0 +#define set_pte(pteptr, pteval) ((*(pteptr)) = (pteval)) +extern inline int pte_none(pte_t pte) { return !(pte_val(pte)); } +extern inline void pte_clear(pte_t *ptep) { pte_val(*ptep)=0; } +#else +#define SUPER_PAGE_MASK 0x0060 +#define SUPER_PAGE_MASK_SHIFT 5 +#define SUPER_PAGE_NR 4 +extern int super_page_order[]; +extern pgprot_t super_page_prot[]; +extern inline int pte_none(pte_t pte) { return !(pte_val(pte) & ~SUPER_PAGE_MASK); } +#define pte_to_sp_index(x) ((pte_val(x) & SUPER_PAGE_MASK) >> SUPER_PAGE_MASK_SHIFT) +extern inline pte_t mk_pte_sp_clean(pte_t pte) {pte_val(pte) &= ~SUPER_PAGE_MASK; return pte;} +extern inline void down_pte_sp(pte_t *pteptr, int index) { + int i,order; + pte_t *addr; + order = super_page_order[index]; + addr = (pte_t *)((unsigned long) pteptr & ~((1UL<<(order + SIZEOF_PTR_LOG2)) - 1)); + for ( i=0; i < 1<<order; i++) { + pte_val(*(addr+i)) = (pte_val(*(addr+i)) & ~SUPER_PAGE_MASK) | pgprot_val(super_page_prot[index -1]); + } + } +extern inline void clear_pte_sp(pte_t *pteptr, int index) { + int i,order; + pte_t *addr; + order = super_page_order[index]; + addr = (pte_t *)((unsigned long) pteptr & ~((1UL<<(order + SIZEOF_PTR_LOG2)) - 1)); + for ( i=0; i < 1<<order; i++) { + pte_val(*(addr+i)) &= ~SUPER_PAGE_MASK; + } + } +extern inline void set_pte_raw(pte_t *pteptr, pte_t pteval) { + retry: + if ( pte_present(*pteptr) && ( pte_val(*pteptr) & SUPER_PAGE_MASK )) { + down_pte_sp(pteptr, pte_to_sp_index(*pteptr)); + goto retry; + } + *pteptr = pteval; +} +extern inline void set_pte(pte_t *pteptr, pte_t pteval) { + if ( pte_none(*pteptr) && ( pte_val(*pteptr) & SUPER_PAGE_MASK )) { + clear_pte_sp(pteptr, pte_to_sp_index(*pteptr)); + } + set_pte_raw(pteptr, mk_pte_sp_clean(pteval)); +} +extern inline void pte_clear(pte_t *ptep) { pte_t pte; pte_val(pte)=0; set_pte(ptep, pte); } +#endif + /* * The following only work if pte_present() is true. * Undefined behaviour if not.. diff -urN linux-2.4.19/include/asm-i386/mmu_context.h linux/include/asm-i386/mmu_context.h --- linux-2.4.19/include/asm-i386/mmu_context.h Sat Aug 3 09:39:45 2002 +++ linux/include/asm-i386/mmu_context.h Tue Aug 6 20:48:00 2002 @@ -42,7 +42,11 @@ set_bit(cpu, &next->cpu_vm_mask); set_bit(cpu, &next->context.cpuvalid); /* Re-load page tables */ +#ifdef CONFIG_SUPER_PAGE + load_cr3(next->pgd+PTRS_PER_PGD); +#else load_cr3(next->pgd); +#endif } #ifdef CONFIG_SMP else { diff -urN linux-2.4.19/include/asm-i386/pgalloc.h linux/include/asm-i386/pgalloc.h --- linux-2.4.19/include/asm-i386/pgalloc.h Sat Aug 3 09:39:45 2002 +++ linux/include/asm-i386/pgalloc.h Tue Aug 6 20:48:00 2002 @@ -11,6 +11,29 @@ #define pte_quicklist (current_cpu_data.pte_quick) #define pgtable_cache_size (current_cpu_data.pgtable_cache_sz) +#if CONFIG_SUPER_PAGE +static inline void super_page_populate(unsigned long address, + struct page *page, pgprot_t prot, int spindex) { + pgd_t *pgd; + pmd_t *pmd; + union { + pmd_t entry; + pte_t pte_entry; + } x; + pgd = pgd_offset(current->mm, address); + pmd = pmd_offset(pgd, address); + x.pte_entry = mk_pte(page, __pgprot(pgprot_val(prot)|_PAGE_PSE)) ; + x.pte_entry = pte_mkwrite(pte_mkdirty(x.pte_entry)); +#if defined (CONFIG_X86_PAE) + set_pmd_raw(pmd + PTRS_PER_PMD, x.entry); +#else + set_pmd_raw(pmd + PTRS_PER_PGD, x.entry); +#endif +} +#else +#define super_page_populate(address, page, prot, spindex) do {} while(0) +#endif + #define pmd_populate(mm, pmd, pte) \ set_pmd(pmd, __pmd(_PAGE_TABLE + __pa(pte))) @@ -36,15 +59,31 @@ if (pgd) { for (i = 0; i < USER_PTRS_PER_PGD; i++) { +#if CONFIG_SUPER_PAGE + unsigned long pmd = __get_free_pages(GFP_KERNEL,1); + if (!pmd) + goto out_oom; + clear_page(pmd + (PTRS_PER_PMD*sizeof(pmd_t))); +#else unsigned long pmd = __get_free_page(GFP_KERNEL); if (!pmd) goto out_oom; +#endif clear_page(pmd); set_pgd(pgd + i, __pgd(1 + __pa(pmd))); +#if CONFIG_SUPER_PAGE + set_pgd_raw(pgd + i + PTRS_PER_PGD, + __pgd(1 + __pa(pmd + (PTRS_PER_PMD*sizeof(pmd_t))))); +#endif } memcpy(pgd + USER_PTRS_PER_PGD, swapper_pg_dir + USER_PTRS_PER_PGD, (PTRS_PER_PGD - USER_PTRS_PER_PGD) * sizeof(pgd_t)); +#if CONFIG_SUPER_PAGE + memcpy(pgd + PTRS_PER_PGD + USER_PTRS_PER_PGD, + swapper_pg_dir + USER_PTRS_PER_PGD, + (PTRS_PER_PGD - USER_PTRS_PER_PGD) * sizeof(pgd_t)); +#endif } return pgd; out_oom: @@ -58,6 +97,21 @@ static inline pgd_t *get_pgd_slow(void) { +#if CONFIG_SUPER_PAGE + pgd_t *pgd = (pgd_t *)__get_free_pages(GFP_KERNEL,1); + + if (pgd) { + memset(pgd, 0, USER_PTRS_PER_PGD * sizeof(pgd_t)); + memcpy(pgd + USER_PTRS_PER_PGD, + swapper_pg_dir + USER_PTRS_PER_PGD, + (PTRS_PER_PGD - USER_PTRS_PER_PGD) * sizeof(pgd_t)); + memset(pgd + PTRS_PER_PGD, + 0, USER_PTRS_PER_PGD * sizeof(pgd_t)); + memcpy(pgd + PTRS_PER_PGD + USER_PTRS_PER_PGD, + swapper_pg_dir + USER_PTRS_PER_PGD, + (PTRS_PER_PGD - USER_PTRS_PER_PGD) * sizeof(pgd_t)); + } +#else pgd_t *pgd = (pgd_t *)__get_free_page(GFP_KERNEL); if (pgd) { @@ -66,6 +120,7 @@ swapper_pg_dir + USER_PTRS_PER_PGD, (PTRS_PER_PGD - USER_PTRS_PER_PGD) * sizeof(pgd_t)); } +#endif return pgd; } @@ -96,12 +151,21 @@ #if defined(CONFIG_X86_PAE) int i; +#if CONFIG_SUPER_PAGE + for (i = 0; i < USER_PTRS_PER_PGD; i++) + free_pages((unsigned long)__va(pgd_val(pgd[i])-1), 1); +#else for (i = 0; i < USER_PTRS_PER_PGD; i++) free_page((unsigned long)__va(pgd_val(pgd[i])-1)); +#endif kmem_cache_free(pae_pgd_cachep, pgd); #else +#if CONFIG_SUPER_PAGE + free_pages((unsigned long)pgd, 1); +#else free_page((unsigned long)pgd); #endif +#endif } static inline pte_t *pte_alloc_one(struct mm_struct *mm, unsigned long address) diff -urN linux-2.4.19/include/asm-i386/pgtable-2level.h linux/include/asm-i386/pgtable-2level.h --- linux-2.4.19/include/asm-i386/pgtable-2level.h Fri Jul 27 05:40:32 2001 +++ linux/include/asm-i386/pgtable-2level.h Tue Aug 6 20:48:00 2002 @@ -39,13 +39,28 @@ * within a page table are directly modified. Thus, the following * hook is made available. */ +#ifndef CONFIG_SUPER_PAGE #define set_pte(pteptr, pteval) (*(pteptr) = pteval) +#endif /* * (pmds are folded into pgds so this doesnt get actually called, * but the define is needed for a generic inline function.) */ +#if CONFIG_SUPER_PAGE +#define set_pmd_raw(pmdptr, pmdval) (*(pmdptr) = pmdval) +#define set_pgd_raw(pgdptr, pgdval) (*(pgdptr) = pgdval) +#define set_pmd(pmdptr, pmdval) do {\ + set_pmd_raw(pmdptr, pmdval);\ + set_pmd_raw(pmdptr+PTRS_PER_PGD, pmdval);\ + } while (0) +#define set_pgd(pgdptr, pgdval) do {\ + set_pgd_raw(pgdptr, pgdval);\ + set_pgd_raw(pgdptr+PTRS_PER_PGD, pgdval);\ + } while (0) +#else #define set_pmd(pmdptr, pmdval) (*(pmdptr) = pmdval) #define set_pgd(pgdptr, pgdval) (*(pgdptr) = pgdval) +#endif #define pgd_page(pgd) \ ((unsigned long) __va(pgd_val(pgd) & PAGE_MASK)) @@ -55,9 +70,9 @@ return (pmd_t *) dir; } #define ptep_get_and_clear(xp) __pte(xchg(&(xp)->pte_low, 0)) -#define pte_same(a, b) ((a).pte_low == (b).pte_low) +#define pte_same(a, b) ((a).pte_low == (b).pte_low) #define pte_page(x) (mem_map+((unsigned long)(((x).pte_low >> PAGE_SHIFT)))) -#define pte_none(x) (!(x).pte_low) +#define pte_none(x) (!((x).pte_low&~SUPER_PAGE_MASK)) #define __mk_pte(page_nr,pgprot) __pte(((page_nr) << PAGE_SHIFT) | pgprot_val(pgprot)) #endif /* _I386_PGTABLE_2LEVEL_H */ diff -urN linux-2.4.19/include/asm-i386/pgtable-3level.h linux/include/asm-i386/pgtable-3level.h --- linux-2.4.19/include/asm-i386/pgtable-3level.h Fri Jul 27 05:40:32 2001 +++ linux/include/asm-i386/pgtable-3level.h Tue Aug 6 20:48:00 2002 @@ -43,16 +43,33 @@ * not possible, use pte_get_and_clear to obtain the old pte * value and then use set_pte to update it. -ben */ +#ifndef CONFIG_SUPER_PAGE static inline void set_pte(pte_t *ptep, pte_t pte) { ptep->pte_high = pte.pte_high; smp_wmb(); ptep->pte_low = pte.pte_low; } +#endif +#if CONFIG_SUPER_PAGE +#define set_pgd_raw(pgdptr,pgdval) \ + set_64bit((unsigned long long *)(pgdptr),pgd_val(pgdval)) +#define set_pmd_raw(pmdptr,pmdval) \ + set_64bit((unsigned long long *)(pmdptr),pmd_val(pmdval)) +#define set_pmd(pmdptr,pmdval) do {\ + set_pmd_raw(pmdptr,pmdval);\ + set_pmd_raw(pmdptr+PTRS_PER_PMD,pmdval);\ + } while(0) +#define set_pgd(pgdptr,pgdval) do {\ + set_pgd_raw(pgdptr,pgdval);\ + set_pgd_raw(pgdptr+PTRS_PER_PGD,pgdval);\ + } while(0) +#else #define set_pmd(pmdptr,pmdval) \ set_64bit((unsigned long long *)(pmdptr),pmd_val(pmdval)) #define set_pgd(pgdptr,pgdval) \ set_64bit((unsigned long long *)(pgdptr),pgd_val(pgdval)) +#endif /* * Pentium-II erratum A13: in PAE mode we explicitly have to flush @@ -87,7 +104,7 @@ } #define pte_page(x) (mem_map+(((x).pte_low >> PAGE_SHIFT) | ((x).pte_high << (32 - PAGE_SHIFT)))) -#define pte_none(x) (!(x).pte_low && !(x).pte_high) +#define pte_none(x) (!((x).pte_low&~SUPER_PAGE_MASK) && !(x).pte_high) static inline pte_t __mk_pte(unsigned long page_nr, pgprot_t pgprot) { diff -urN linux-2.4.19/include/asm-i386/pgtable.h linux/include/asm-i386/pgtable.h --- linux-2.4.19/include/asm-i386/pgtable.h Sat Aug 3 09:39:45 2002 +++ linux/include/asm-i386/pgtable.h Tue Aug 6 20:48:00 2002 @@ -21,7 +21,11 @@ #include <asm/bitops.h> #endif +#if CONFIG_SUPER_PAGE +extern pgd_t swapper_pg_dir[2048]; +#else extern pgd_t swapper_pg_dir[1024]; +#endif extern void paging_init(void); /* Caches aren't brain-dead on the intel. */ @@ -98,6 +102,14 @@ #endif /* !__ASSEMBLY__ */ +#if CONFIG_SUPER_PAGE +#define SUPER_PAGE_MASK _PAGE_SUPER +#define SUPER_PAGE_MASK_SHIFT 9 +#define SUPER_PAGE_NR 2 +#else +#define SUPER_PAGE_NR 1 +#define SUPER_PAGE_MASK 0 +#endif /* * The Linux x86 paging architecture is 'compile-time dual-mode', it * implements both the traditional 2-level x86 page tables and the @@ -185,6 +197,7 @@ #define _PAGE_DIRTY 0x040 #define _PAGE_PSE 0x080 /* 4 MB (or 2MB) page, Pentium+, if present.. */ #define _PAGE_GLOBAL 0x100 /* Global TLB entry PPro+ */ +#define _PAGE_SUPER 0x200 /* SuperPage candidated page */ #define _PAGE_PROTNONE 0x080 /* If not present */ @@ -293,6 +306,71 @@ static inline void ptep_set_wrprotect(pte_t *ptep) { clear_bit(_PAGE_BIT_RW, ptep); } static inline void ptep_mkdirty(pte_t *ptep) { set_bit(_PAGE_BIT_DIRTY, ptep); } +#if CONFIG_X86_PAE +#define SIZEOF_PTR_LOG2 3 +#else +#define SIZEOF_PTR_LOG2 2 +#endif +#if CONFIG_SUPER_PAGE +extern pgprot_t super_page_prot[]; +extern int super_page_order[]; +#define pte_to_sp_index(x) (((x).pte_low & SUPER_PAGE_MASK) >> SUPER_PAGE_MASK_SHIFT) +static inline pte_t mk_pte_sp_clean(pte_t pte) + {(pte).pte_low &= ~SUPER_PAGE_MASK; return pte;} +static inline void clear_pmd_sp(pmd_t *pmd) { +#if CONFIG_X86_PAE + set_pmd_raw((pmd+PTRS_PER_PMD), *pmd); +#else + set_pmd_raw((pmd+PTRS_PER_PGD), *pmd); + +#endif + } +static inline void down_pte_sp(pte_t *pteptr, int index) { + int i,order; + pte_t *addr; + order = super_page_order[index]; + addr = (pte_t *)((unsigned long) pteptr & + ~((1UL<<(order + SIZEOF_PTR_LOG2)) - 1)); + for ( i=0; i < 1<<order; i++) { + (*(addr+i)).pte_low = + ((*(addr+i)).pte_low & ~SUPER_PAGE_MASK) | + pgprot_val(super_page_prot[index -1]); + } + } +static inline void clear_pte_sp(pte_t *pteptr, int index) { + int i,order; + pte_t *addr; + order = super_page_order[index]; + addr = (pte_t *)((unsigned long) pteptr & + ~((1UL<<(order + SIZEOF_PTR_LOG2)) - 1)); + for ( i=0; i < 1<<order; i++) { + (*(addr+i)).pte_low &= ~SUPER_PAGE_MASK; + } + } +#if CONFIG_X86_PAE +extern inline void set_pte_raw(pte_t *pteptr, pte_t pteval) { + if ( pte_present(*pteptr) && ( (*pteptr).pte_low & SUPER_PAGE_MASK )) { + down_pte_sp(pteptr, pte_to_sp_index(*pteptr)); + } + pteptr->pte_high = pte.pte_high; + smp_wmb(); + pteptr->pte_low = pte.pte_low; +} +#else +extern inline void set_pte_raw(pte_t *pteptr, pte_t pteval) { + if ( pte_present(*pteptr) && ( (*pteptr).pte_low & SUPER_PAGE_MASK )) { + down_pte_sp(pteptr, pte_to_sp_index(*pteptr)); + } + *pteptr = pteval; +} +#endif +static inline void set_pte(pte_t *pteptr, pte_t pteval) { + if ( pte_none(*pteptr) && ( (*pteptr).pte_low & SUPER_PAGE_MASK )) { + clear_pte_sp(pteptr, pte_to_sp_index(*pteptr)); + } + set_pte_raw(pteptr, mk_pte_sp_clean(pteval)); +} +#endif /* * Conversion functions: convert a page and protection to a page entry, * and a page entry and page directory to the page they refer to. @@ -342,11 +420,12 @@ /* Encode and de-code a swap entry */ #define SWP_TYPE(x) (((x).val >> 1) & 0x3f) -#define SWP_OFFSET(x) ((x).val >> 8) -#define SWP_ENTRY(type, offset) ((swp_entry_t) { ((type) << 1) | ((offset) << 8) }) +#define SWP_OFFSET(x) ((x).val >> 12) +#define SWP_ENTRY(type, offset) ((swp_entry_t) { ((type) << 1) | ((offset) << 12) }) #define pte_to_swp_entry(pte) ((swp_entry_t) { (pte).pte_low }) #define swp_entry_to_pte(x) ((pte_t) { (x).val }) + #endif /* !__ASSEMBLY__ */ /* Needs to be defined here and not in linux/mm.h, as it is arch dependent */ diff -urN linux-2.4.19/include/asm-sparc64/pgtable.h linux/include/asm-sparc64/pgtable.h --- linux-2.4.19/include/asm-sparc64/pgtable.h Sat Aug 3 09:39:45 2002 +++ linux/include/asm-sparc64/pgtable.h Tue Aug 6 19:52:31 2002 @@ -61,13 +61,16 @@ #define PGDIR_SIZE (1UL << PGDIR_SHIFT) #define PGDIR_MASK (~(PGDIR_SIZE-1)) +#define SIZEOF_PTR_LOG2 3 + #ifndef __ASSEMBLY__ /* Certain architectures need to do special things when pte's * within a page table are directly modified. Thus, the following * hook is made available. - */ +Moved for the SuperPage. by Nahiko Shimizu. #define set_pte(pteptr, pteval) ((*(pteptr)) = (pteval)) + */ /* Entries per page directory level. */ #define PTRS_PER_PTE (1UL << (PAGE_SHIFT-3)) @@ -135,7 +138,7 @@ #elif PAGE_SHIFT == 19 #define _PAGE_SZBITS _PAGE_SZ512K #elif PAGE_SHIFT == 22 -#define _PAGE_SZBITS _PAGE_SZ4M +#define _PAGE_SZBITS _PAGE_SZ4MB #else #error Wrong PAGE_SHIFT specified #endif @@ -217,9 +220,7 @@ (pgd_val(*(pgdp)) = (__pa((unsigned long) (pmdp)) >> 11UL)) #define pmd_page(pmd) ((unsigned long) __va((pmd_val(pmd)<<11UL))) #define pgd_page(pgd) ((unsigned long) __va((pgd_val(pgd)<<11UL))) -#define pte_none(pte) (!pte_val(pte)) #define pte_present(pte) (pte_val(pte) & _PAGE_PRESENT) -#define pte_clear(pte) (pte_val(*(pte)) = 0UL) #define pmd_none(pmd) (!pmd_val(pmd)) #define pmd_bad(pmd) (0) #define pmd_present(pmd) (pmd_val(pmd) != 0UL) @@ -229,6 +230,59 @@ #define pgd_present(pgd) (pgd_val(pgd) != 0UL) #define pgd_clear(pgdp) (pgd_val(*(pgdp)) = 0UL) +#define clear_pmd_sp(pmd) do {} while(0) +#define super_page_populate(adr, page, prot, index) do {} while (0) + +#ifndef CONFIG_SUPER_PAGE +#define SUPER_PAGE_NR 1 +#define SUPER_PAGE_MASK 0 +#define set_pte(pteptr, pteval) ((*(pteptr)) = (pteval)) +#define pte_none(pte) (!pte_val(pte)) +#define pte_clear(pte) (pte_val(*(pte)) = 0UL) +#else +#define SUPER_PAGE_MASK (_PAGE_SZ64K|_PAGE_SZ512K|_PAGE_SZ4MB) +#define SUPER_PAGE_MASK_SHIFT 61 +#define SUPER_PAGE_NR 4 +extern int super_page_order[]; +extern pgprot_t super_page_prot[]; +extern inline int pte_none(pte_t pte) { return !(pte_val(pte) & ~SUPER_PAGE_MASK); } +#define pte_to_sp_index(x) ((pte_val(x) & SUPER_PAGE_MASK) >> SUPER_PAGE_MASK_SHIFT) +extern inline pte_t mk_pte_sp_clean(pte_t pte) {pte_val(pte) &= ~SUPER_PAGE_MASK; return pte;} +extern inline void down_pte_sp(pte_t *pteptr, int index) { + int i,order; + pte_t *addr; + order = super_page_order[index]; + addr = (pte_t *)((unsigned long) pteptr & ~((1UL<<(order + SIZEOF_PTR_LOG2)) - 1)); + for ( i=0; i < 1<<order; i++) { + pte_val(*(addr+i)) = (pte_val(*(addr+i)) & ~SUPER_PAGE_MASK) | pgprot_val(super_page_prot[index -1]); + } + } +extern inline void clear_pte_sp(pte_t *pteptr, int index) { + int i,order; + pte_t *addr; + order = super_page_order[index]; + addr = (pte_t *)((unsigned long) pteptr & ~((1UL<<(order + SIZEOF_PTR_LOG2)) - 1)); + for ( i=0; i < 1<<order; i++) { + pte_val(*(addr+i)) &= ~SUPER_PAGE_MASK; + } + } +extern inline void set_pte_raw(pte_t *pteptr, pte_t pteval) { + retry: + if ( pte_present(*pteptr) && ( pte_val(*pteptr) & SUPER_PAGE_MASK )) { + down_pte_sp(pteptr, pte_to_sp_index(*pteptr)); + goto retry; + } + *pteptr = pteval; +} +extern inline void set_pte(pte_t *pteptr, pte_t pteval) { + if ( pte_none(*pteptr) && ( pte_val(*pteptr) & SUPER_PAGE_MASK )) { + clear_pte_sp(pteptr, pte_to_sp_index(*pteptr)); + } + set_pte_raw(pteptr, mk_pte_sp_clean(pteval)); +} +extern inline void pte_clear(pte_t *ptep) { pte_t pte; pte_val(pte)=0; set_pte(ptep, pte); } +#endif + /* The following only work if pte_present() is true. * Undefined behaviour if not.. */ diff -urN linux-2.4.19/include/linux/mm.h linux/include/linux/mm.h --- linux-2.4.19/include/linux/mm.h Sat Aug 3 09:39:45 2002 +++ linux/include/linux/mm.h Tue Aug 6 20:48:00 2002 @@ -403,6 +403,21 @@ #define PageHighMem(page) 0 /* needed to optimize away at compile time */ #endif +#ifdef CONFIG_SUPER_PAGE +extern int super_page_nr; +extern unsigned long super_page_reserve[]; +extern unsigned long super_page_allocate[]; +extern unsigned long super_page_downgrade[]; +void super_page_init(void); +#else +#define pte_to_sp_index(x) (0) +#define down_pte_sp(pte, order) +#define set_pte_raw(ptep, pte) set_pte(ptep, pte) +#define super_page_populate(adr, page, prot, index) +#define clear_pmd_sp(pmd) +#endif + + #define SetPageReserved(page) set_bit(PG_reserved, &(page)->flags) #define ClearPageReserved(page) clear_bit(PG_reserved, &(page)->flags) @@ -481,6 +496,7 @@ extern pte_t *FASTCALL(pte_alloc(struct mm_struct *mm, pmd_t *pmd, unsigned long address)); extern int handle_mm_fault(struct mm_struct *mm,struct vm_area_struct *vma, unsigned long address, int write_access); extern int make_pages_present(unsigned long addr, unsigned long end); +extern int make_ptes_present(unsigned long addr, unsigned long end); extern int access_process_vm(struct task_struct *tsk, unsigned long addr, void *buf, int len, int write); extern int ptrace_readdata(struct task_struct *tsk, unsigned long src, char *dst, int len); extern int ptrace_writedata(struct task_struct *tsk, char * src, unsigned long dst, int len); @@ -516,6 +532,12 @@ extern void si_meminfo(struct sysinfo * val); extern void swapin_readahead(swp_entry_t); +extern void __break_area (struct page *page, unsigned long order); +static inline void break_area(struct page *page, unsigned long order) +{ + __break_area(page, order); +} + extern struct address_space swapper_space; #define PageSwapCache(page) ((page)->mapping == &swapper_space) diff -urN linux-2.4.19/kernel/sysctl.c linux/kernel/sysctl.c --- linux-2.4.19/kernel/sysctl.c Sat Aug 3 09:39:46 2002 +++ linux/kernel/sysctl.c Tue Aug 6 19:52:31 2002 @@ -140,6 +140,9 @@ static void register_proc_table(ctl_table *, struct proc_dir_entry *); static void unregister_proc_table(ctl_table *, struct proc_dir_entry *); #endif +#if CONFIG_SUPER_PAGE +void super_page_init(void); +#endif /* The default sysctl tables: */ @@ -328,6 +331,9 @@ register_proc_table(root_table, proc_sys_root); init_irq_proc(); #endif +#ifdef CONFIG_SUPER_PAGE + super_page_init(); +#endif } int do_sysctl(int *name, int nlen, void *oldval, size_t *oldlenp, diff -urN linux-2.4.19/mm/Makefile linux/mm/Makefile --- linux-2.4.19/mm/Makefile Sat Aug 3 09:39:46 2002 +++ linux/mm/Makefile Tue Aug 6 19:52:31 2002 @@ -18,4 +18,6 @@ obj-$(CONFIG_HIGHMEM) += highmem.o +obj-$(CONFIG_SUPER_PAGE) += super_page.o + include $(TOPDIR)/Rules.make diff -urN linux-2.4.19/mm/memory.c linux/mm/memory.c --- linux-2.4.19/mm/memory.c Sat Aug 3 09:39:46 2002 +++ linux/mm/memory.c Tue Aug 6 21:01:56 2002 @@ -34,6 +34,8 @@ * * 16.07.99 - Support of BIGMEM added by Gerhard Wichert, Siemens AG * (Gerhard.Wichert@pdb.siemens.de) + * 13.06.00 - Support of SUPER_PAGE added by Naohiko Shimizu, Tokai Univ. + <nshimizu@keyaki.cc.u-tokai.ac.jp> */ #include <linux/mm.h> @@ -56,6 +58,15 @@ void * high_memory; struct page *highmem_start_page; +#ifdef CONFIG_SUPER_PAGE +void adj_sp_range(struct mm_struct *mm, int zap, + unsigned long address, unsigned long end); +#else +int super_page_order[] = {0}; +pgprot_t super_page_prot[] = {0x00}; +#endif + + /* * We special-case the C-O-W ZERO_PAGE, because it's such * a common occurrence (no need to read the page to know @@ -184,6 +195,14 @@ unsigned long end = vma->vm_end; unsigned long cow = (vma->vm_flags & (VM_SHARED | VM_MAYWRITE)) == VM_MAYWRITE; +#if 0&&CONFIG_SUPER_PAGE + if(cow) { + spin_lock(&src->page_table_lock); + adj_sp_range(src, 0, address, end); + spin_unlock(&src->page_table_lock); + } +#endif + src_pgd = pgd_offset(src, address)-1; dst_pgd = pgd_offset(dst, address)-1; @@ -313,9 +332,16 @@ size = PMD_SIZE - offset; size &= PAGE_MASK; for (offset=0; offset < size; ptep++, offset += PAGE_SIZE) { - pte_t pte = *ptep; + pte_t pte; + sp_retry: + pte = *ptep; if (pte_none(pte)) - continue; + { + if(pte_to_sp_index(pte)) { + down_pte_sp(ptep,pte_to_sp_index(pte)); + goto sp_retry; + } + continue; } if (pte_present(pte)) { struct page *page = pte_page(pte); if (VALID_PAGE(page) && !PageReserved(page)) @@ -350,6 +376,7 @@ end = ((address + PGDIR_SIZE) & PGDIR_MASK); freed = 0; do { + clear_pmd_sp(pmd); freed += zap_pte_range(tlb, pmd, address, end - address); address = (address + PMD_SIZE) & PMD_MASK; pmd++; @@ -367,6 +394,12 @@ unsigned long start = address, end = address + size; int freed = 0; +#if CONFIG_SUPER_PAGE + spin_lock(&mm->page_table_lock); + adj_sp_range(mm, 1, address, end); + spin_unlock(&mm->page_table_lock); +#endif + dir = pgd_offset(mm, address); /* @@ -781,6 +814,7 @@ pte_t * pte = pte_alloc(mm, pmd, address); if (!pte) return -ENOMEM; + clear_pmd_sp(pmd); zeromap_pte_range(pte, address, end - address, prot); address = (address + PMD_SIZE) & PMD_MASK; pmd++; @@ -796,6 +830,11 @@ unsigned long end = address + size; struct mm_struct *mm = current->mm; +#if CONFIG_SUPER_PAGE + spin_lock(&mm->page_table_lock); + adj_sp_range(mm, 1, address, end); + spin_unlock(&mm->page_table_lock); +#endif dir = pgd_offset(mm, address); flush_cache_range(mm, beg, end); if (address >= end) @@ -861,6 +900,7 @@ pte_t * pte = pte_alloc(mm, pmd, address); if (!pte) return -ENOMEM; + clear_pmd_sp(pmd); remap_pte_range(pte, address, end - address, address + phys_addr, prot); address = (address + PMD_SIZE) & PMD_MASK; pmd++; @@ -877,6 +917,11 @@ unsigned long end = from + size; struct mm_struct *mm = current->mm; +#if CONFIG_SUPER_PAGE + spin_lock(&mm->page_table_lock); + adj_sp_range(mm, 1, beg, end); + spin_unlock(&mm->page_table_lock); +#endif phys_addr -= from; dir = pgd_offset(mm, from); flush_cache_range(mm, beg, end); @@ -959,8 +1004,32 @@ int reuse = can_share_swap_page(old_page); unlock_page(old_page); if (reuse) { +#if 0&&CONFIG_SUPER_PAGE + int order; + struct page *wkpage; + unsigned long offset; + sp_retry: + order = super_page_order[pte_to_sp_index(*page_table)]; + offset = (old_page - mem_map) & ~(1UL<<order -1); + wkpage = mem_map + offset; + if(order) + for(i=0; i<1<<order; i++) { + if(page_count(wkpage+i)>1) { + down_pte_sp(page_table, pte_to_sp_index(*page_table)); + goto sp_retry; + } + } + for(i=0; i<1<<order; i++) { + flush_cache_page(vma, address); + establish_pte(vma, address, page_table+i, + pte_mkyoung(pte_mkdirty(pte_mkwrite(*(page_table+i))))); + address += PAGE_SIZE; + } +#else flush_cache_page(vma, address); - establish_pte(vma, address, page_table, pte_mkyoung(pte_mkdirty(pte_mkwrite(pte)))); + establish_pte(vma, address, page_table, + pte_mkyoung(pte_mkdirty(pte_mkwrite(pte)))); +#endif spin_unlock(&mm->page_table_lock); return 1; /* Minor fault */ } @@ -1191,34 +1260,83 @@ static int do_anonymous_page(struct mm_struct * mm, struct vm_area_struct * vma, pte_t *page_table, int write_access, unsigned long addr) { pte_t entry; - + int i; + unsigned long spaddr; + pte_t oldpte; + int order; + pte_t *wktable; /* Read-only mapping of ZERO_PAGE. */ entry = pte_wrprotect(mk_pte(ZERO_PAGE(addr), vma->vm_page_prot)); /* ..except if it's a write access */ if (write_access) { struct page *page; +retry: + oldpte = *page_table; + order = super_page_order[pte_to_sp_index(oldpte)]; + wktable = + (pte_t *)((unsigned long)page_table & + ~((1UL << (order + SIZEOF_PTR_LOG2)) -1)); + + for (i=0; i < 1 << order; i++) { + if(!pte_none(*(wktable+i))) { + down_pte_sp(page_table, pte_to_sp_index(oldpte)); + goto retry; + } + } /* Allocate our own private page. */ spin_unlock(&mm->page_table_lock); - page = alloc_page(GFP_HIGHUSER); - if (!page) - goto no_mem; - clear_user_highpage(page, addr); - + page = alloc_pages(GFP_HIGHUSER, order); + if (!page) { + if (order) { + spin_lock(&mm->page_table_lock); + super_page_downgrade[pte_to_sp_index(oldpte)]++; + down_pte_sp(page_table, pte_to_sp_index(oldpte)); + goto retry; + } + else goto no_mem; + } + spaddr = addr & ~((PAGE_SIZE << order) - 1); + for (i=0; i < 1 << order; i++) { + clear_user_highpage(page+i, spaddr); + spaddr += PAGE_SIZE; + } + if (order) { + break_area(page, order); + super_page_allocate[pte_to_sp_index(oldpte)]++; + spin_lock(&mm->page_table_lock); + spaddr = addr & ~((PAGE_SIZE << order) - 1); + super_page_populate(spaddr, page, vma->vm_page_prot, + pte_to_sp_index(oldpte)); + for (i=0; i < 1 << order; i++) { + entry = pte_mkwrite(pte_mkdirty( + mk_pte(page+i, __pgprot(pgprot_val(vma->vm_page_prot)| + pgprot_val(super_page_prot[pte_to_sp_index(oldpte)]))))); + mm->rss++; + flush_page_to_ram(page+i); + lru_cache_add(page+i); + mark_page_accessed(page+i); + set_pte_raw(wktable+i, entry); + spaddr += PAGE_SIZE; + } + } else{ spin_lock(&mm->page_table_lock); if (!pte_none(*page_table)) { page_cache_release(page); spin_unlock(&mm->page_table_lock); return 1; } - mm->rss++; - flush_page_to_ram(page); - entry = pte_mkwrite(pte_mkdirty(mk_pte(page, vma->vm_page_prot))); + mm->rss++; + flush_page_to_ram(page); + entry = pte_mkwrite(pte_mkdirty( + mk_pte(page, vma->vm_page_prot))); lru_cache_add(page); mark_page_accessed(page); - } + set_pte(page_table, entry); + } + } else set_pte(page_table, entry); diff -urN linux-2.4.19/mm/mmap.c linux/mm/mmap.c --- linux-2.4.19/mm/mmap.c Sat Aug 3 09:39:46 2002 +++ linux/mm/mmap.c Wed Aug 7 20:30:52 2002 @@ -583,6 +583,18 @@ mm->locked_vm += len >> PAGE_SHIFT; make_pages_present(addr, addr + len); } + /* + * mmap may provide new pages and we have the chance to set the + * SUPER_PAGE bits in the TLB entries, then we call make_ptes_present. + * MAP_PRIVATE flag gives a chance to tell us + * that it is a plain allocation. + * <N.Shimizu> + */ +#ifdef CONFIG_SUPER_PAGE + if (vm_flags & MAP_PRIVATE) { + make_ptes_present(addr, addr + len); + } +#endif return addr; unmap_and_free_vma: @@ -610,13 +622,28 @@ * This function "knows" that -ENOMEM has the bits set. */ #ifndef HAVE_ARCH_UNMAPPED_AREA +#if CONFIG_SUPER_PAGE +extern int super_page_vm_align; +#endif static inline unsigned long arch_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) { struct vm_area_struct *vma; + unsigned long super_page_mask=0; if (len > TASK_SIZE) return -ENOMEM; +#if CONFIG_SUPER_PAGE + if(super_page_vm_align) { + int i; + for(i=super_page_nr-1; i>0; i--) { + if(len>(PAGE_SIZE << super_page_order[i])) { + super_page_mask = (PAGE_SIZE << super_page_order[i]) -1; + break; + } + } + } +#endif if (addr) { addr = PAGE_ALIGN(addr); vma = find_vma(current->mm, addr); @@ -633,6 +660,9 @@ if (!vma || addr + len <= vma->vm_start) return addr; addr = vma->vm_end; +#if CONFIG_SUPER_PAGE + addr = (addr + super_page_mask)&~super_page_mask; +#endif } } #else @@ -1099,6 +1129,14 @@ mm->locked_vm += len >> PAGE_SHIFT; make_pages_present(addr, addr + len); } + /* + * brk provide new pages and we have the chance to set the + * SUPER_PAGE bits for TLB, then we call make_ptes_present. + * <N.Shimizu> + */ +#ifdef CONFIG_SUPER_PAGE + make_ptes_present(addr, addr + len); +#endif return addr; } diff -urN linux-2.4.19/mm/page_alloc.c linux/mm/page_alloc.c --- linux-2.4.19/mm/page_alloc.c Sat Aug 3 09:39:46 2002 +++ linux/mm/page_alloc.c Tue Aug 6 19:52:31 2002 @@ -10,6 +10,7 @@ * Reshaped it to be a zoned allocator, Ingo Molnar, Red Hat, 1999 * Discontiguous memory support, Kanoj Sarcar, SGI, Nov 1999 * Zone balancing, Kanoj Sarcar, SGI, Jan 2000 + * Page Granularity Hint support, Naohiko Shimizu, Tokai Univ., Jul 2000 */ #include <linux/config.h> diff -urN linux-2.4.19/mm/super_page.c linux/mm/super_page.c --- linux-2.4.19/mm/super_page.c Thu Jan 1 09:00:00 1970 +++ linux/mm/super_page.c Sat Aug 10 18:26:50 2002 @@ -0,0 +1,248 @@ +/* + Linux Super Page internal functions. +*/ +#include <linux/mm.h> +#include <linux/mman.h> +#include <linux/swap.h> +#include <linux/smp_lock.h> +#include <linux/swapctl.h> +#include <linux/iobuf.h> +#include <linux/highmem.h> +#include <linux/pagemap.h> + +#include <asm/pgalloc.h> +#include <asm/uaccess.h> +#include <asm/tlb.h> +#include <linux/proc_fs.h> +#include <linux/sysctl.h> + +/* We use arbitrary high number for the sysctl. You may have to change it.*/ + +#define CTL_SUPER_PAGE 4649 +#define CTL_SET_NR 1 +#define CTL_SET_ALIGN 2 +#define CTL_SET_BITMASK 3 +#define CTL_SET_LOGRES 4 + +int super_page_nr = 1; /* We start without super_page at first. */ +int super_page_vm_align = 0; /* We start without super_page align at first. */ +int super_page_tail_align = 0; /* We start without super_page tail align at first. */ +int super_page_bitmask = (1<<SUPER_PAGE_NR)-1; /* To control each order of the reservation. */ +int super_page_logreset = 0; /* If 1 then reset counter when dumped */ + +unsigned long super_page_reserve[SUPER_PAGE_NR]; +unsigned long super_page_allocate[SUPER_PAGE_NR]; +unsigned long super_page_downgrade[SUPER_PAGE_NR]; +#if CONFIG_SYSCTL +static ctl_table super_page_table[] = { + {CTL_SET_NR, "nr", &super_page_nr, sizeof(int), + 0644, NULL, &proc_dointvec}, + {CTL_SET_ALIGN, "vm_align", &super_page_vm_align, sizeof(int), + 0644, NULL, &proc_dointvec}, + {CTL_SET_ALIGN, "tail_align", &super_page_tail_align, sizeof(int), + 0644, NULL, &proc_dointvec}, + {CTL_SET_BITMASK, "bitmask", &super_page_bitmask, sizeof(int), + 0644, NULL, &proc_dointvec}, + {CTL_SET_LOGRES, "logreset", &super_page_logreset, sizeof(int), + 0644, NULL, &proc_dointvec}, + {0} +}; +static ctl_table sys_table[] = { + {CTL_SUPER_PAGE, "super_page", NULL, 0, 0555, super_page_table}, + {0} +}; + +#endif + +#if CONFIG_PROC_FS +int super_page_getinfo(char *buf, char **start, off_t fpos, int length) +{ + int i; + char *p = buf; + + p += sprintf(p, "current nr: %d\n", super_page_nr); + p += sprintf(p, "current vm_align: %d:%d\n", + super_page_vm_align, super_page_tail_align); + p += sprintf(p, "current bitmask: %d\n", super_page_bitmask); + p += sprintf(p, "order\treserve\tallocate\tfail \n"); + for(i=1;i<SUPER_PAGE_NR;i++) { + p += sprintf(p, "%d:\t%ld\t%ld\t%ld\n", + super_page_order[i], + super_page_reserve[i], + super_page_allocate[i], + super_page_downgrade[i] + ); + } + if(super_page_logreset) + for(i=1;i<SUPER_PAGE_NR;i++) { + super_page_reserve[i] = 0; + super_page_allocate[i] = 0; + super_page_downgrade[i] = 0; + } + return p - buf; + } +#endif + +void super_page_init() { + int i; + super_page_nr = SUPER_PAGE_NR; +#if CONFIG_PROC_FS + for(i=0;i<SUPER_PAGE_NR;i++) { + super_page_reserve[i] = 0; + super_page_allocate[i] = 0; + super_page_downgrade[i] = 0; + } + create_proc_info_entry("super_page", 0, NULL, super_page_getinfo); +#endif +#if CONFIG_SYSCTL + register_sysctl_table(sys_table,0); +#endif +} + + /* + * Allocating PTEs for future falt handling. + */ + +void set_sp_range(unsigned long address, int order, pgprot_t prot) + { + int i; + pgd_t * dir; + pmd_t *pmd; + pte_t * pte; + struct mm_struct *mm = current->mm; + + spin_lock(&mm->page_table_lock); + dir = pgd_offset(mm, address); + pmd = pmd_alloc(mm, dir, address); + if (!pmd) goto out; + address &= ~PGDIR_MASK; + pte = pte_alloc(mm, pmd, address); + if (!pte) + goto out; + for (i = 0; i < 1<<order; i ++) + if(!pte_none(*(pte+i))) goto out; + for (i = 0; i < 1<<order; i ++) { + set_pte_raw(pte, pte_modify(*pte, prot)); + pte++; + } +out: + spin_unlock(&mm->page_table_lock); + return; + } + + /* + * Simplistic new page table allocation for sys_brk.. + * Only GH bit != 0 tables will be allocated. + * At this time, we will not allocate real storage, it remains + * for the page_fault handler. + */ +int make_ptes_present(unsigned long addr, unsigned long end) + { + int i; + unsigned long rem; + if (addr >= end) + BUG(); + /* + * The first order(i=0) is the ordinary pte (1page), then we skip to + * allocate the pte. + */ + for (i = 0; i < super_page_nr - 1; i++) { + rem = (~addr + 1) & ((PAGE_SIZE << super_page_order[i+1]) - 1); + while (rem && + (addr & ((PAGE_SIZE << super_page_order[i]) - 1)) == 0UL && + ((end - addr ) >= (PAGE_SIZE << super_page_order[i]))) { + if(i&&super_page_bitmask & (1<<i)) { + super_page_reserve[i]++; + set_sp_range(addr, super_page_order[i], super_page_prot[i]); + } + addr += PAGE_SIZE << super_page_order[i]; + rem -= PAGE_SIZE << super_page_order[i]; + } + } + for (i = super_page_nr - 1; i > 0; i--) { + while ( + (addr & ((PAGE_SIZE << super_page_order[i]) - 1)) == 0UL && + ((end - addr ) >= (PAGE_SIZE << super_page_order[i]))) { + if(super_page_bitmask & (1<<i)) { + super_page_reserve[i]++; + set_sp_range(addr, super_page_order[i], super_page_prot[i]); + } + addr += PAGE_SIZE << super_page_order[i]; + } + } + return 0; + } + +void adj_sp_pte(struct mm_struct *mm, int zap, + unsigned long address, int order) + { + int i,downgrade; + pgd_t * dir; + pmd_t *pmd; + pte_t * pte; + + downgrade=0; + dir = pgd_offset(mm, address); + pmd = pmd_offset(dir, address); + if (!pmd) return; + address &= ~PGDIR_MASK; + pte = pte_offset(pmd, address); + if (!pte) return; +/* +We assume that the largest super page is less or equal to the +mapped area by the pmd. Then following code does not take +the pmd_offset again. If you want to use a larger super page, +you need to check the code. +*/ + for (i = 0; i < 1<<order; i ++) { + retry: + if(super_page_order[pte_to_sp_index(*(pte+i))]>order) { + down_pte_sp(pte+i,pte_to_sp_index(*(pte+i))); + downgrade=1; + goto retry; + } + } + if(zap) clear_pte_sp(pte, pte_to_sp_index(*pte)); + if(zap||downgrade) clear_pmd_sp(pmd); + return; + } + +void adj_sp_range(struct mm_struct *mm, int zap, + unsigned long addr, unsigned long end) + { + int i; + unsigned long rem; + if (addr >= end) + BUG(); + + for (i = 0; i < super_page_nr - 1; i++) { + rem = (~addr + 1) & ((PAGE_SIZE << super_page_order[i+1]) - 1); + while (rem && + (addr & ((PAGE_SIZE << super_page_order[i]) - 1)) == 0UL && + ((end - addr ) >= (PAGE_SIZE << super_page_order[i]))) { + adj_sp_pte(mm, zap, addr, super_page_order[i]); + addr += PAGE_SIZE << super_page_order[i]; + rem -= PAGE_SIZE << super_page_order[i]; + }; + } + for (i = super_page_nr - 1; i >= 0; i--) { + while ( + (addr & ((PAGE_SIZE << super_page_order[i]) - 1)) == 0UL && + ((end - addr ) >= (PAGE_SIZE << super_page_order[i]))) { + adj_sp_pte(mm, zap, addr, super_page_order[i]); + addr += PAGE_SIZE << super_page_order[i]; + }; + } + return; + } + +void __break_area (struct page *page, unsigned long order) { + int i; + unsigned long size = 1 << order; + + for ( i = 0; i < size; i++ ) { + set_page_count(page + i, 1); + } + return; +} + /* This program is for testing large stride data transfer I put this program under the GPL. Contact:Naohiko Shimizu, School of Engineering, Tokai University. 1117 Kitakaname, Kanagawa 259-12 Japan email:nshimizu@keyaki.cc.u-tokai.ac.jp TEL: +81-463-58-1211(ext.4084) FAX: +81-463-58-8320 <URL:http://shimizu-lab.et.u-tokai.ac.jp/> */ #include <stdio.h> #include <stdlib.h> #include <unistd.h> #include <math.h> #include <string.h> #include <sys/time.h> #include <sys/resource.h> struct rusage rusage; double dtime() { double q; getrusage(RUSAGE_SELF,&rusage); q = (double)(rusage.ru_utime.tv_sec); q = q + (double)(rusage.ru_utime.tv_usec) * 1.0e-06; return q; } double bench1(double *a, double *b, int dim, int iter) { int i,j,kk,idim, jdim; double starttime, endtime; for(i=0; i< dim*dim; i++) a[i] = b[i] = 0.0; starttime = dtime(); for(kk=0; kk<iter; kk++) { idim=0; for(i=0; i<dim; i++) { jdim=0; for(j=0; j< dim; j++) { a[jdim+i] = b[idim+j]; jdim+=dim; } idim+=dim; } } endtime = dtime(); return endtime-starttime; } double bench2(double *a, double *b, int dim, int iter) { int i,j,kk,idim,jdim; double starttime, endtime; for(i=0; i< dim*dim; i++) a[i] = b[i] = 0.0; starttime = dtime(); for(kk=0; kk<iter; kk++) { jdim=0; for(j=0; j< dim; j++) { idim=0; for(i=0; i<dim; i++) { a[jdim+i] = b[idim+j]; idim+=dim; } jdim+=dim; } } endtime = dtime(); return endtime-starttime; } int main() { double *a, *b; int dim,iter; double time1,time2; for(dim=201; dim<3500; dim+=2) { a = (double *)malloc(sizeof(double)*(dim*dim*2)); b = a + dim*dim; iter=3500*3500/(dim*dim); if(a==NULL) { printf("not enough memory\n"); exit(1); } time1=bench1(a,b,dim,iter); time2=bench2(a,b,dim,iter); free(a); printf("%d ", dim); printf(" %5.2f", iter*dim*dim*sizeof(double)*1e-6/time1); printf(" %5.2f\n", iter*dim*dim*sizeof(double)*1e-6/time2); } exit(0); }