|
|
Subscribe / Log in / New account

Building header files into the kernel

Building header files into the kernel

Posted Mar 22, 2019 2:07 UTC (Fri) by neilbrown (subscriber, #359)
Parent article: Building header files into the kernel

Why not move it to swapable memory?
Store the compressed image in the __init section (which is discarded somewhere in the boot sequence) and have code to copy it into a tmpfs filesystem (which can be paged out).
Maybe put /proc/config.gz there too.
(and maybe a "cowsay" binary too, just it case it is ever needed).


to post comments

Building header files into the kernel

Posted Mar 22, 2019 10:02 UTC (Fri) by mjthayer (guest, #39183) [Link] (1 responses)

> Why not move it to swapable memory?
> Store the compressed image in the __init section (which is discarded somewhere in the boot sequence) and have code to copy it into a tmpfs filesystem (which can be paged out).
> Maybe put /proc/config.gz there too.

Glad to see someone else had the same idea. Might it make sense to reduce the in-kernel policy a bit by letting user-space handle the tmpfs? One possible (!) implementation would be to have a device which a) lets you read out the archive and b) lets you free the in-kernel memory holding the archive after you have done with it. (Maybe that could mean putting it into __init first and copying it from there to free-able kernel memory.)

Then again, perhaps the same thing could be achieved without it ever being in kernel memory. Tagged onto the end of the kernel binary? Something similar to initramfs? I was going to write "put into initramfs", which would avoid kernel changes altogether, but the thought of bloating that even more is not appealing.

Building header files into the kernel

Posted Mar 23, 2019 22:12 UTC (Sat) by Mattimo (subscriber, #129903) [Link]

Actually you could use a compressed cpio archive of the headers and just present it as a subtree in procfs because initramfs already has all the primitives available for that. Maybe that would even allow something like /proc/include that just contains the headers in plain sight.

Building header files into the kernel

Posted Mar 22, 2019 12:33 UTC (Fri) by excors (subscriber, #95769) [Link] (2 responses)

> Why not move it to swapable memory?

I think most Android devices don't have swap. They prefer to just kill background apps whenever RAM gets low.

Building header files into the kernel

Posted Mar 22, 2019 15:20 UTC (Fri) by dezgeg (subscriber, #92243) [Link] (1 responses)

Plenty of devices use zram, I think.

Building header files into the kernel

Posted Mar 22, 2019 17:45 UTC (Fri) by excors (subscriber, #95769) [Link]

Ah, true, I suppose that technically is swap. But moving a gz file from RAM into tmpfs that gets swapped to zram that is stored in RAM, doesn't sound especially useful.

Building header files into the kernel

Posted Mar 24, 2019 1:29 UTC (Sun) by _joel_ (subscriber, #112763) [Link] (5 responses)

Neil, thanks for the suggestion. How do you create an invisible tmpfs mount in the kernel and later mount it in userspace though? I am not aware of any such thing. Any existing code or examples would be appreciated.

Also note that on Android, we don't use disk-based swap. We have a memory based compressed swap called ZRAM, but the archive is already compressed so the suggested idea would provide no benefit (to us).

One thing I have thought of doing in the future is to make the /proc entry of the archive writeable and write an empty string into it thus freeing the archive's allocated memory and requiring a reboot. However at the moment, I am considering only building this as a module for production Android, and as a built-in when debugging. After the patches can make it, and if others want to free that memory, we can cross that bridge there IMO - such as by writing an empty string into the proc entry.

Building header files into the kernel

Posted Mar 24, 2019 1:38 UTC (Sun) by _joel_ (subscriber, #112763) [Link]

About requiring a reboot, I meant "requiring a reboot if headers are needed again"

Building header files into the kernel

Posted Mar 24, 2019 4:28 UTC (Sun) by neilbrown (subscriber, #359) [Link]

You can create an invisible tmpfs mount with kern_mount(). mm/shmem.c does this.
I don't think there is an existing way to mount this into a mount namespace, but I suspect you could add a mount option to tmpfs to say "use pre-exist superblock named foo".

If you don't have real swap, this doesn't help you, but it might be a useful answer to people who complain about a waste of kernel memory, or who emphasize that it is non-swapable memory (the article mentions both).

Building header files into the kernel

Posted Mar 24, 2019 5:39 UTC (Sun) by jsmith45 (guest, #125263) [Link] (1 responses)

Joel, why even bother remounting? You can just directly use the pre-existing internal kernel mount as if it was a special form of swappable kernel memory. CONFIG_BIG_KEYS actually does that, and the the bpfilter user mode helper module copies data that is baked into its module (as discardable init data) into the main internal tmpfs mount, so there is precedent for most of this.

I'll be showing code as if making the file swappable was unconditional, but it should be very obvious how to add a config option that hybridizes what patch V5 has, and the below.

One word of warning: This basically requires CONFIG_TMPFS. It will also work if !CONFIG_SHMEM, because then it uses tiny-shmem (which is ramfs). However using CONFIG_SHMEM without CONFIG_TMPFS yields a tmpfs that is excessively limited.

Now lets create the file in tmpfs, and copy the data into it:


static struct path *mem_path;

static int __init ikheaders_init(void)
{
	struct proc_dir_entry *entry;
        size_t kernel_headers_data_size = &kernel_headers_data_end - &kernel_headers_data;
        int err;

        ssize_t written;
	loff_t pos = 0;

        /* copy data to a new tmpfs file, so it can be swapped out */
        mem_file = shmem_kernel_file_setup("", kernel_headers_data_size, 0);
        if (IS_ERR(mem_file)) {
		err = PTR_ERR(mem_file);
		goto err_no_fput;
	}

        written = kernel_write(mem_file, kernel_headers_data, kernel_headers_data_size, &pos);
        if (written != kernel_headers_data_size) {
		err = written;
		if (err >= 0)
			err = -ENOMEM;
		goto error;
	}

	/* create the current headers file */
	entry = proc_create("kheaders.tar.xz", S_IRUGO, NULL,
			    &ikheaders_file_ops);
	if (!entry) {
		err = -ENOMEM;
		goto error;
	}
	
	proc_set_size(entry,
		      kernel_headers_data_size);
	return 0;
error:
	fput(mem_file);
error_no_fput:
	return ret;
}

static void __exit ikheaders_cleanup(void)
{
	remove_proc_entry("kheaders.tar.xz", NULL);
	path_put(mem_path);
}

OK. Next up, we need to change the section the baked-in data is stored in to be .init.data so it gets discarded after init.


asm (
"	.pushsection .init.data, \"a\"	\n"
/*...*/
);

Almost done. All that is left is to make reads of the proc file return data from the in memory file. There are actually several ways to approach this. We could use a magic symlink to the internal file, similar to how the proc file descriptor symlinks work. But to make this act as much like a normal file as possible, the following might be easiest. It is definitely a bit odd to have the read delegate back to vfs_read, but as far as I can tell it should work. I'd bet one of VFS guys can come up with some improvements to this approach.


static int ikheaders_open_current(struct inode *inode, struct file *file)
{
	struct file *mem_file
	file = dentry_open(path, O_RDONLY, current_cred());

	if (IS_ERR(file)) 
		return PTR_ERR(file);

	file->private_data = mem_file;
	return 0;
}
static ssize_t
ikheaders_read_current(struct file *file, char __user *buf,
		      size_t len, loff_t *offset)
{
        struct file *mem_file = file->private_data;
	return vfs_read(mem_file, buf, len, offset);
}
static int ikheaders_release_current(struct inode *inode, struct file *file)
{
        struct file *mem_file = file->private_data;
	fput(mem_file);
	return 0;
}

static const struct file_operations ikheaders_file_ops = {
        .open = ikheaders_open_current,
        .release = ikheaders_release_current,
	.read = ikheaders_read_current,
	.llseek = default_llseek,
};

Please note that all this code is untested, and is intended to a show the basic approach, although it probably works as-is, modulo any typos (since most of it was copy-pasted from existing parts of the kernel or your patch).

Hope you find this helpful, or at least interesting

Building header files into the kernel

Posted Mar 26, 2019 23:53 UTC (Tue) by quotemstr (subscriber, #45331) [Link]

> One word of warning: This basically requires CONFIG_TMPFS. It will also work if !CONFIG_SHMEM, because then it uses tiny-shmem (which is ramfs). However using CONFIG_SHMEM without CONFIG_TMPFS yields a tmpfs that is excessively limited.

It occurs to me that we shouldn't even have these options. The kernel would be simpler to reason about if basic, fundamental, and cheap things like tmpfs and shmem (and procfs!) were hardwired to =y. I've never understood the rationale for extreme configurability. Fundamental things should always be available.

Building header files into the kernel

Posted Mar 26, 2019 7:34 UTC (Tue) by minchan (subscriber, #61813) [Link]

>> Also note that on Android, we don't use disk-based swap. We have a memory based compressed swap called ZRAM, but the archive is already compressed so the suggested idea would provide no benefit (to us).

zRAM supports idle and/or incompressible page writeback once admin configures backing store.
If the data is used rarely or incompressible, it will be written back to the storage from the memory of zram.


Copyright © 2025, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds