LWN.net Logo

Kernel configuration for distributions

By Jake Edge
July 18, 2012

Configuring a kernel was once a fairly straightforward process, only requiring knowledge of what hardware needs to be supported. Over time, things have gotten more complex in general, but distributions have added their own sets of dependencies on specific kernel features—dependencies that can be difficult for regular users to figure out. That led Linus Torvalds to put out an RFC proposal to add distribution-specific kernel configuration options.

The problem stems from distributions' user space needing certain configuration options enabled in order to function correctly. Things like tmpfs and devtmpfs support, control groups, security options (e.g. SELinux, AppArmor), and even raw netfilter table support were listed by Torvalds as "support infrastructure" options that are required by various distributions. But, in addition to being hard to figure out, those options tend to change over time, so a configuration that worked for Fedora N may not work for Fedora N+1. The resulting problems can be hard to find as Torvalds pointed out: "There's been several times when I started with my old minimal config, and the resulting kernel would boot, but something wouldn't quite work right, and it can be very subtle indeed."

So, he suggested adding distribution-specific Kconfig files:

The point I'm slowly getting to is that I would actually love to have *distro* Kconfig-files, where the distribution would be able to say "These are the minimums I *require* to work". So we'd have a "Distro" submenu, where you could pick the distro(s) you use, and then pick which release [...] it would make it much easier for a normal user (and quite frankly, I want to put myself in that group too) to make a kernel config that "just works".

There are others ways to get there, of course, but they leave something to be desired, Torvalds said. Copying the distribution config file would work, but would bring along a bunch of extra options that aren't really necessary for the proper operation of the distribution. Using make localmodconfig (which selects all of options from the running kernel) suffers from much the same problem, he said. The ultimate goal is to have more people able to build kernels:

I really think that "How do I generate a kernel config file" is one of those things that keeps normal people from compiling their own kernel. And we *want* people to compile their own kernel so that they can help with things like bisecting etc. The more, the merrier.

In general, the idea was met with approval on linux-kernel. There were concerns about how the distribution-specific files would be maintained, and that sometimes they might get out of sync with the distribution's requirements. Dave Jones noted that he sometimes gets blindsided by Fedora kernel requirements (and he is the Fedora kernel maintainer).

Torvalds is pretty explicitly not looking for a perfect solution, however, just one that is better: "even a 'educated guess' config file is better than what we have now". In that message, he outlines two requirements that he sees for the feature. The first is that each configuration option that is selected for a particular distribution version come with a comment explaining why it is needed. The second is that the configuration options be the minimum required to make the system function properly—not that it "grow to contain all the options just because somebody decided to just add random things until things worked".

Commenting the options may be difficult even for those who work directly on distribution kernels though. Ben Hutchings (who maintains the Debian kernel) pointed out that he sometimes does not know the reason that a particular option is needed, particularly at some later point: "just because an option was requested and enabled to support some bit of userland, doesn't mean I know what's using or depending on it now".

Other kinds of configuration options are possible, of course. In his original message, Torvalds mentioned configurations for "common platforms", such as a "modern PC laptop" that would choose options typically required for those (USB storage, FAT/VFAT, power management, etc.). He specifically said that platform configuration should be considered an entirely separate feature from the distribution idea.

KVM (and other virtualization) users were also interested in creating an option that would select all of the drivers and other options needed for those kernels. Currently "you need to hunt through 30+ different menus in order to find what you need to run in a basic KVM virtual machine", as Trond Myklebust put it. There was a lot of discussion (and much agreement) on the need for better configuration options for virtualization, but some of that got rather far afield from Torvalds's original proposal.

Unsurprisingly, kernel developers started thinking about how they could use the feature. There was concern that choosing a particular distribution and its dependencies would make it harder for kernel developers to further customize the configuration. David Lang had some specific complaints about the approach suggested in the RFC, noting that it would be hard to choose a Fedora kernel without getting SELinux for example. He also was concerned about the amount of churn these defconfig-like files might cause (referencing the movement to reduce the number of defconfigs in the ARM tree). But Torvalds makes it clear that Lang and other kernel hackers are not the target of the feature:

The thing I'm asking for is for normal people. Make it easy for people who DO NOT CARE about the config file to just build a kernel for their machine.

Don't complicate the issue by bringing up some totally unrelated question. Don't derail a useful feature for the 99% because you're not in it.

There may be ways to satisfy both camps—Lang seemed to think so anyway—but until someone actually posts some code, it's hard to say. While there was general agreement that the feature would be useful, so far no one has stepped up to do the work. Whether Torvalds plans to do that or was just floating a trial balloon and hoping someone else would run with it is unclear, but it does seem like a feature worth having.


(Log in to post comments)

Kernel configuration for distributions

Posted Jul 19, 2012 2:28 UTC (Thu) by pr1268 (subscriber, #24648) [Link]

I thought the various distros did this already. Not just in the config file, but mostly through the addition of downstream code to make the kernel work for that particular distro (like the various security modules as mentioned in the article).

At least that's been my experience with Fedora. The one time I tried compiling a vanilla kernel on Fedora (way back in FC2 or 3 or so), it was a configuration nightmare and a boot time disaster. Same with SuSE 9.{0,1}.

Kernel configuration for distributions

Posted Jul 19, 2012 3:43 UTC (Thu) by awesomeman (guest, #85116) [Link]

The distributions publish their own configs, and certainly this is the sanest place to start, although even paring down the hardware can take a bit of time. (now building video4linux will save you alot of compile time) And if you pare down the wrong thing, your resulting kernel won't boot. This last part is what the proposal is meant to address, to prevent from removing features that are 100% required by the user-space you are going to boot. The kernel already has this concept for some drivers and features (CONFIG_EXPERT).

The biggest problem I see with this, is that, 1. for distributions like Debian, this type of definition doesn't really exist, as core things like init system are swappable. and 2. often the software that requires these things doesn't really know it requires these things.

I've submitted patches making systemd a little better when it comes to kernels with slightly less that the Cadillac model of configs. (these were in non-essential helper programs, the system still worked fine)

Kernel configuration for distributions

Posted Jul 19, 2012 6:35 UTC (Thu) by drjones (subscriber, #60990) [Link]

CONFIG_EXPERT is a great idea, but it doesn't work as advertised. The idea, according to its help text, is to hide config options from the config menu that most people would never want to modify (options for "non-standard" kernels). However, if you flip this option on, you not only expose those options for modification, but several defaults are *changed*, giving your config an embedded personality. If you're not building an embedded kernel, then EXPERT isn't useful. Getting EXPERT to work as advertised, or implementing some other scheme that hides/exposes standard options, makes sense in order to reduce config menu complexity. An overhaul of config option default value management (maybe with some hierarchical defconfig type scheme) might also be nice.

Kernel configuration for distributions

Posted Jul 19, 2012 15:15 UTC (Thu) by WolfWings (guest, #56790) [Link]

The problem is actually that 'EXPERT=n' sets several options certain ways, but their default values are not those settings.

So technically, turning EXPERT=y is returning things to their true defaults, that 'embedded personality' as you put it.

I.E. I believe this would count as a kernel bug of sorts: The defaults for some values do not match their 'EXPERT=n' case, causing toggling that to change other unrelated features when nothing else has been changed.

Kernel configuration for distributions

Posted Jul 19, 2012 16:00 UTC (Thu) by drjones (subscriber, #60990) [Link]

I can see how the use of 'default !EXPERT' in Kconfig files can be interpreted this way, and it may be the case for some of 'default !EXPERT' options. However, if you take the options in usr/Kconfig, for example, and look at commits

73d8a12f0529 bzip2/lzma: move CONFIG_RD_* options under CONFIG_EMBEDDED
6a108a14fa35 kconfig: rename CONFIG_EMBEDDED to CONFIG_EXPERT

you'll see from the commit message of 73d8a12f0529 that the toggling default was selected with the init section size of the kernel for embedded systems in mind, and then later 6a108a14fa35 simply renamed EMBEDDED. IMHO, the general symbol name EXPERT should do nothing except expose more options to the menu. Toggling defaults to be more embedded friendly was better left to the symbol EMBEDDED.

Kernel configuration for distributions

Posted Jul 19, 2012 5:05 UTC (Thu) by nwatkins (guest, #61119) [Link]

I /completely/ support this. Too many times have I been constructing a minimal configuration by removing drivers for hardware that I don't have, removing 'tmpfs' because I expected the 'do you want ext4 partition?' during install to be enough, etc... only to find that the excruciating process of paring down the config was wasted because I had no idea where i went wrong until it was too late.

Kernel configuration for distributions

Posted Jul 28, 2012 3:54 UTC (Sat) by Spudd86 (guest, #51683) [Link]

Yea... tmpfs is not really something you can remove these days and IIRC it actually says so in the description (ie: mentions you need it for certain things you almost certainly have to work)

Kernel configuration for distributions

Posted Jul 20, 2012 0:37 UTC (Fri) by thedevil (subscriber, #32913) [Link]

So how is the 1% supposed to do it?

Will it be possible to select "None of the above" ?

Kernel configuration for distributions

Posted Jul 20, 2012 1:16 UTC (Fri) by jake (editor, #205) [Link]

> Will it be possible to select "None of the above" ?

Linus's proposal is that there be a choice for "Distribution configuration" or something like that, if that doesn't get chosen, nothing changes from today's situation. Or one could choose F17, say, get its defaults, then go in and remove that option from .config and reconfigure however you like.

jake

Kernel configuration for distributions

Posted Jul 20, 2012 4:46 UTC (Fri) by dirtyepic (subscriber, #30178) [Link]

While I support this completely, I also think a lot of pain in configuring a kernel would be relieved by providing more user-centric information in the help text. Most help documents what an option does (more or less), but rarely does it give any indication of why you might want it to do that. I have a text file that I've slowly filled over the last 6 years with pointers to all the random emails, references, blog posts, LWN articles, and research I've had to do to just figure out if I actually need to enable a particular option or not.

I know the kernel is very complex and needs to serve the needs of all types of users and many options are going to require you to do a little research to be able to make an educated choice, but for the basic stuff it'd be nice to see a "Disabling this will make consolekit/powertop/systemd puke on your shoes." or "Here's how to determine if your system supports this kind of device." once in a while.

Kernel configuration for distributions

Posted Jul 20, 2012 16:04 UTC (Fri) by apoelstra (subscriber, #75205) [Link]

>I have a text file that I've slowly filled over the last 6 years with pointers to all the random emails, references, blog posts, LWN articles, and research I've had to do to just figure out if I actually need to enable a particular option or not.

If this text file is in a readable (and polite) state, it'd be great if you could post it somewhere so others can learn from it. I've tried compiling a kernel 4 or 5 times over the last decade, and never got one that booted.

Kernel configuration for distributions

Posted Jul 21, 2012 12:37 UTC (Sat) by foom (subscriber, #14868) [Link]

Indeed. The only way I get working custom kernels is by taking my distro kernel's config and modifying it slightly.

Kernel configuration for distributions

Posted Jul 22, 2012 20:53 UTC (Sun) by Julie (guest, #66693) [Link]

>While I support this completely, I also think a lot of pain in configuring a kernel would be relieved by providing more user-centric information in the help text.

As someone who spends a lot of free time testing mainline -rc and -next kernels, I totally and absolutely agree - and maybe for kernel maintainers of specific kernel features and modules to check from time to time that their help text is up-to-date - obsolete documentation has caught me out more than once, and not only wasn't I knowledgeable enough about those features/modules to provide an update patch, I didn't even know where to start looking :-/

Tips like make localmodconfig are really useful, but only go so far, and the trial-and-error approach of squeezing a kernel down gets old *really* quickly. But it has to be done; if simply building a new kernel takes a lot longer with a bloated distro config, imagine how much worse this is when you encounter a bug and have to do a bisect...and all those extra new and sometimes arcane features disappearing and popping up again as you whang back and forth from one kernel version to another...

The last time I had to pare down a distro kernel for a new test setup it took me more than *two weeks* to come up with something that was minimal enough to be efficient and still worked, and even then I had delayed starting it until I had a long weekend because I knew it was going to be such a PITA.

Some of this can be put down to naivety and lack of time, but I'll bet that's the case with a lot of testers - we're not developers and are doing this in our spare hours, after all.

>I have a text file that I've slowly filled...

Like the others say, it would be great if this could somehow be available to the rest of us...possibly at kernelnewbies?

Kernel configuration for distributions

Posted Jul 20, 2012 5:09 UTC (Fri) by dlang (✭ supporter ✭, #313) [Link]

actually Alan Cox posted the code that would need to be included in the kernel tree (see http://lists.opensuse.org/opensuse-kernel/2012-07/msg0008... for Alan explaining what I was fumbling for in a very concise way)

> However providing you separate the initial profile from the later tools
> it simply becomes
>
> make distroconfig
> [cp /etc/system-kconfig(.$ARCH?) .config
> make oldconfig]
>
> make menuconfig (if you want to customise)

the effort will be around creating the distro specific options.

If the solution ends up being miniconfig like, then the distros produce snippets of .config that contain their requirements

If the solution is additional selection items in the kconfig system, then the distros will need to produce the appropriate kconfig code to select the right things.

In either case, the hardest work is going to be figuring out what the correct options should be.

the biggest reason I would prefer to see the miniconfig approach is that it allows each distro to provide the appropriate file as part of the distro update process.

This avoids the risk of a huge amount of distro specific code in the kernel.org tree. And the political questions like how big does a distro need to be to get it's config included in kernel.org, how long after release does it remain there. There are hundreds of distros out there and at the other extreme a single Debian release can produce 30-40 different kernel configs (although defining 'minimum' tightly enough can reduce this).

Linus complained a few years ago about the horrible defconfig churn, putting the per distro, per release info in the kernel config seems like it is begging for the same sorts of problem.

There will also be a gap between the time the distro releases a version and when the minimum config for that version can get into the kernel.org tree. You will have a ~3-5 month lag between the time the distro defines a minimum and when a kernel release comes out that includes it. As a result, the first kernel.org kernel that will have the 'proper' minimum config for a distro will be at least two revisions newer than the version the distro ships.

However, I'm not a kernel developer, I'm 'just' a sysadmin who's been using linux since the 0.99 days My contribution consists of testing and feedback, so while I can warn of problems that I see with an approach, and try to convince others. Ultimately this is going to be up to the distro maintainers to decide which approach they are going to be willing to implement.

Kernel configuration for distributions

Posted Jul 22, 2012 9:33 UTC (Sun) by DavidS (subscriber, #84675) [Link]

> You will have a ~3-5 month lag between the time the distro defines
> a minimum and when a kernel release comes out that includes it.

Except when those updates qualify as stable updates. Which I believe they should.

Or when everyone get's their act together and when a new kernel releases it already has the "right" set of definitions.

Kernel configuration for distributions

Posted Jul 22, 2012 9:53 UTC (Sun) by dlang (✭ supporter ✭, #313) [Link]

> Except when those updates qualify as stable updates. Which I believe they should.

-stable could be redefined to allow this, but since this is adding functionality, not fixing a bug, it wouldn't fit the current definition.

> Or when everyone get's their act together and when a new kernel releases it already has the "right" set of definitions.

I don't see how that is possible.

month M kernel.org releases 3.4, starts development on 3.5

Month M+1 distro A decides their next release (version 10, to be released in 3 months) will be based on 3.4 starts defining minimum requirements for the next release

Month ~M+3 kernel.org releases 3.5 starts work on 3.6

Month M+4 distro A finalizes the version 10 release (based on 3.4) and now has the final definition for the minimum requirements. Distro A submits these upstream, Linus allows the patches past the merge window.

Month ~M+6 kernel.org releases 3.6 (with distro A version 10 minimum config embedded) and starts working on 3.7

multiply this out by hundreds of different distros.....

Remember that the kernel.org schedule is not fixed, it's working out to about 2.5-3 months (two week merge window plus 6-9 -rc releases before the final release) while the distros are on a fairly strict 6 month schedule. the distros aren't going to be trying to release the kernel that kernel.org is currently working on (if we are lucky, they are working on the release only one back, but it may very well be older than that)

Kernel configuration for distributions

Posted Jul 22, 2012 9:56 UTC (Sun) by dlang (✭ supporter ✭, #313) [Link]

If you are going to allow such patches in to -stable releases, and backport them to older kernels, how far back do you need to go? back to the kernel the distro uses for it's release? or back before that the the kernel the distro used for the prior release (so that you can try to find problems that were introduced by the new kernel in the release, think Intel video drivers a few years ago as an example)

If it's maintained outside of the kernel.org tree, then the distro can ship the appropriate config as it goes along, and it won't matter (much) which kernel you try to compile, the make oldconfig process will work with older and newer kernels (as long as they have the feature, and the feature has not grown a dependency one some other feature, it's not perfect, but it's pretty good)

Kernel configuration for distributions

Posted Jul 23, 2012 19:21 UTC (Mon) by BenHutchings (subscriber, #37955) [Link]

Why would you downgrade the kernel on a new distribution?

Kernel configuration for distributions

Posted Jul 23, 2012 19:50 UTC (Mon) by apoelstra (subscriber, #75205) [Link]

>Why would you downgrade the kernel on a new distribution?

I think the disconnect occurs when a distribution takes a (current) kernel, does several months on QA, then releases it -- and in the meantime, a new mainline kernel has come out.

Kernel configuration for distributions

Posted Jul 23, 2012 20:39 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]

if the kernel supplied by the new distro doesn't work on your hardware.

for example, there was a big problem a few years ago with Intel drivers in a Ubuntu release not working for many people. That is a perfect example of where you would want to move to an older kernel to keep working.

Kernel configuration for distributions

Posted Jul 26, 2012 16:52 UTC (Thu) by landley (guest, #6789) [Link]

I'm still happily using my old miniconfig stuff from 2005 for this sort of thing:

http://lwn.net/Articles/161086/
http://landley.net/aboriginal/FAQ.html#dev_miniconfig

For me, it makes config files much smaller, easier to read, and reasonably portable between distros. I've digested ubuntu's .config file into a miniconfig to see what actual things it's enabling without having to wade through thousands of lines to do it.

Unfortunately, I haven't got Philip Lougher's patience to try to get it upstream. It works for me, I submitted it more than once, life goes on...

Kernel configuration for distributions

Posted Aug 2, 2012 13:13 UTC (Thu) by Fredo (guest, #37386) [Link]

Relating to compile the kernel for virtualization, I started a page on the KVM wiki:

http://www.linux-kvm.org/page/Tuning_Kernel

Copyright © 2012, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds