Weekly Edition
Return to the Kernel page
| |||||||||||||
Reviving linux-tinyAn announcement of the revival of linux-tiny, a set of patches aimed at reducing the footprint of the kernel, mainly for the embedded world, has led to a number of linux-kernel threads. The conversations range from the proper place for linux-tiny to reside to the removal of the enormous number of printk() strings in the kernel. They provide an interesting glimpse into the kernel development process. The linux-tiny project was started by Matt Mackall in December 2003 with the aim to "collect patches that reduce kernel disk and memory footprint as well as tools for working on small systems." LWN covered the announcement at the time and tried out the patches more than a year ago. Many of the linux-tiny features have found their way into the mainline, but quite a few still remain outside. The Consumer Electronics Linux Forum (CELF) is behind the effort to revive the project, with Tim Bird, architecture group chair, announcing the plan, including a new maintainer, Michael Opdenacker. The first step has been mostly completed, bringing the patches forward from the 2.6.14 kernel to 2.6.22. A status page has been established to track the progress of updating the patches, but it is clear that moving them into the mainline, rather than maintaining them as patches, is a big motivation behind the revival. Andrew Morton immediately volunteered to manage the linux-tiny patches in an answer to the revival message:
Seriously, putting this stuff into some private patch collection should
be a complete last resort - you should only do this with patches which
you (and the rest of us) agree have no hope of ever getting into mainline.
Reactions were quite favorable, with the maintainer, Opdenacker responding:
Andrew, you're completely right... The patches should all aim at being
included into mainline or die.
I'm finishing a sequence of crazy weeks and I will have time to send you patches one by one next week, starting with the easiest ones. The full patchset will live in a separate repository as the individual patches are being worked on for inclusion, but it is clear that no one wants to continuously maintain and out-of-tree patchset for a long time. The cost of ensuring that the patches do not bitrot is large and their inclusion in the mainline will get them in the hands of more developers. From there, more detailed discussion of how to structure the patches - and tiny features in general - ensued. A separate discussion also came about regarding printk() and the large amounts of memory it consumes with all of its static strings. printk() has long been seen as an area that could be improved to reduce the memory footprint of the kernel. All sorts of kernel messages are printed to logfiles or the console via printk(); there are something on the order of 60,000 calls in 2.6. There can be a severity level associated with a specific call, which provides a primitive syslog-style categorization of the messages. Unfortunately, in the mainline, those calls are either present, with all the associated memory for the strings, or completely absent, compiled out via a config option. It is rather difficult to diagnose problems without at least some printk() information, but keeping all of the data in can increase the size of the kernel 5-10%. Rob Landley started things off with a way to make it possible to only compile in messages based on their severity level. An embedded developer could remove KERN_NOTICE, KERN_DEBUG and similar low severity messages while keeping the more critical messages:
[...] the compiler's dead code eliminator zaps the printks you don't
care about so they
don't bloat the kernel image. But this doesn't _completely_ eliminate
printks, so you can still get the panic() calls and such. You tweak precisely
how much bloat you want, using the granularity information that's already
there in the source code.
Landley's suggestion has a drawback in that it would require a flag day for printk() or the creation of a new function that implemented his suggestion with relevant changes trickling into the kernel over time. In the meantime, small-system developers would still be looking for ways to get the messages they want, while removing the others from the code. There was also discussion of using separate calls for each severity level, where pr_info(), or some similar name, would produce messages with that level. The preprocessor could then be used to remove those that a developer is not interested in. The discussion led Vegard Nossum to put together an RFC for a new kernel-message logging API. He starts with requirements that the API be backwards-compatible with the existing printk() usage, with the output format being extensible at either compile or run time. The RFC also tries to handle the case of multiple printk() calls to emit what is essentially a single message, but it seems like an over-engineered solution to what should be a fairly straightforward problem. Another contender, one that is already part of the linux-tiny patchset, is Tim Bird's DoPrintk patch. This allows developers to selectively choose source code files for which printk() will be enabled, removing it from the rest of the code and resulting kernel image. While not allowing fine-grained selection of messages based on severity, it does put more control into the hands of developers. It is too early to say which, if any, printk() changes are coming down the pike. There does seem to be a lot of interest in helping small systems reduce their kernel footprint without sacrificing all diagnostic messages. printk() is claimed to be one of the lowest hanging fruit for significant kernel size reduction, which would seem to make it a likely candidate for change. (Log in to post comments)
Reviving linux-tiny Posted Sep 27, 2007 5:40 UTC (Thu) by felixfix (subscriber, #242) [Link] Why not change all printk to a message number and parameters, and bundle all messages into the source tree only? Waaaay back in mainframe days, you had to look up errors by number for just that reason. I realize it wouldn't be as handy, but for the ones that only developers care about, wouldn't that be good enough? I suppose you could have a build-time option to include the full text, but wouldn't most production system be better off with just a message ID?
I don't think managing message IDs would be tricky. They only have to be unique, not consecutive. Never reuse old obsolete numbers. 32 bits surely ought to be enough, a considerable shrinkage.
Well, it's just an idea. Please don't flame me!
Reviving linux-tiny Posted Sep 27, 2007 8:37 UTC (Thu) by jengelh (subscriber, #33263) [Link] That is horrible. You already end up often enough in Windows with messages like "Error 1026" and dunno wtf it means without doing lengthy searches on MSDN.
Reviving linux-tiny Posted Sep 27, 2007 9:01 UTC (Thu) by filipjoelsson (subscriber, #2622) [Link] So put a translator in userland!
Let "make install" put a printk.map-2.6.24 in /boot for vmlinuz-2.6.24 - and syslog/dmesg could grab the relevant message and make the switch. For embedded systems you could trim the lower levels of interest, zip the file, or whatever. You should get the error code only if the relevant string isn't found - and that should happen only if you have actively chosen to trim that class of printks (or printks alltogether - but in that case, and error code would be an improvent on no-message-at-all, right?).
This leaves the problem of getting the message out in case of a kernel panic - so maybe a subset of strings still need to reside in the kernel. But I don't believe all of it is needed there.
I agree we don't need messages like "Error 1026" and lengthy MSDN searches to uncover that it means "Memory corruption" - but bloating the RAM with error strings isn't the only alternative.
Seriously, I know putting too much stuff in userland is frowned upon - but putting error translation there looks way better than putting filesystems in userland.
Reviving linux-tiny Posted Sep 27, 2007 13:34 UTC (Thu) by nix (subscriber, #2304) [Link] And when userspace is dead, or if the machine dies before the necessary daemon is running, now the poor damn users are just hit with gibberish error codes again... just when natural-language messages might help them climb out of the mire.
(Also, what about the printf()-style format substitutions printk() allows? An error ID scheme provides no space for that at all. There's a reason the old catgets scheme is unused except by masochists.)
Reviving linux-tiny Posted Sep 27, 2007 9:11 UTC (Thu) by james (subscriber, #1325) [Link] On the other hand, truly unique error numbers (especially if they are all printed as LINUX44353, for example) are easy to Google...
Error message numbers... Posted Oct 3, 2007 16:00 UTC (Wed) by hummassa (subscriber, #307) [Link] are the salvation for anyone who has ever used Oracle. If you don't understand the short message, you just google for ORA.01234 or whatnot and voila -- tonnes of info about what, where and how the error often occurs.
Error message numbers... Posted Oct 3, 2007 16:07 UTC (Wed) by jengelh (subscriber, #33263) [Link] Well too bad if your only machine's network card driver quit its job.
Error message numbers... Posted Oct 3, 2007 16:25 UTC (Wed) by felixfix (subscriber, #242) [Link] Also too bad if you can't read the comment that suggests it be a compile time option, so only those who want to save the memory need enable it.
Error message numbers... Posted Oct 3, 2007 19:08 UTC (Wed) by nix (subscriber, #2304) [Link] Speaking as someone who uses Oracle every day, they're bloody awful. Ohlook, I got an ORA-xxyy complaining about some column in a big query. Which column? Oh, look, even though the RDBMS knows it refuses to tell me because there's no way to fit that into its precious bloody message number.
And so on.
I'd say that 25% of my error-message-revealed bug-fixing time is spent
[ot] sqlplus or oracle developer Posted Oct 22, 2007 23:32 UTC (Mon) by hummassa (subscriber, #307) [Link] use one of sqlplus or oracle developer to repeat your query, they usually mark the right spot where the error ocurred.
[ot] sqlplus or oracle developer Posted Oct 23, 2007 13:09 UTC (Tue) by nix (subscriber, #2304) [Link] Very amusing, well done. (they always mark *a* spot, often an entirely different one. I don't castigate them for this: parser error recovery is notoriously difficult. But the numbered-error-messages thing just makes everything so much harder than it should be...)
So change your config Posted Sep 27, 2007 14:32 UTC (Thu) by felixfix (subscriber, #242) [Link] If it's a config item, you can choose the bulky kernel with complete messages or the slim kernel with just numbers.
Reviving linux-tiny Posted Sep 28, 2007 22:02 UTC (Fri) by nlucas (subscriber, #33793) [Link] So, because Windows do it (even if it is in no way any innovation at all), Linux can't.
I'm still waiting for someone to come up with a better idea.
A message in a foreign language you don't know (remember the majority of the world population doesn't speak English) doesn't seem to me the brightest idea around.
Reviving linux-tiny Posted Sep 27, 2007 20:33 UTC (Thu) by moxfyre (subscriber, #13847) [Link] > Why not change all printk to a message number and parameters, and bundle all messages into the source tree only? Waaaay back in mainframe days, you had to look up errors by number for just that reason. I realize it wouldn't be as handy, but for the ones that only developers care about, wouldn't that be good enough? I suppose you could have a build-time option to include the full text, but wouldn't most production system be better off with just a message ID?I think that's a pretty good idea! For embedded systems, the console is usually a serial port connected to a desktop computer. That computer could run a translator that would simply convert the numeric codes to strings, so the user would see the same thing. For example, the embedded system might send out a kernel message like:
0xDEADBEEF,/dev/hda,123... and the translator would convert that to:
<CRITICAL>sata-driver: CRC error on /dev/hda in block 123 Basically, you'd be compressing the kernel by converting human-readable strings to 32-bit numbers. The numbers wouldn't even have to be globally unique! Just unique to that particular kernel build! When you built a kernel for such a system, in addition to a bzImage you would get a Strings.db file. Put the bzImage into the device firmware, and give Strings.db to your serial console translator... voila, ready to debug. This could be a really seamless solution.
Reviving linux-tiny Posted Sep 27, 2007 22:25 UTC (Thu) by felixfix (subscriber, #242) [Link] I hadn't thought of only needing to be unique per build. Interesting idea. You'd lose the ability to google for a universally unique number, but maybe that doesn't matter so much. People who want to read them on a production server or desktop with plenty of RAM would keep the messages themselves in the kernel. It's really only the firmware and other small-footprint people who need to lose the message text.
Reviving linux-tiny Posted Sep 28, 2007 1:09 UTC (Fri) by moxfyre (subscriber, #13847) [Link] > I hadn't thought of only needing to be unique per build. Interesting idea. You'd lose the ability to google for a universally unique number, but maybe that doesn't matter so much. People who want to read them on a production server or desktop with plenty of RAM would keep the messages themselves in the kernel. It's really only the firmware and other small-footprint people who need to lose the message text.Yeah, I think it would be a very useful and not a very complex modification. Store the table of strings on the "big" computer, and just the indices on the "small/embedded" system. As far as I can tell, it would require a kernel build option to modify printk, and then something to strip the strings out of the object files. Here's a rough sketch of what printk() would look like in this setup: printk() for an embedded system:
#include <stdarg.h>
#define unique_id(string) \
/* some hash function of the string */
#define printk(format,...) \
extern const char *__printk_string_ ## unique_id(format) = format; \
printk_embedded(unique_id(format), __VA_ARGS__, NULL)
void
printk_embedded(int id, ...)
{
va_list args;
unsigned int arg;
va_start (args, id)
for (arg=id; arg != NULL; arg=va_arg(args, unsigned int))
kprint_integer_followed_by_comma(arg)
kprint_char('\n')
va_end (args)
}
This should work for all printk() argument fields except for strings (since they're not passed by value). And then there would have to be a utility to go through the ELF image and strip out all the __printk_string_XXXXXXXX items... not too difficult I don't think.There are a few details to work out (how to make string arguments work!!) but otherwise that's about it. Any opinions?
Reviving linux-tiny Posted Sep 28, 2007 11:54 UTC (Fri) by etienne_lorrain@yahoo.fr (guest, #38022) [Link] If you go down that route, your "message number" could be the address (in hexadecimal) of the string and you instruct the linker/objcopy to remove the ".string" section - plus or minus some special cases.
Reviving linux-tiny Posted Sep 27, 2007 14:17 UTC (Thu) by kjp (guest, #39639) [Link] I dont understand why, for the majority of printk's where the format string is a literal, the compiler couldn't do a compile time check for the <Severity> code and decide whether or not to generate code for it.
I do similar things and it has worked for ages with gcc.
|
|||||||||||||