Compile as one huge object

Posted Dec 16, 2017 12:59 UTC (Sat) by epa (subscriber, #39769)
Parent article: Shrinking the kernel with link-time garbage collection

I would have expected to skip the linking step altogether by concatenating all the source files into a big lump and then building that with optimization turned on. Some munging might be needed for symbols which are defined in multiple source files, but surely it's manageable.

Compile as one huge object

Posted Dec 16, 2017 16:16 UTC (Sat) by mpr22 (subscriber, #60784) [Link]

Well, it would at least let you know whether your compiler's execution time is polynomial in the size of the translation unit.

Compile as one huge object

Posted Dec 16, 2017 17:40 UTC (Sat) by pbonzini (subscriber, #60935) [Link]

That's exactly what link-time optimization does.

Compile as one huge object

Posted Dec 16, 2017 17:44 UTC (Sat) by gutschke (subscriber, #27910) [Link]

Compilers aren't optimized for this use case, and tend to run out of memory if you try to naively load everything into them at the same time. But even if that was taken care of, you are going to run into a lot of conflicts with incompatible reuse of symbols. You should try this one day on any project that is bigger than just a "toy" example. It usually turns out to be insanely painful.

You also lose out on a lot of the error messages that you would normally *want* to get, when there are layering violations and code reaches into private implementation details of other parts of the system. So, in practice, you'd have to maintain the code so that it can be compiled in the traditional fashion (that would happen during normal development cycles) and so that it can be compiled as one big file. That's very painful for the developers.

As far as I can tell, there are some compilers that internally do something similar to what you are suggesting. And with proper compiler support, it can of course be made to work. But honestly, from the developer's point of view, it ends up looking very similar to what is described in this article. And it would still need the same source-code annotations to help the compiler figure out which references are life and which ones aren't.

Compile as one huge object

Posted Dec 18, 2017 16:06 UTC (Mon) by mwsealey (subscriber, #71282) [Link]

Multi-file compilation is a thing. You can actually use both together - the key aspect if LTO as per the article is putting each function and data item in it's own named section, because the linker can only act on individual sections as an unbreakable unit. Multi-file complication would still allow you to do this, but also provide greater inlining opportunities at the compiler level (for functional equivalence or tail-calling) which the linker has less information about.

It's probably best to leave the MFC part on a per module/subsystem basis (so, anything that gets a built-in.o) and then LTO both the main kernel binary and any modules. The difficulty is going to be making sure that public functions do not
get LTO'd away, while C99 inlining somewhat forces preservation of the 'original' function even if it inlines everything, LTO could strip it out if there's no out-of-line reference to it, and then your modules wouldn't load.

It's a pain in the backside, all told, but the results are always worth it.