Linus on what remains to be merged [LWN.net]

Linus on what remains to be merged

Posted Oct 31, 2002 17:16 UTC (Thu) by mwg (subscriber, #5278) [Link]

Well, there are additional criteria - and one overriding aspect is to
avoid kernel bloat: whatever you can do in user space should not be
done by the kernel (increasing it's size - that's the windows way).

Thus, you feature must be needed by many people and it must be very
cumbersome to do said feature with user-space tools.

Linus on what remains to be merged

Posted Oct 31, 2002 17:18 UTC (Thu) by proski (subscriber, #104) [Link]

User base is essential for a feature to exist. Maintainers come and go. There is no way Linus can ensure that the feature will remain maintained except when it going to have many users. Impact on the other parts of the kernel is also not trivial to estimate. Added complexity can discourage new developers from even touching that code. Time of security specialists is too valuable to ask them to audit the code that just a few people are going to use.

I cannot speak for Linus, but I understand his attitide.

Linus on what remains to be merged

Posted Oct 31, 2002 17:22 UTC (Thu) by tjc (guest, #137) [Link]

If a patch adds new functionality to the kernel, is actively being maintained, has proven stable and more or less bug-free and does not hurt / impact other parts of the kernel, why not merge it?

For one thing, code can never be proven stable or bug free. The closest one can come is to demonstrate that it works as intended under certain circumstances while running on certain hardware. Besides that, every feature added increases the amount of integration testing to be done at an above linear rate. A lot of it never gets done.

The best way to improve nearly any large and/or complex piece of software is to remove functionality, or at the very least resist the temptation to add stuff that few people will use. Unfortunately it's hard to find free software projects that accept patches of this sort. :^)

Linus on what remains to be merged

Posted Oct 31, 2002 18:51 UTC (Thu) by proski (subscriber, #104) [Link]

Unfortunately it's hard to find free software projects that accept patches of this sort. :^)

Shameless plug: GNU Midnight Commander

Actually, patches that remove something are very rarely suggested by developers other than the core maintainers. On the other hand, there is always a vocal minority who demand to leave their feature alone, even if it has been broken for years, and nodody has ever reported the bug, let alone fixed it.

That's another reason to be conservative when adding new features.

Linus on what remains to be merged

Posted Oct 31, 2002 21:19 UTC (Thu) by Schneelocke (guest, #3535) [Link]

For one thing, code can never be proven stable or bug free. The closest one can come is to demonstrate that it works as intended under certain circumstances while running on certain hardware. Besides that, every feature added increases the amount of integration testing to be done at an above linear rate. A lot of it never gets done.

True, and maybe I shouldn've said "proven" in the first place - that was a poor choice of words, since I did not mean to imply that there was an actual proof, but rather that the evidence collected through extensive test runs suggested that the code was mature.

The best way to improve nearly any large and/or complex piece of software is to remove functionality, or at the very least resist the temptation to add stuff that few people will use. Unfortunately it's hard to find free software projects that accept patches of this sort. :^)

I gotta admit I don't really agree with that - although I understand your point and do think that you have a valid point, I personally still think that creeping featuritis is a Good Thing(tm). :)

I'd also like to take this opportunity to reply to some of the other objections raised. First of all, mwg talked about kernel bloat; I don't think that this is really an issue with patches that do not touch the rest of the kernel. Simply don't compile in what you don't need, and your kernel should be fine. :) About the only bloat people could complain about would be that of the kernel tarballs, and quite frankly, I don't think that this should *that* important for the decision what is merged into the kernel and what isn't.

Second, mwg also said that user-space tools should be preferred over in-kernel solutions whenever possible. I second that, but there are some things that cannot be done in user space, or at least not without considerable hassle. An example might be the crash dumps that Rusty mentioned in his list; I maybe wrong since I don't really know about the internal workings of either the kernel as such in general or the LKCD patch in particular, but I'd think that it would be very hard to duplicate this functionality using only user-space tools.

Another example for this might be new filesystems - there is a reason why things like XFS, JFS and so on have gone into the kernel and why Reiser4 will hopefully be merged when it's ready, too. While probably almost all kernel work could be done in userspace, sometimes it's just not the right thing to do.

That side, proski, in his comment, stated that a feature without a large userbase might become unmaintained at some time; this is true, of course, but I don't see why a non-intrusive patch could not, at that time, simply be pulled from the kernel again. Besides, a feature might only attract a larger userbase when it actually *is* available to users, including Joe Average who does not want to track down individual patches himself. :)

With regard to impact on other kernel parts - it's true that it may be difficult to estimate just how much problems the inclusion of a particular patch will cause, but I would expect that a new feature, the implementation of which touches few if any kernel files outside its scope, should not be too much of a problem. The code should be modular enough to allow checking without too much hassle in this case, and if it isn't, there is probably a bigger problem. :)

But all in all, of course, I cannot speak for Linus, either, and as I mentioned above, I don't know that much about the actual inner workings of the kernel, either, so my opinion may simply be that of a layman. And it's not like I'm not happy with what has been merged into 2.5 already, either - I really think that a very impressive array of features and improvements has went in, but still, the things on Rusty's list are interesting as well, and as far as I understand it, he has already picked only those which he himself considers to be ready for prime time. So it's not the obscurest of patches at least... :)

A practical definition of bloat

Posted Nov 1, 2002 2:54 UTC (Fri) by chad.netzer (✭ supporter ✭, #4257) [Link]

'bloat' is often used as a reason for not including this or that. And I've seen many with the attitude of "it's not bloat, it can be compiled out. And source code download size is not a problem", etc.

But most of the current patch rejections, where 'bloat' is a factor, are due to feature bloat (orthogonal to size of executable, or size of download, or running size in memory, etc.) And feature bloat is the most important point (from Linus's perspective), because as was stated earlier, once it goes in, no matter what comes along that is better, it is hard to remove it. And maintaining features that are used by but a few, can be a REAL burden. With all the features being pushed for the deadline, even very technically sound patches can be justifiably dropped.

I for one thought epoll() should have been held back a release, to see if an overall better API could be conceived. Just like vfork(), stat(), ioctl(), <other legacy crap>, etc. once it has users it cannot be dropped, and others will adopt it, even if a more elegant method comes along in a year.

So, just keep that in mind when you see 'bloat', that whether or not something can be compiled in or not, whether it touches few parts of the core, or whether its run-time impact is low, it is still bloated if it creates a maintenance burden that is unjustified.

Of course, agreeing on what is bloat or not is a whole other issue, separate from the shady definition of the word itself.

In defense of epoll

Posted Nov 1, 2002 9:41 UTC (Fri) by Peter (guest, #1127) [Link]

(+1, Insightful) - nice explanation of what bloat actually is in this context and why it must be avoided even when it doesn't directly affect the object code. But -

I for one thought epoll() should have been held back a release, to see if an overall better API could be conceived.

I disagree, in this instance. For one thing, epoll() isn't exactly new. A port of the Solaris /dev/poll first appeared somewhere in early 2.3.x, if I remember correctly. Linus shot it down on the spot - "we don't need no stinking device file", as it were - and thus the syscall approach was born. So this isn't someone's last-minute feature-freeze brainfart (like sendfile(), which was conceived, implemented and integrated all within about a week, just before 2.2.0) - it's been kicking around for quite awhile.

For another thing, note that the epoll stuff is itself a second system, quite aside from the /dev/poll business. It is already a redesign of the classic select() and poll() interfaces. In fact, you can go further: midway through 2.1.x, Richard Gooch (I think) introduced poll() as an independent syscall; previously it had been emulated using select(), which is not as efficient in the case of a sparse fd set.

So to summarise, epoll is a reimplementation of poll() which is a reimplementation of the select-emulation-based poll(), and people have been thinking about this interface for at least 2 years now.

I for one rather like the epoll API as described by Jon on these pages. It seems to map rather well to the familiar FD_ZERO(), FD_SET(), etc. used with select() ... so converting code to use epoll should not be too hard. My only question is whether it would have been worthwhile to specify a way to pass a whole array of fd's rather than just a single fd per syscall, but I suspect it doesn't matter much - syscalls are cheap in Linux, and you only have to set up your epoll structure once (that being kind of the whole point!).

In defense of epoll

Posted Nov 1, 2002 21:53 UTC (Fri) by chad.netzer (✭ supporter ✭, #4257) [Link]

Thanks. I don't mean to pick on epoll(), other than that it is a current example of a feature that is immediately useful, relatively small and unintrusive to add to the source, but could still be considered 'bloat' (rightly or wrongly), by those who might want the freedom to adopt yet another event delivery mechanism in the future.