The current development kernel is...linux-next?

One of the development process advantages brought by git (and by BitKeeper before it) is the ability to see the up-to-the-second, bleeding-edge status of Linus's tree. So any developer who wants to know where the front edge of development lies can grab that tree and make patches fit into it. But the value of the mainline repository for development would appear to be less than it once was. The mainline is no longer where the action is.

Consider, for example, this response from Andrew Morton after finding that a patch posted to linux-kernel would not compile for him:

He followed up with this statement:

So the message would appear to be clear: development work should be done against the linux-next tree rather than against the mainline kernel. There are some clear advantages to having work done in this way. Patches developed against linux-next should merge cleanly during the next merge window. Developers will be testing each other's trees as they work, causing bugs to turn up earlier in the process. And, of course, Andrew won't have to complain about patches which fail to build for him - at least, not as often.

Linux-next is a somewhat strange base on which to try to develop, though. It is built anew every day from over 100 subsystem trees, each of which can, itself, change from one day to the next. So linux-next is a moving target, just like the mainline is. But, unlike the mainline, linux-next has no consistent or coherent history. Every day's linux-next tree is a completely new creation with a unique - and transient - history.

Consider a developer who bases some work on a mainline release - 2.6.26-rc9, say. That developer's work will be derived from a specific commit in the mainline tree, known as b7279469d66b55119784b8b9529c99c1955fe747 in this case. The history from 2.6.26-rc9 is well defined, and that series of patches can be merged into any other repository which also contains 2.6.26-rc9; the identity of that commit is consistent and immutable across all repositories. With such a development tree, it is (relatively) easy to track the mainline as it advances, and to merge one's work when the time comes. A git tree based on the mainline sits on a solid foundation.

It is not possible to base a tree on linux-next in the same way. Development can begin at a specific commit, but tomorrow's linux-next tree may not contain that commit at all. The various component trees will have advanced independently of the previous day's linux-next tree, which can, in itself, complicate things. But the process of making all those trees come together can involve tasks like moving patches from one tree to another, or fixing intermediate patches which break things. That makes the end result better, but at the cost of rebasing those trees. Rebasing completely rewrites the development history, causing the old history to disappear from the tree. So a patch series based on the previous history loses its foundation.

And, since linux-next is built from its components every day, a patch developed on top of linux-next may, when integrated into that tree, be merged somewhere in the middle of the sequence; in other words, the patch will be merged into a tree which differs considerably from the tree on which it was developed. As Stephen Rothwell, the maintainer of the linux-next tree, put it:

Another interesting aspect of linux-next development involves API changes. The longstanding rule in kernel development is that internal kernel interfaces can be changed if there is a good reason to do so, but that the person making the change is obligated to fix all in-tree code broken by that change. If an API change is introduced into linux-next, though, the developer is simply not able to fix any code which enters linux-next by way of the other subsystem trees. If the developer does get patches into those trees for the API change, they can no longer be built on top of kernels which lack that change - the mainline, for example. API changes have, in other words, become harder to do - a situation which some may see as a good thing.

What all this means is that API changes must be handled through techniques like the creation of backward-compatibility layers; those layers can then be removed a development cycle or two later once the transition is complete. Or changes can be split up and added to individual subsystem trees; that, however, can lead to interesting ordering dependencies between the trees. In some cases, we are seeing 2.6.27 changes being merged into 2.6.26 in stub form as a way of making all of the pieces fit together.

Then, there is the simple matter that developers like to have a stable base upon which to create their code. The linux-next tree, since it contains large amounts of relatively new code, will also contain its share of new bugs. That makes developers, who are often having enough trouble just tracking down their own bugs, somewhat grumpy. Development against the mainline tends to have a lower probability of forcing developers to look for bugs which are not of their own making.

Many of these complaints have an easy answer: the pain which comes from making all the pieces fit together in linux-next must be faced at some point anyway. The real difference is that linux-next allows those problems to be dealt with at leisure, while the older "merge everything in the mainline" model compressed much of that work into the merge window. How beneficial that really is will be seen for the first time in the 2.6.27 merge window; if linux-next is serving its intended function, 2.6.27 should come together with rather less hassle than its immediate predecessors did.

But, regardless of the value provided by linux-next for integration and testing purposes, the fact remains that it is a difficult platform upon which to develop patches. That process is somewhat like building a house on a sand bar; overnight the tide comes in and completely reshapes the land underneath you. That is why most (possibly all) of the subsystem trees used to assemble linux-next are, themselves, based on the mainline.

The solution to that problem will have to evolve over time. The linux-next tree is a new institution which is still finding its proper place in the development process. Easier ways to develop patches against the linux-next tree will certainly be worked out; it may well turn out that quilt-like tools work better for this task than git. But, for now, linux-next is an excellent integration and testing resource, but it has not quite yet managed to become the true Linux kernel development tree.

---

(Log in to post comments)

well, shouldnt development being done in the subsystem-trees?  that way their maintainers are
looking after the integration... this puts burden on the maintainers...but perhaps scales
well?

dunno, maybe...

Aren't there something like 100 subtrees?  If you develop in one or even a few of them, all
the others are still more or less unseen.  They don't come together until linux-next.

So why is a rebase each day necessary for linux-next?  Can't it just keep a smooth history
with subsystem changes pulled in each day?  I thought git would support that kind of thing.

It's exploring what history will look like after the next merge window.  To do that, it needs
to try many different versions of that history until everything fits together with, hopefully,
the least amount of effort.

I still don't understand.  Why is the rebasing necessary?  It is because we don't *want* to
see the intermediate states (the entire history of linux-next) when we later look back at the
history of trunk?

If that were all that were desired, couldn't you, um, rebase just before merging it back to
trunk?

Sorry if these are dumb questions, but I really don't understand git very well.

"I still don't understand.  Why is the rebasing necessary?  It is because we don't *want* to
see the intermediate states (the entire history of linux-next) when we later look back at the
history of trunk?"

Yes.  Though it'd also make the history of linux-next itself pretty complicated.  If they were
willing to add a ton of merges they could preserve all of the old versions of linux-next in
its history.  The resulting history would be very messy--e.g. you'd probably see the same
changes made multiple times in multiple places (since a new commit would be created each time
a developer revised or rebased a patch).

"If that were all that were desired, couldn't you, um, rebase just before merging it back to
trunk?"

Yeah, so linux-next would end up carrying a bunch of "meta-history" that wasn't ever submitted
to mainline.

I dunno whether it'd help or just confuse people.

I think Andrew is just wrong about what people should be developing against. If you develop
against -next, then your changes won't be acceptable for inclusion in -next if there are
conflicts, because of the way -next is generated. If, for example, after you start working,
some series that was already in -next gets revised, your tree will now generate a slew of
conflicts in code you've never looked at, because you'll have Stephen merge all of that
particular old -next plus your series with the current -next when he pulls, and do it as if
the two -next versions were developed independently.

What would be more useful is to develop based on what -next bases on, but test the merge of
your code and -next, and make that merge available. That gets the development history clean
and clear, checks that integration with the rest of -next works, tests that the rest of -next
works for you, and also should allow git to get information on the proper merge with various
-next versions so that the integration is smoother.

I assume from your comments that you're using git not quilt?  (since quilt doesn't really care
if you change the tree under it, you just do a quilt pop -a; install new tree; quilt push -a)

The way to develop against linux-next in git is to use git rebase --onto to move your patches
as linux-next changes.

So, assume you have a git tree with your patches on the master branch.  You also need to
record the base of your branch with either a tag or a branch (you do this before you start
adding patches) so that git format-patch base produces exactly all the patches you're
managing.  The way I set this up is to clone a tree (git clone <tree>) record the base (git
branch base) and simply develop committing along the way.

Now set up remote tracking branches (I tend to do one for linux-next and one for linus as):

git remote add -t master -f linus
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

git remote add -t master -f next
git://git.kernel.org/pub/scm/linux/kernel/git/sfr/linux-next.git

The point of keeping the linus tree is that eventually all the branches that comprise next are
merged into linus, at which point your patches will be ready for submission against vanilla
linus.

To download the current trees of the day, simply do a

git remote update

and to put your patches onto the current linux-next do

git rebase --onto remotes/next/master base

If this is successfully completed, you need to record the fact that you've moved the base of
your patches:

git branch -M base remotes/next/master

You can keep doing this as many times as linux-next updates.

If you think you're ready to try to place your patches onto vanilla linus, just repeat the
recipe but with remotes/linus/master instead of remotes/next/master.  If your attempt to
rebase onto linus fails (usually because of missing commits that were in linux-next that
you're relying on) simply do git rebase --abort.

The problem isn't that developing against a rebasing tree is too difficult, because it's
generally not too hard to update things. The problem is that developing against a rebasing
tree generates a repository which isn't suitable for inclusion in the rebasing tree. That is,
following your instructions would be fine if -next weren't a useful tree to try to get your
stuff into, but the resulting repository isn't something that Stephen can pull into -next.

I suppose if you're targeting 2.6.n+3 (or later) for inclusion, you could develop against
-next, but that's sort of pessimistic. And I think that, if it makes sense to publish your
changes at all, it makes sense to have them in a state where they go against vanilla linus.

Also note that, if you develop against -next, either it doesn't matter, because you don't have
any interactions with other people's changes, or you're going to waste a lot of time when a
tree that you interact with gets some conflict and Stephen drops it (requiring you to resolve
a slew of conflicts) and then picks it up again the next day (requiring you to resolve the
same slew of conflicts the other way, back to essentially the state you had before). And that
applies to quilt as well; quilt will happily let the base change under it, but you'll have
days when your code won't work without a pile of fixups, and the "bug" it will be due to is
that some code that you now require is temporarily and intentionally missing.

Well, if you're running a development tree, it isn't eligible for inclusion into linux-next,
since linux-next only consists of everything that will go in in the next merge window.  What
you'd be developing this way is a patch series being posted to a mailing list and refined.
Once it's ready for inclusion, and everyone's signed off, then you get your own tree or send
them to a subsystem maintainer (and usually the latter if you have conflicts against linus
because you need to go through the tree of the subsystem you're in conflict with).

With this method of development, you can be targeting 2.6.n+1 since the delta between linus
and linux-next goes to zero within the merge window.

The whole point of developing against linux next is to see where you do pick up conflicts.  If
you do and you don't resolve them (which the rebasing method allows you to), Linus is hardly
going to choose to merge your tree over the subsystem tree you just conflicted with.  It's far
easier to resolve conflicts incrementally (as in daily) than wait for weeks and see how bad it
becomes (believe me, I've done both).

mkdir -p ~/src/kernel/
cd ~/src/kernel/
git init linux-next
cd linux-next
git remote add -t master -f linus git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
git remote add -t master -f next git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
git checkout remotes/next/master
git branch base

# vi/edit/commit/*rebase/view/ect till your harts content.

I, for one, welcome our new linux-next overlords.  From what the article describes, this
should really bite those devs in the ass who just keep churning the code to boost their own
egos.  After a few rounds with linux-next as the development branch, developers will finally
get really angry at anyone who churns code without a good reason.  Code churners will get
their asses kicked, and things should calm down and stabilize, with people being much more
cautious about changes to the internal interfaces of the kernel.

Good on you, Andrew Morton!  Linux will finally move toward being as stable as the BSDs.

I would of thought the Linus tree would be what you want to develop on. Once it's hit rcN
status API changes should be minimal and stability approaching release quality.

Fundamentally if it's going to get pulled into the Linux tree it will need to be based on it,
otherwise git will get confused as to the common ancestor.

However it seems sensible that you should test your feature branch against the linux-next du
jour and check it won't get broken on that merge. If it does then you need to track down the
sub-system that's changing and work with that tree, possibly re-basing on their for-linus
branch. linux-next is far too much of a churn fest to base work on it, and it shouldn't be
used that way.

Since long ago in the arm world there is nearly the same picture as we are faced now with
linux-next. Russell puts patches in several branches as he finds usefull and then merged them
into single tree. For managing development in such configuration I found extremly usefull to
use guilt by jsipek. It's like quilt, but with full git integration.

Is it just me or is b7279469d66b55119784b8b9529c99c1955fe747 utterly obscure?

I guess I don't get git yet. Yikes.

It's just a unique identifier. All the git commands accept these (as well 
as tags, branch names and a huge bunch of other ways of identifying 
commits, trees, and so on, see 'Specifying Revisions' in the 
git-rev-parse(1) manpage).

Try 'git show b7279469d66b55119784b8b9529c99c1955fe747' to see what it is.

I'm not sure that I have understood all the git/quilt obskure usages here. 
But I have the following question: Why is it necessary to rebuild 
linux-next on a daily basis? I understand that it is build on a bunch of 
special purpose trees. But after aggregating all this, can't we make a 
diff to last day linux-next and take this patch to check in a new 
linux-next??? This way all linux-next would be based on each other... 

"can't we make a diff to last day linux-next and take this patch to check in a new
linux-next???"

The resulting diff would represent all the work done by all maintainers in a day.  It would be
large, and not very logical.

If I have a conflict with something in linux-next, then I want to be able to identify exactly
the smallest logical change that produces the conflict, and who did it (in case I need to ask
them about it).  It would be difficult to do that if a single change that's checked in is a
big agglomerations of unrelated stuff by lots of different developers.