LWN: Comments on "The current development kernel is...linux-next?"

How to develop against linux-next

cheako — Sun, 13 Nov 2011 20:56:57 +0000

An example:

mkdir -p ~/src/kernel/
cd ~/src/kernel/
git init linux-next
cd linux-next
git remote add -t master -f linus git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
git remote add -t master -f next git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
git checkout remotes/next/master
git branch base

# vi/edit/commit/*rebase/view/ect till your harts content.

* Rebase done using instructions in original article, AFAIK.

How to develop against linux-next

cheako — Sun, 13 Nov 2011 20:46:06 +0000

This is the way git says "Repo not found."

How to develop against linux-next

cheako — Sun, 13 Nov 2011 20:43:17 +0000

Linux-Next is has [1]moved from the old location.

1. git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git

How to develop against linux-next

cheako — Sun, 13 Nov 2011 18:50:15 +0000

I've some newb questions, just act like I'm dumb and any one explain it to me, please.

What tree should be cloned? Does one clone linux-next and then configure it as a remote?

What's the reason for "git branch base" prior to configuring the remotes?

Reading the help files is confusing, I belief I'll only learn via experience.
What I did was "get init" and then configured the remotes. Upon configuring the -next remote, I got this lovely response from GIT.

# git2.kernel.org
[pid 21426] connect(4, {sa_family=AF_INET, sin_port=htons(9418), sin_addr=inet_addr("149.20.4.72")}, 16) = 0
[pid 21426] dup(4) = 5
[pid 21426] write(5, "0055git-upload-pack /pub/scm/lin"..., 85) = 85
[pid 21426] read(4, "", 4) = 0
[pid 21426] write(2, "fatal: The remote end hung up unexpectedly
"..., 43) = 43

Must have sent null byte first?

The current development kernel is...linux-next?

bfields — Thu, 10 Jul 2008 14:48:37 +0000

"can't we make a diff to last day linux-next and take this patch to check in a new
linux-next???"

The resulting diff would represent all the work done by all maintainers in a day.  It would be
large, and not very logical.

If I have a conflict with something in linux-next, then I want to be able to identify exactly
the smallest logical change that produces the conflict, and who did it (in case I need to ask
them about it).  It would be difficult to do that if a single change that's checked in is a
big agglomerations of unrelated stuff by lots of different developers.

Why is linux-next rebased every day?

bfields — Thu, 10 Jul 2008 14:40:52 +0000

"I still don't understand.  Why is the rebasing necessary?  It is because we don't *want* to
see the intermediate states (the entire history of linux-next) when we later look back at the
history of trunk?"

Yes.  Though it'd also make the history of linux-next itself pretty complicated.  If they were
willing to add a ton of merges they could preserve all of the old versions of linux-next in
its history.  The resulting history would be very messy--e.g. you'd probably see the same
changes made multiple times in multiple places (since a new commit would be created each time
a developer revised or rebased a patch).

"If that were all that were desired, couldn't you, um, rebase just before merging it back to
trunk?"

Yeah, so linux-next would end up carrying a bunch of "meta-history" that wasn't ever submitted
to mainline.

I dunno whether it'd help or just confuse people.

The current development kernel is...linux-next?

aanno — Thu, 10 Jul 2008 12:31:00 +0000

I'm not sure that I have understood all the git/quilt obskure usages here. 
But I have the following question: Why is it necessary to rebuild 
linux-next on a daily basis? I understand that it is build on a bunch of 
special purpose trees. But after aggregating all this, can't we make a 
diff to last day linux-next and take this patch to check in a new 
linux-next??? This way all linux-next would be based on each other...

The current development kernel is...linux-next?

nix — Thu, 10 Jul 2008 12:17:41 +0000

It's just a unique identifier. All the git commands accept these (as well 
as tags, branch names and a huge bunch of other ways of identifying 
commits, trees, and so on, see 'Specifying Revisions' in the 
git-rev-parse(1) manpage).

Try 'git show b7279469d66b55119784b8b9529c99c1955fe747' to see what it is.

The current development kernel is...linux-next?

dale77 — Thu, 10 Jul 2008 08:29:03 +0000

Is it just me or is b7279469d66b55119784b8b9529c99c1955fe747 utterly obscure?

I guess I don't get git yet. Yikes.

Compiled it today

proski — Thu, 10 Jul 2008 04:28:06 +0000

First I had to enable ACPI sleep, or PCI code won't compile. And then it won't "indentify" my SATA drive. Yet I needed to test some wireless stuff, and my time was limited. Back to wireless-testing.

Why is linux-next rebased every day?

zooko — Thu, 10 Jul 2008 01:48:30 +0000

I still don't understand.  Why is the rebasing necessary?  It is because we don't *want* to
see the intermediate states (the entire history of linux-next) when we later look back at the
history of trunk?

If that were all that were desired, couldn't you, um, rebase just before merging it back to
trunk?

Sorry if these are dumb questions, but I really don't understand git very well.

Use guilt, Luke!

Lumag — Wed, 09 Jul 2008 20:45:50 +0000

Since long ago in the arm world there is nearly the same picture as we are faced now with
linux-next. Russell puts patches in several branches as he finds usefull and then merged them
into single tree. For managing development in such configuration I found extremly usefull to
use guilt by jsipek. It's like quilt, but with full git integration.

How to develop against linux-next

jejb — Wed, 09 Jul 2008 19:14:56 +0000

Well, if you're running a development tree, it isn't eligible for inclusion into linux-next,
since linux-next only consists of everything that will go in in the next merge window.  What
you'd be developing this way is a patch series being posted to a mailing list and refined.
Once it's ready for inclusion, and everyone's signed off, then you get your own tree or send
them to a subsystem maintainer (and usually the latter if you have conflicts against linus
because you need to go through the tree of the subsystem you're in conflict with).

With this method of development, you can be targeting 2.6.n+1 since the delta between linus
and linux-next goes to zero within the merge window.

The whole point of developing against linux next is to see where you do pick up conflicts.  If
you do and you don't resolve them (which the rebasing method allows you to), Linus is hardly
going to choose to merge your tree over the subsystem tree you just conflicted with.  It's far
easier to resolve conflicts incrementally (as in daily) than wait for weeks and see how bad it
becomes (believe me, I've done both).

How to develop against linux-next

iabervon — Wed, 09 Jul 2008 17:44:16 +0000

The problem isn't that developing against a rebasing tree is too difficult, because it's
generally not too hard to update things. The problem is that developing against a rebasing
tree generates a repository which isn't suitable for inclusion in the rebasing tree. That is,
following your instructions would be fine if -next weren't a useful tree to try to get your
stuff into, but the resulting repository isn't something that Stephen can pull into -next.

I suppose if you're targeting 2.6.n+3 (or later) for inclusion, you could develop against
-next, but that's sort of pessimistic. And I think that, if it makes sense to publish your
changes at all, it makes sense to have them in a state where they go against vanilla linus.

Also note that, if you develop against -next, either it doesn't matter, because you don't have
any interactions with other people's changes, or you're going to waste a lot of time when a
tree that you interact with gets some conflict and Stephen drops it (requiring you to resolve
a slew of conflicts) and then picks it up again the next day (requiring you to resolve the
same slew of conflicts the other way, back to essentially the state you had before). And that
applies to quilt as well; quilt will happily let the base change under it, but you'll have
days when your code won't work without a pile of fixups, and the "bug" it will be due to is
that some code that you now require is temporarily and intentionally missing.

How to develop against linux-next

jejb — Wed, 09 Jul 2008 15:16:31 +0000

I assume from your comments that you're using git not quilt?  (since quilt doesn't really care
if you change the tree under it, you just do a quilt pop -a; install new tree; quilt push -a)

The way to develop against linux-next in git is to use git rebase --onto to move your patches
as linux-next changes.

So, assume you have a git tree with your patches on the master branch.  You also need to
record the base of your branch with either a tag or a branch (you do this before you start
adding patches) so that git format-patch base produces exactly all the patches you're
managing.  The way I set this up is to clone a tree (git clone <tree>) record the base (git
branch base) and simply develop committing along the way.

Now set up remote tracking branches (I tend to do one for linux-next and one for linus as):

git remote add -t master -f linus
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

git remote add -t master -f next
git://git.kernel.org/pub/scm/linux/kernel/git/sfr/linux-next.git

The point of keeping the linus tree is that eventually all the branches that comprise next are
merged into linus, at which point your patches will be ready for submission against vanilla
linus.

To download the current trees of the day, simply do a

git remote update

and to put your patches onto the current linux-next do

git rebase --onto remotes/next/master base

If this is successfully completed, you need to record the fact that you've moved the base of
your patches:

git branch -M base remotes/next/master

You can keep doing this as many times as linux-next updates.

If you think you're ready to try to place your patches onto vanilla linus, just repeat the
recipe but with remotes/linus/master instead of remotes/next/master.  If your attempt to
rebase onto linus fails (usually because of missing commits that were in linux-next that
you're relying on) simply do git rebase --abort.

Test not develop against linux-next

alex — Wed, 09 Jul 2008 10:17:06 +0000

I would of thought the Linus tree would be what you want to develop on. Once it's hit rcN
status API changes should be minimal and stability approaching release quality.

Fundamentally if it's going to get pulled into the Linux tree it will need to be based on it,
otherwise git will get confused as to the common ancestor.

However it seems sensible that you should test your feature branch against the linux-next du
jour and check it won't get broken on that merge. If it does then you need to track down the
sub-system that's changing and work with that tree, possibly re-basing on their for-linus
branch. linux-next is far too much of a churn fest to base work on it, and it shouldn't be
used that way.

LOL! End to Code Churn

roskegg — Wed, 09 Jul 2008 05:33:37 +0000

I, for one, welcome our new linux-next overlords.  From what the article describes, this
should really bite those devs in the ass who just keep churning the code to boost their own
egos.  After a few rounds with linux-next as the development branch, developers will finally
get really angry at anyone who churns code without a good reason.  Code churners will get
their asses kicked, and things should calm down and stabilize, with people being much more
cautious about changes to the internal interfaces of the kernel.

Good on you, Andrew Morton!  Linux will finally move toward being as stable as the BSDs.

I'm afraid it won't work well

proski — Wed, 09 Jul 2008 03:52:24 +0000

Developing against the mainline tree or a tree based on it limits the complexity of the changes that are allowed to happen between the releases. It assures that development happens in short, manageable steps. It may be hard to merge, but the changes are mostly limited to subsystems, and possible interference is cleaned up before the release.

When working on linux-next, the changes start interacting with each other early. That would allow bigger, intertwined changes. In may be easier to merge because the patches "grow" together, but debugging and cleaning the kernel for the release may be harder.

linux-next would also limit developers' exposure to the code that will be released as the next kernel version. It could make releases less stable.

The current development kernel is...linux-next?

felixfix — Wed, 09 Jul 2008 02:11:38 +0000

Aren't there something like 100 subtrees?  If you develop in one or even a few of them, all
the others are still more or less unseen.  They don't come together until linux-next.

Why is linux-next rebased every day?

russell — Tue, 08 Jul 2008 22:57:12 +0000

It's exploring what history will look like after the next merge window.  To do that, it needs
to try many different versions of that history until everything fits together with, hopefully,
the least amount of effort.

The current development kernel is...linux-next?

iabervon — Tue, 08 Jul 2008 22:14:07 +0000

I think Andrew is just wrong about what people should be developing against. If you develop
against -next, then your changes won't be acceptable for inclusion in -next if there are
conflicts, because of the way -next is generated. If, for example, after you start working,
some series that was already in -next gets revised, your tree will now generate a slew of
conflicts in code you've never looked at, because you'll have Stephen merge all of that
particular old -next plus your series with the current -next when he pulls, and do it as if
the two -next versions were developed independently.

What would be more useful is to develop based on what -next bases on, but test the merge of
your code and -next, and make that merge available. That gets the development history clean
and clear, checks that integration with the rest of -next works, tests that the rest of -next
works for you, and also should allow git to get information on the proper merge with various
-next versions so that the integration is smoother.

Why is linux-next rebased every day?

epa — Tue, 08 Jul 2008 21:35:18 +0000

So why is a rebase each day necessary for linux-next?  Can't it just keep a smooth history
with subsystem changes pulled in each day?  I thought git would support that kind of thing.

The current development kernel is...linux-next?

dmk — Tue, 08 Jul 2008 20:47:25 +0000

well, shouldnt development being done in the subsystem-trees?  that way their maintainers are
looking after the integration... this puts burden on the maintainers...but perhaps scales
well?

dunno, maybe...