Commitment to Community(?)

Posted Mar 5, 2011 12:26 UTC (Sat) by coriordan (guest, #7544)
In reply to: Commitment to Community(?) by asherringham
Parent article: Commitment to Open (Red Hat News)

> in what way does this make it "difficult" [...]?

When you get a kernel from Debian, Ubuntu, OpenSuSe, etc. you can see the original source plus a few hundred patches. Each patch addresses a single change, and does so by modifying one or more files. Fictional example: power_management.c, power.h, and core.c.

If you want to see how they changed something in the power_management.c code, you can see which patch modifies this code, and you will also see the modifications to power.h and core.c that were part of this change.

If there were two unrelated changes made in the power_mangement.c code, you'll see two patches and you can choose the one you're interested in.

This makes it easy to make the same change to your distro, or to push the change upstream. Until recently, all distros did this.

Under Red Hat's new system, all changes are lumped together into a single patch. Their single patch will modify 1,000 files. If you see something useful in their power_management.c code, you've no way to see which of the other 1,000 modifications are part of this change (in our example: power.h and core.c), and there's nothing to indicate if two changes to power_management.c are related or not.

Commitment to Community(?)

Posted Mar 5, 2011 14:46 UTC (Sat) by msnitzer (subscriber, #57232) [Link] (25 responses)

The key piece here is that Red Hat does follow an "upstream first" policy. Red Hat is the biggest corporate contributor to upstream Linux.

The upstream community, downstream distros, and Red Hat benefit from later versions of upstream Linux having the fixes Red Hat (and others) identified as necessary. The benefit comes in the form of a guarantee that the next time an upstream Linux kernel is selected as the starting point for RHEL++, Fedora++, SLES++, etc. the previously required fixes will all be in place (eliminating most concerns about regression: performance, feature, etc).

So it is in Red Hat's interest to continue its strong commitment to "upstream first". As a Red Hat kernel developer I can tell you that "upstream first" isn't just some mantra -- it governs my development workflow. If a change isn't accepted upstream it has an _extremely_ difficult time of getting into RHEL.

Commitment to Community(?)

Posted Mar 5, 2011 22:59 UTC (Sat) by coriordan (guest, #7544) [Link] (24 responses)

There's something of a contradiction in Red Hat's official response.

On one hand, Red Hat says that by discontinuing the itemization of kernel patches, they can put other distros at a disadvantage and thus "compete" against rivals, and on the other hand they say that since all the important stuff is "upstream first" they're not really being unfreindly to the community.

Which is it? Are they being obstructive other distros or not? The Debian Linux package maintainer says they are.

So, the two unanswered questions are:
* Is Red Hat saying that the Debian Linux package maintainer is wrong (or exaggerating, or being blown out of proportion by the media)?
* Has Red Hat found a way to impede nasty rivals without being obstructive to the general kernel community?

Commitment to Community(?)

Posted Mar 6, 2011 1:36 UTC (Sun) by jmalcolm (subscriber, #8876) [Link] (6 responses)

I do not see a contradiction at all.

They are trying to be unfriendly to distributions that do not exist primarily as part of the "community" but rather as satellites of the RHEL universe. Think "Unbreakable Linux".

If all the Red Hat changes go into the upstream first then everything that Red Hat does contributes to the common pool of software. You can see all the patches that Red Hat does to the Linux kernel for example by tracking the mainline kernel itself. A true "community" distribution will have no problems.

However, if what you are really trying to do is piggy-back on Red Hat by pretending that you have assembled a simliar product from the available Open Source technologies then you have a tougher time knowing exactly what you are offering your customers. If you are just trying to poach support revenue away from businesses running RHEL then you have some more work to do.

If we want to have companies that behave a well as Red Hat does overall we are going to have to be somewhat more understanding about what it will take for them to survive. Should life be hardest on those that try the hardest?

Commitment to Community(?)

Posted Mar 6, 2011 1:41 UTC (Sun) by jrn (subscriber, #64214) [Link] (5 responses)

> You can see all the patches that Red Hat does to the Linux kernel for example by tracking the mainline kernel itself. A true "community" distribution will have no problems.
>
> However, if what you are really trying to do is piggy-back on Red Hat by pretending that you have assembled a simliar product from the available Open Source technologies then you have a tougher time knowing exactly what you are offering your customers.

What this analysis misses is that various people are working to maintain v2.6.32 as a longterm release. I don't see how that is less of a "true community" endeavor than tracking mainline.

Commitment to Community(?)

Posted Mar 6, 2011 2:01 UTC (Sun) by rahulsundaram (subscriber, #21946) [Link] (4 responses)

Vanilla 2.6.32 is pretty different from what is included in RHEL 6 in so many ways. Not a very relevant platform for coordination for Red Hat.

Commitment to Community(?)

Posted Mar 6, 2011 10:36 UTC (Sun) by makomk (guest, #51493) [Link] (3 responses)

Except that as someone pointed out earlier, the maintainer of the official 2.6.32 long-term release has said that Red Hat's refusal to publish broken-out patches for their kernel has made maintaining it as a long-term release harder for them. In fact, if you recall the original LWN article, the only reason we found out about this in the first place is because it was obstructing the 2.6.32-series maintainers!

Commitment to Community(?)

Posted Mar 6, 2011 10:56 UTC (Sun) by rahulsundaram (subscriber, #21946) [Link] (2 responses)

The fact that Red Hat picked 2.6.32 as a base kernel to build on for EL 6 doesn't seem to part of any coordinated effort. It just happens to fall in the right timeline. In the case of EL6, we are talking about 7 to 10 years with increased divergence over a period of time considering the enterprise model of backporting fixes or in some cases rewriting or creating new patches because upstream has moved ahead in a different way.

The decision to maintain 2.6.32 series for a longer term is a independent one and while there is a small level of intersection between them at this point, that won't last. It is unfortunate, certainly that stable release process is not made easier now but the individual patches get pushed upstream anyway and the fixes end up in the Linus tree and the benefits shared.

Commitment to Community(?)

Posted Mar 7, 2011 23:40 UTC (Mon) by rahvin (guest, #16953) [Link] (1 responses)

Blame Oracle. No one wants to say it but they are the problem. Redhat never had a major issue with CentOS, but when Oracle showed up, copied RHEL, renamed it and tried to start poaching Redhat's clients that created the problem. Obviously Redhat has lost some clients do to this and is going to make Oracle spend some cash to simply copy them.

It's too bad some companies are only interested in the revenue and not supporting the community. I'd personally like to see this hurt Oracles revenue to the point that the entire project is killed. This is the result of a bad community member and it would be better if they just went away.

Commitment to Community(?)

Posted Mar 8, 2011 0:32 UTC (Tue) by jmalcolm (subscriber, #8876) [Link]

I think that Novell had a hand in it too but yes, Oracle seems quite happy to be the tragedy in the commons.

Commitment to Community(?)

Posted Mar 6, 2011 20:19 UTC (Sun) by ceplm (subscriber, #41334) [Link] (16 responses)

We are trying to make it difficult to *support* RHEL, not to develop Linux kernel. And BTW, I have just checked (that's regarding concerns about making life difficult for Debian & other free distros development). In folder with unpacked src.rpm for Fedora Rawhide kernel:

bradford:kernel $ ls *.patch *.diff |wc -l
69
bradford:kernel $

All this is just about the RHEL kernel, not the Fedora one, IIUC.

This is partly a communication problem

Posted Mar 6, 2011 23:59 UTC (Sun) by coriordan (guest, #7544) [Link] (11 responses)

> We are trying to make it difficult to *support* RHEL, not to develop Linux kernel.

No one's questioning that, AFAICT.

What the community is asking is: what's Red Hat's response to the complaints of kernel maintainers Maximilian Attems and Greg Kroah-Hartman?

No one's questioning Red Hat's motives, their contributions, or their right to impede the business of parasites. I'm not anti-Red Hat, and I don't think that posing the above question is an attack on Red Hat. It's just a valid question, but so far, there's no answer.

This is partly a communication problem

Posted Mar 7, 2011 0:36 UTC (Mon) by msnitzer (subscriber, #57232) [Link] (1 responses)

Have you seen this answer?

This is partly a communication problem

Posted Mar 8, 2011 14:29 UTC (Tue) by coriordan (guest, #7544) [Link]

Your post makes the points that:
* RH pushes stuff upstream
* Porting patches from RH's kernel to stable is inherently difficult

I think the kernel developers already know this. They didn't question or complain about either of these things, they complained about the new Big Kernel Patch.

A sample good answer from RH would be "Ok, in future we'll set up a phone call between the relevant person here and the stable kernel maintainer before each release".

I'm not trying to embarrass RH by asking this question. On the contrary, I see them often as the role model for free software participation. So when they do something which shouldn't be copied by others, I think the question is worth asking: can this problem be fixed?

If this story was about Oracle, I wouldn't bother hoping for a fix.

This is partly a communication problem

Posted Mar 7, 2011 3:49 UTC (Mon) by ricwheeler (subscriber, #4980) [Link] (8 responses)

RHEL6 is *not* 2.6.32 stable based and never has been.

If you are a developer with a distro based on a stable series, you have to do your homework. When another upstream developer flags a patch as stable material, you need to understand if that patch is applicable to your kernel. That patch might - or might not - apply. It might be totally irrelevant.

Should Red Hat developers backport their fixes for other developers' branches? Why just .32 stable which Oracle and SLES use? Why not backport to the embedded stable series as well?

This is partly a communication problem

Posted Mar 7, 2011 4:50 UTC (Mon) by foom (subscriber, #14868) [Link] (4 responses)

I think you're being disingenuous. RHEL6 is 2.6.32-based. So it seems reasonable for maintainers of other distros based on 2.6.32 (like Maximilian for Debian, or GregKH for upstream-stable) to want be able to check RedHat's kernel source for patches which might be applicable to their kernels as well.

RedHat of course doesn't need to themselves backport fixes to other people's kernels (including 2.6.32-stable)...now, ideally, they *would* also participate in upstream stable branch maintenance themselves for whatever kernel they've based on (everyone else seems to be doing so...), but, it's okay if they don't. However, it's *not fine* to intentionally make it difficult for others to do that work. That's incredibly lame.

I hadn't seen this link referenced here before, so, here it is for anyone else that missed it:
http://lwn.net/Articles/430600/

This is partly a communication problem

Posted Mar 7, 2011 11:27 UTC (Mon) by ricwheeler (subscriber, #4980) [Link] (3 responses)

Strange, as someone directly involved in developing RHEL6, I thought that I would know more about how we did than you did.

Here are the facts:

RHEL6 is based on Fedora which was our point of forking when it had a 2.6.32 based kernel. Note I did not say 2.6.32 stable.

After forking from mainline, we continued to work on tip of upstream actively during the alpha and beta phases on key subsystems.

Our developers would post to upstream tip, flag patches for stable and provide a backport to RHEL6.

Before RHEL6 went out the door, many subsystems in the kernel were 2.6.34 or 2.6.35 based. At this point, some subsystems are much closer to 2.6.37 (specifically things like file systems that we have a very large and active upstream developer pool).

This is a key misconception - RHEL6 is not a 32 stable series product and never has been.

This is partly a communication problem

Posted Mar 7, 2011 13:09 UTC (Mon) by foom (subscriber, #14868) [Link] (1 responses)

> Strange, as someone directly involved in developing RHEL6, I thought that I would know more about how we did than you did.

I'm sure you do. However, you might note that *I* did not say it was 2.6.32-stable based either, only that it was 2.6.32 based...

> Our developers would post to upstream tip, flag patches for stable and provide a backport to RHEL6.

So you're saying Maximilian didn't actually need to look at the RHEL6 source at all, because all of the bugfix patches RHEL6's kernel includes have already been flagged for backport to upstream stable branches? I get the impression that he wouldn't agree with that statement, but for myself, I have no information for or against.

If that is true, that would certainly make it *much* less of a problem that it's difficult for other people to do the work of checking out the patches.

This is partly a communication problem

Posted Mar 8, 2011 11:33 UTC (Tue) by pbonzini (subscriber, #60935) [Link]

> So you're saying Maximilian didn't actually need to look at the RHEL6
> source at all, because all of the bugfix patches RHEL6's kernel includes
> have already been flagged for backport to upstream stable branches?

Only he can answer, but my hypothesis is that

- he spelunked the RHEL6 changelog to distinguish: 1) features, 2) bugfixes which were in stable, 3) bugfixes which were not in stable because the original author forgot to CC stable@kernel.org.

- he focused on the third group here, which may have included patches from both RH and non-RH authors. RH is _not_ actively hiding their stuff from stable@kernel.org.

- he then used the summary of the patches in the changelog to identify upstream commits

- he backported those

Assuming he selected patches whose backports were relatively easy, no, he didn't need to look at the RHEL6 source at all.

To be fair, he would have probably benefited from the full commit messages; these are part of the patch files so they are also not included in RHEL6.

This is partly a communication problem

Posted Mar 8, 2011 1:42 UTC (Tue) by riel (subscriber, #3142) [Link]

Several subsystems in the RHEL 6 kernel have been upgraded to be close to 2.6.33, 2.6.34 or newer. Fixes to that code may simply not apply to the 2.6.32-stable codebase.

People who use 2.6.32-stable may need to do their own backports, since many of ours won't apply anyway.

This is partly a communication problem

Posted Mar 7, 2011 13:54 UTC (Mon) by lmb (subscriber, #39048) [Link] (2 responses)

Hi Ric, yes, well, there is a point to that. But it is also common for partners and customers to report "works with kernel XY", and now one of the parties has made it much more difficult for other maintainers to look up why - while the others still publish their broken out series.

While I personally wish the whole enterprise distribution approach would go away (and people would instead pay us for supporting them on latest upstream - hey, a boy can dream), I also know that there's a pretty good cooperation between the engineering teams. I hope that is not affected.

This is partly a communication problem

Posted Mar 7, 2011 14:31 UTC (Mon) by ricwheeler (subscriber, #4980) [Link] (1 responses)

I don't see any reason that work upstream is impacted by how we package patches against the RHEL6 tree.

In addition, I fully expect our kernel team to continue to help any one on any distro understand, debug, etc just like we always do.

This is partly a communication problem

Posted Mar 7, 2011 20:30 UTC (Mon) by dlang (guest, #313) [Link]

how long can the engineers on the kernel team continue to help others when management is explicitly trying to make it harder for those same others?

customers also have to support the product

Posted Mar 7, 2011 14:34 UTC (Mon) by mcoleman (guest, #70990) [Link] (3 responses)

I'm sympathetic to RedHat's plight, but I'll point out that as an admin, I *also* have to support the kernel. It appears that this makes it more difficult for me to do so, and thus would seem to make RHEL a less attractive option.

(I don't really have a dog in this, since I'm currently not using RHEL anyway, but obviously it's something to mention if someone asks whether we should start.)

customers also have to support the product

Posted Mar 8, 2011 0:40 UTC (Tue) by jmalcolm (subscriber, #8876) [Link] (2 responses)

I think that if you are a 'paying' admin that you do have access to this information. It is the 'public' SRPM that is all rolled-up into one.

customers also have to support the product

Posted Mar 8, 2011 0:53 UTC (Tue) by makomk (guest, #51493) [Link] (1 responses)

Yes, except that you're not allowed to tell anyone else what patches Red Hat have applied to the kernel they gave you. So for example, if you encountered a problem with part of the kernel and managed to convince one of the upstream maintainers of that part of the kernel to help you diagnose it, you wouldn't be allowed to tell him or her what changes Red Hat had made that may affect his or her code. That could be an issue.

Red Hat's new policy essentially makes it impossible for third parties to help RHEL customers with any kernel problems they encounter, should Red Hat fail to deal with them itself.

customers also have to support the product

Posted Mar 8, 2011 12:05 UTC (Tue) by anselm (subscriber, #2796) [Link]

Red Hat can't prevent you (or anybody) from running diff against a (small number of) file(s) from the vanilla vs. RHEL kernel sources (from the SRPM). This is infeasible for all of the kernel, but if a problem can be localised to an individual driver or other manageable subsystem, this may be all that is needed for a third party to figure out what changes Red Hat has made to that particular part of the kernel.