One week of infrastructure issues [LWN.net]

One week of infrastructure issues

Posted Aug 21, 2008 1:26 UTC (Thu) by rahulsundaram (subscriber, #21946)
In reply to: One week of infrastructure issues by sbergman27
Parent article: One week of infrastructure issues

What are you talking about? I don't recall any plans to setup a public subversion repository
within Fedora Project and certainly not one for years. Can you provide a reference?

to post comments

One week of infrastructure issues

Posted Aug 21, 2008 1:40 UTC (Thu) by sbergman27 (guest, #10767) [Link] (17 responses)

You cannot possibly mean that you don't remember.  Although it turns out it was actually a CVS
repository, rather than Subversion, which had been promised since July 2003 (when Fedora was
still called the Red Hat Linux Project) and remained unrealized until, I believe, sometime in
2006, despite a persistent public clamoring and grumbling about the issue the entire time.
Does that jog your memory or do you still need links?

One week of infrastructure issues

Posted Aug 21, 2008 1:55 UTC (Thu) by sbergman27 (guest, #10767) [Link] (2 responses)

Actually, it looks like it was not 2006, but spring of 2007.  So 3 years and 10 months for the
promised CVS server.

One week of infrastructure issues

Posted Aug 21, 2008 6:48 UTC (Thu) by wtogami (subscriber, #32325) [Link] (1 responses)

Um... November 2004 the CVS server went public for Fedora Extras where we collaboratively
maintained thousands of packages around Core before the distro fully merged prior to Fedora 7.

One week of infrastructure issues

Posted Aug 21, 2008 21:02 UTC (Thu) by sbergman27 (guest, #10767) [Link]

Warren, yes. (I'll take your word on the date for Extras.  Sounds believable.)  Since the
early days, Fedora has had many auxiliary repositories like "Extras".  And yes, "Extras" was
kinda sorta special. But what people were clamoring for (for almost 4 years) was for the
"Core" of Fedora Core to open up.  You know, the "community distro" itself.  Not the auxiliary
repos.  And *you know that very well*.  How could you not?  At this late date, isn't it really
best to just admit it and go on?  As I recall, it was only *very shortly* before the end of
May 2007 release of F7 that the merged cvs repo was completed.  

http://lwn.net/Articles/232958/

And I could dig up plenty of "What the hell is taking the Fedora guys so long?" articles over
the 4 year period, including at least one pretty good one from LWN, IIRC.  Want me to?

One week of infrastructure issues

Posted Aug 21, 2008 2:02 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link] (13 responses)

"You cannot possibly mean that you don't remember"

Of course, I can quite honestly claim that. Why not?

"Although it turns out it was actually a CVS
repository, rather than Subversion"

Ah, there in lies the rub. I knew for a fact that Subversion was never really considered a
serious contender for anything in Fedora Project although it is a supported option in
fedorahosted.org. So your claim of subversion repository threw me off the mark. 

Also opening up existing infrastructure that has existed for over a decade within a
organization and in many cases rewriting it to be community facing is a entirely different
ball game from merely setting up a cvs repository. For a open setup, you need public facing
build systems for exampleand that was written from scratch for Fedora, twice even. Sure, there
was a well known delay during the initial stages of the project but it was also a non trivial
effort to put it mildly. You seem to be drawing parallels between very different situations.

One week of infrastructure issues

Posted Aug 21, 2008 2:41 UTC (Thu) by sbergman27 (guest, #10767) [Link] (12 responses)

"""
Ah, there in lies the rub.
"""

I find this surprising.  The issue was a high profile one for a very long time.  You really
did not make the connection because I said Subversion instead of cvs?  And does it really
matter if the server it took them nearly 4 years to set up was cvs or subversion?

"""
Also opening up existing infrastructure that has existed for over a decade within a
organization and in many cases rewriting it to be community facing is a entirely different
ball game from merely setting up a cvs repository.
"""

That was the party line during most of that time.  I will agree that this is a somewhat
different situation.  But not for the party-line reason you restate here.  Red Hat did not
really want to open up Fedora that much.  They felt they had too much riding on it for that.
(I remember a Red Hat official even coming close to saying it in so many words.)  In this case,
Red Hat also likely has a keen self interest, and are likely to want to get the servers up as
soon as possible.

BTW, that is not intended to come off as anti Red Hat.  I greatly respect Red Hat.  But the
low priority foot dragging was pretty obvious.  You can bet that if "opening up existing
infrastructure that has existed for over a decade within a organization" had been perceived as
critical to the RHEL product, they could have done it in a matter of months if not less.

One week of infrastructure issues

Posted Aug 21, 2008 3:03 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link] (2 responses)

"You really did not make the connection because I said Subversion instead of cvs?"

Yes and also so because it is not the same people like you claimed and I am not sure your
dates are right either. I am very specific that way and that is one of the reason I ask for
references. 

"And does it really matter if the server it took them nearly 4 years to set up was cvs or
subversion?"

Simply setting up a cvs or subversion system wouldn't result in much without a open
buildsystem, compose tools etc to go along with it and these things take time. It took until
Fedora 7 in fact to convince everybody including internally to really unify all these
infrastructure bits. Like you admit, this situation is pretty different so I am not even sure
why you bought it up in the first place.

One week of infrastructure issues

Posted Aug 21, 2008 13:21 UTC (Thu) by interalia (subscriber, #26615) [Link] (1 responses)

Well I have to say that I also felt it took a long time for Fedora to open up. I'm just
posting as a disinterested Debian user with nothing against Red Hat or Fedora, but it took an
age, with little feeling of progress from the vantage point of someone who just generally
follows Linux and distro news (ie. a typical LWN reader). In hindsight it was perhaps simply a
failure to manage expectations, but I can understand the frustration that other people felt.

One week of infrastructure issues

Posted Aug 21, 2008 17:26 UTC (Thu) by jspaleta (subscriber, #50639) [Link]

"Well I have to say that I also felt it took a long time for Fedora to open up."

No one is going to deny that it took longer than would have hoped to clear the huddles
presented in opening up what use the be completely internal infrastructure used to build rhl
and build an open community infrastructure that would allow the eventual merging of Fedora
Core and Extras. I personally remember a few rounds of discussion concerning issues with
compliance of the Sarbanes-Oxley Act which came into effect in 2002.. a new law that affected
every single publicly traded company in the US. Maybe in hindsight, the discussions concerning
sarbox compliance as it relates to opening up things for Fedora were unneeded..but in the
context of what was happening at the time.. it was probably an unavoidable reality which
slowed the process down.

I'm sure there's a great coffeetable book to be made of the anecdotal stories related to the
building of the Fedora Project. I'd probably buy it too..as long as the proceeds were churned
into the Fedora Scholarship program. I'd love to read seth's and gafton's recollections of
the process of getting a public cvs system up and running for Fedora Extras.. with a lovely
cartoon illustration of them in a boxing ring fighting about it.

Yes, it was a painful process, and an important one, but in a way that most probably don't
think about. The great achievement is not the building of an external community around Fedora.
Debian proved well before Fedora came into its own that you can build a community of
volunteers and get something significant done. There was a community already in place at
Fedora.us before RHL was ended and the Fedora project began. The community organization was
already happening on its own..that's how cool the open ecosystem is.. RHL users were already
self-organizing at Fedora.us before Red Hat decided to take the plunge.

No the most important result of what Fedora as a project has achieved is the deep
internalization of an open development culture inside of Red Hat itself. The fact that Red
Hat continually re-invests in the community in ways that allow technology to grow outside of
its direct control. The most recent battle in the culture war was won with the release of
Spacewalk as an open technology. And now Red Hat is taking its experience earned in that hard
fought cultural war and helping its own customers better understand how to internalize and
benefit from the same open development culture that powers Debian or Fedora or hundreds of
other individual project pieces out in the ecosystem.

As members of a larger open development ecosystem.. larger than Fedora or Debian or whatever
project you have a personal commitment to... we are better off now with Red Hat as a full
partner, than we would be by encouraging Red Hat, the corporate entity, to stand outside of
that process.

We can nitpick previous mistakes in the process of how we got to where we are forever, but its
not particularly helpful. Hell, its not even the same group of people in the discussion that
was happening in 2002, there are a lot more voices now, because the project has grown so much.
People who use to be external 'community' are now 'red hat' and there are new external voices
doing new work and pushing things forward in new ways. What matters is understanding how
Fedora exists right now and whether its doing its job to push open innovation forward in a way
that every single person using a linux distribution benefits from the work.

-jef

One week of infrastructure issues

Posted Aug 21, 2008 11:09 UTC (Thu) by motk (subscriber, #51120) [Link] (8 responses)

You really know not of what you speak here, and it shows. I recall the old build system and
source repositories, and they were definitely not something you'd just switch over to the
other side of the DMZ and ring the bell for dinner. It took a lot of time and effort for the
ISIT team to build out infrastructure to bootstrap everything, for Legal to ensure that
nothing lurked in there that could either bite RH or encumber the nascent Fedora project, for
community structures to be formalised - all the hard work you seem to casually discount.

You really are making a habit of sinking the slipper into anything vaguely Red Hat-ish,
despite your protestations otherwise.

One week of infrastructure issues

Posted Aug 21, 2008 21:17 UTC (Thu) by sbergman27 (guest, #10767) [Link] (7 responses)

"""
It took a lot of time and effort for the ISIT team to build out infrastructure to bootstrap
everything, for Legal to ensure that nothing lurked in there that could either bite RH or
encumber the nascent Fedora project,
"""

Sun Microsystems opened the very proprietary Solaris in far less time.  I guess they just have
smarter engineers.

"""
You really are making a habit of sinking the slipper into anything vaguely Red Hat-ish,
"""

I'm very pro-RedHat.  I haven't a single server at any of my client sites which is not either
CentOS or Fedora. I tend to be critical of Fedora where it is deserved. (And yes, I am
migrating the remaining Fedora servers to CentOS for reasons of practicality and pragmatism.)

If my comments about Red Hat seem negative, please imagine how you would act if your company
was as visible, had lots of cash in the bank that some litigious entities might want a slice
of, and had a 3.2 billion dollar market cap to protect.  Red Hat does the right thing whenever
they possibly can because they truly believe, in their corporate heart, that if they stick to
the straight and narrow they will be rewarded.  And that has worked quite well, indeed, for
them, so far.  And I am gratified to behold it.  But they are not stupid.  And I'm quite
certain that they consult legal before every action they make.  And if some things need to
remain secret to mitigate liability...

One week of infrastructure issues

Posted Aug 21, 2008 23:06 UTC (Thu) by jspaleta (subscriber, #50639) [Link] (6 responses)

"Sun Microsystems opened the very proprietary Solaris in far less time. "

Sun opened up its build infrastructure they use to actually build the versions of OpenSolaris
that they provide in binary form? I think you are confused between offering an open source
codebase.. and building an open community infrastructure. They are not the same things and
one is easier to do than the other.

Now I may be wrong, but it seems to me that opensolaris development is gated through Sun
developers.
These pages would certainly suggest it works like that:
http://www.opensolaris.org/os/communities/participation/
http://opensolaris.org/os/bug_reports/request_sponsor/

If you want to contribute fixes or enhancements to OpenSolaris you file bugs or RFE's in the
opensolaris bug tracker and those patches are reviewed by Sun employees for integration. In
fact you actually have to sign over joint copyright ownership to Sun for any contributions you
make..before you are allowed to contribute anything.

You want to contribute, you send in a patch...it gets reviewed, the Sun employee applies it
for you. Actually its more complicated than that.. you actually have to request a Sun employee
sponsor you and give you the go ahead to actually work on a patch.

Besides the whole, must give Sun joint copyright detail and the whole you must request a
sponsor detail, the basic mechanism by which non-Sun employeed contributors can contribute to
OpenSolaris compares directly to community participated in Fedora before the Core/Extras
merge...as far back as RHL 6.x if not prior to that. But I only speak of history that I have
personally lived so I won't speak to anything before RHL 6.x.

If filing bugs and patches in a bug tracker is bar you want open source distributions to
meet... RHL met that a long long time ago. With Fedora as of Fedora 7, community involvement
goes well beyond that, with community members having equal access to actually commit code for
the entire infrastructure which builds the distribution. But we've even moved beyond that
now. Now, there are community members shoulder to shoulder with redhat employees as part of
the infrastructure and release engineering teams. Community members aren't just contributing
the bits... community members are responsible for grinding the bits and cranking out the
sausage every six months. At every level where responsibility can be shared inside Fedora, Red
Hat shares it with community members... because its a partnership. What other software
company is making that level of commitment that Red Hat is making to work with community as a
development partner?

-jef

Outside of Debian, what other linux distribution gives that much access to

One week of infrastructure issues

Posted Aug 22, 2008 0:06 UTC (Fri) by gman (guest, #40493) [Link] (1 responses)

While that process has certainly been true of the past, things are progressing. OpenSolaris has now had a successful migration from Teamware to Mercurial internally, and the next phase is to push that gate outside the firewall. Once this happens, people will be able to commit directly to the kernel (though, much of the review process will still be in place). The bug tracker is still internal (due to confidentiality issues), though we now have a bugzilla instance (defect.opensolaris.org) that many project groups are using.

There is obviously still a significant amount of work to be where we want to be.

One week of infrastructure issues

Posted Aug 22, 2008 2:54 UTC (Fri) by jspaleta (subscriber, #50639) [Link]

Yes opening up infrastructure and and shaking up an existing established workflow inside a
company fence line takes time and non-trivial effort.  Both from an engineering standpoint and
from a legal standpoint.  If doing that work is OpenSolaris's intentions then I applaud the
effort and I look forward to each of the incremental steps as  they take place.  Its hard work
clearing internal company hurdles to make those incremental steps happen.  

Just what ever you do, don't reach for or build a new set of proprietary tools to form the
basis of your community infrastructure and community contribution process.  Don't build your
own launchpad.  If you can avoid that trap, and take advantage of existing open tech or if you
have to build your own open tech... it will pay off when your community wants to help adjust
the community facing tools, because they'll be able to step in and do it for themselves. 

Human code review isn't the issue, someone will always have to be the final say on some issue
and for something like the kernel that person should be someone who is deeply invovled in its
development. Not going to get any arguments from me there.  Cultivating deeply technical
community contribution is a process in itself. It starts with code access, and then it grows
via development process participation, and then in culminates in development process
leadership in new areas...areas your company didn't really think about at the beginning of the
process of opening up. 

-jef

One week of infrastructure issues

Posted Aug 23, 2008 21:05 UTC (Sat) by sbergman27 (guest, #10767) [Link] (3 responses)

"""
Sun opened up its build infrastructure they use to actually build the versions of OpenSolaris that they provide in binary form?
"""

No. I was responding to motk's statement:

"It took a lot of time and effort for the ISIT team to build out infrastructure to bootstrap everything, for Legal to ensure that nothing lurked in there that could either bite RH or encumber the nascent Fedora project"

You might recognize that as the stock argument for why it takes so long to open large bodies of closed source code. In response, I pointed out that Sun managed to open a gigantic and completely closed code base (with a long history), in far less time than it took Fedora/Redhat to open up the supposedly 100% open-source build infrastructure of a 100% open-source project for public use. Different goals, yes. But opening Solaris was at least an order of magnitude more ambitious than what took Fedora so very, very long to accomplish.

It's hard to believe that people whom I know *must* be familiar with the situation are actually trying to deny the obvious and major foot-dragging that occurred for years.

One week of infrastructure issues

Posted Aug 24, 2008 1:04 UTC (Sun) by motk (subscriber, #51120) [Link] (2 responses)

Thanks for implying that I'm a liar. I really appreciate your contributions to the public FOSS discourse.

One week of infrastructure issues

Posted Aug 24, 2008 4:20 UTC (Sun) by sbergman27 (guest, #10767) [Link] (1 responses)

"""
Thanks for implying that I'm a liar.
"""

motk,

What do you mean by that? I disagreed with your argument, but never intended to imply that you were a liar. I do perceive some general desire by some to sweep certain things under the rug at this late date, when the evidence is not still hanging in everyone's faces. But that's a different thing.

All's well that ends well. But that does not mean that all was always well. Best to admit past mistakes and go on, I think...

One week of infrastructure issues

Posted Aug 24, 2008 18:22 UTC (Sun) by rahulsundaram (subscriber, #21946) [Link]

"I pointed out that Sun managed to open a gigantic and completely closed code base (with a long history), in far less time than it took Fedora/Redhat to open up the supposedly 100% open-source build infrastructure of a 100% open-source project for public use."

It wasn't open build infrastructure at all before and what was used internally was completely unusable outside of Red Hat. It had to be written from scratch before it was used for Fedora as I have already indicated earlier. It wasn't 100% open source either. That's just two of the several things you have got wrong in this discussion. This discussion seems completely unrelated to the article at this point even if it was tangentially related earlier and If you got more questions, feel free to email me and I will happy to tell you all about it.