The Success of Gentoo [LWN.net]

November 19, 2003

This article was contributed by Ladislav Bodnar

Gentoo Linux is one of the most remarkable success stories of this year. At the time when more and more new Linux distribution are clamoring for our attention, it is very hard for most of them to attract new users, let alone break into the top league. Yet, Gentoo has done it. It has become one of the most widely used distributions in a very short time.

But don't take our word for it, let some of these figures illustrate the success. Gentoo has one of the most active user forums with over 34,000 registered users who have posted close to 650,000 messages over the last 19 months. On average, nearly 60 new users join the forums every day. Besides forums, Gentoo also provides 36 mailing lists, 10 of which are for non-English speakers. The sizable Gentoo Weekly Newsletter is now translated into 12 languages, including Japanese, Russian and Turkish. Many Gentoo-oriented community web sites sprouted around the globe - from China to Sweden. And the number of Gentoo packages (or ebuilds) in the stable repository has now risen to over 4,700.

What has Gentoo done right to become such a runaway success? Let's look at some of the factors.

Originality. There is no denying that Gentoo has come up with an interesting idea. At the time of increasing availability of fast Internet connections and more powerful CPUs, downloading source codes and compiling them locally is no longer as painful as it once was. Although the concept wasn't new, there is no denying that many Linux users found Gentoo refreshingly different from the mainstream Linux distributions, especially in terms of user control over most aspects of the operating system as well as software optimization.

Persistence. While Gentoo is often seen as a new distribution, its actual development began in early 1999 under the name of Enoch Linux. In "Making the distribution" (part 1, part 2, part 3), Gentoo's creator Daniel Robbins reveals how he started with Linux, how he became a Stampede Linux developer and how he eventually left to start his own distribution. He also mentions his brief encounter with FreeBSD (and its "ports", which later formed the basis of "Portage") and reasons for his return to Linux: "FreeBSD was a peaceful home, but a little too boring, too staid. Linux is where the action was, where major progress was being made. There's no doubt that if you're looking for excitement and innovation, Linux is the place to be." But the fact that Gentoo Linux will soon be 5 years old serves as a reminder that its current fame was preceded by many years of hard, persistent and often thankless work.

Packages. For many users, one of the main attractions of Gentoo is the almost instantaneous availability of new package versions as they are released by their upstream developers. "I just love Portage" is a phrase often heard on public forums. Portage, written in Python, is the Gentoo package management system with many convenient features. The most essential among its commands is "emerge", which has the ability to download, unpack, configure, compile and install a given package in one swoop. Portage also supports OpenBSD-style "fake" installs, safe removal, system profiles, package masking, it has an elaborate dependency system and many other features.

Documentation. Gentoo has some of the best documentation of any Linux distribution. It is written in a style that is easily understandable by all users, irrespective of their skill levels, with detailed, step-by-step instructions and explanations. Color highlighting and additional notes help to keep the text from becoming too tedious and dull. It is fair to say that installing Gentoo is one of the best and most effective ways to learn about Linux internals and system administration, hands-on.

Community. Interacting with the user community is a critical factor contributing to a success or a failure of a Linux distribution, yet sadly, many tend to underestimate it. Not Gentoo. They provide both mailing lists and user forums, as well as a regular weekly newsletter full of useful information about the latest happenings in the development of Gentoo, developer profiles, wealth of practical tips and tricks and other topics of interest. Contrast that to newsletters published by some of the commercial Linux distributions offering little valuable information beyond urging subscribers to join the club, visit the online store or enroll for a certification program.

Although other factors, such as availability of Gentoo for many different platforms or surprise releases of live game CDs have also contributed to a certain degree, the above points illustrate why Gentoo has become one of the most popular and fastest growing Linux distributions ever. An amazing achievement by any standard, but even more so in case of a non-commercial project relying mostly on volunteer effort. No, this success did not come without a considerable amount of hard work, but in the end it was certainly worth it.

(Log in to post comments)

Unbreakable (almost)

Posted Nov 20, 2003 19:15 UTC (Thu) by etwilson (guest, #8459) [Link]

The amazing thing to me is how hard it seems to be to get either portage or the system in general into a broken or wierd state. I've got a seriously bad habit of reconfiguring my home computer constantly; I'm always installing or uninstalling things and I've never managed to break portage and only once managed to (semi-) break the whole system. In contrast, it seems like normal use is enough to corrupt the RPM database on Redhat 9 (the system I have at work).
The one weak link in portage that I see is the problem with updating configuration files, mostly in /etc, after running an update. The utility etc-update" mostly does the right thing but: a. you have to remember to run it; b. its manual editing feature is pretty hard to use. Mis-using etc-update is how I managed to break the system that one time, it was a simple thing to fix but it took me a day to figure it out.

Unbreakable (almost)

Posted Feb 20, 2004 14:33 UTC (Fri) by xentric (guest, #19628) [Link]

Mistakes with etc-update are a very common problem for people new to Gentoo. I have written a script (with ebuild) that makes updating a little less dangerous and a lot easier. The script is called cfg-update. See: GUI alternative for etc-update -> cfg-update It determines if configuration files have been modified after being installed by portage and presents both the original and update in a graphical diff/merge tool of your choice (default is xxdiff). Unmodified files can be updated automatically and you can easily restore the backups if you encounter any problems.

The Success of Gentoo

Posted Nov 22, 2003 7:16 UTC (Sat) by dkite (guest, #4577) [Link]

I forgot to mention the main reason I use Gentoo. It allows you to easily
have two or more versions of KDE available. I run the cvs version, with
all it's quirks, but the rest of the family can run a stable version.

All automatic. Just select from kdm.

I don't know any other distro that allows this without some fussing.

Derek

The Success of Gentoo

Posted Nov 28, 2003 9:13 UTC (Fri) by mdm42 (guest, #10466) [Link]

Yes, portage! Yes, knowing exactly what's going-on on your machine! Yes, the logical configuration of environment variables, rc scripts, etc.! Yes to all those techie things!

But most of all I found a helpful developer community refreshingly free of the egoism and arrogance so frequently visible in other OSS projects.

I'm a Gentoo Holdout

Posted Nov 20, 2003 6:18 UTC (Thu) by torsten (guest, #4137) [Link]

I've had Gentoo recommended to me many a time, but I still haven't tried it. I worry about the Linux community reliance on "package maintainers" and script maintainers. By automating many of the things that are done manually by maintainers, we can reduce bugs, human error, and save thousands of hours of developer time. I am also unconvinced that the portage system solves the dependency problem, rather, it seems to band-aid it.

*Shouldn't we embed dependency information, compile options, and the many other attributes specific to a program [and defined in a portage script] in the program's source code or binary?

*Shouldn't these attributes be automatically generated by gcc or a script during programming-time?

It seems to me most of the information in the portage system is manually maintained, and it could be automated. Am I missing something?

I'm a Gentoo Holdout

Posted Nov 20, 2003 7:58 UTC (Thu) by nix (subscriber, #2304) [Link]

Embedding of dependency information is already done for shared libraries, of course (DT_NEEDED). We could always add something for non-shared-libraries, but they'd be useless to the binary itself and we'd need a call of some kind (or an extension to libbfd) to get access to them. Both seem rather painful.
Secondly, we'd need to store the dependency information elsewhere *anyway*, or how else does it get into the binary?

Embedding of compilation options into binaries has been suggested, but nobody can think of very much use for it. Embedding compilation options into *source code* is something that many people have wished for (mostly to turn on or off erroneous warnings over short blocks of code); but nobody's done the work yet.

I'm a Gentoo Holdout

Posted Nov 20, 2003 8:01 UTC (Thu) by Klavs (subscriber, #10563) [Link]

>*Shouldn't we embed dependency information, compile options, and the many >other attributes specific to a program [and defined in a portage script] in >the program's source code or binary?
Sure. If you think you are going to get everyone who develops software to do this right - then feel free :)
Besides that part is pretty easy to do in the portage ebuild scripts.
The harder part is those that do not follow the ./configure && make && make install way of doing things to the letter :)
And even that is best done in portage ebuilds. And I say this, having written RPM .spec files for many years.

>*Shouldn't these attributes be automatically generated by gcc or a script >during programming-time?
feel free to join the gcc-community and tell them how it can be done.

Contribute your ideas (see the Gentoo GLEP proposal for bigger ideas) that's what it's all about.

Just waiting out till someone else does it, might just mean it never gets done ;)

I'm a Gentoo Holdout

Posted Nov 20, 2003 8:52 UTC (Thu) by omez (guest, #6904) [Link]

Embeding the dependency information and whatnot in the original source doesn't obviate the need for someone to manually intervene at some point. You'll still have the same number of bugs, errors, and time spent combating them. An automated tool would involve all 4,700 projects (in just the Gentoo case) adopting a single package system. And just how are you going to convince every distro of every Unix and Unix-like system to follow suit? There are projects that don't even use autoconf, much less favor a particular packaging system. It couldn't compete with the distro-agnostic, write once, run anywhere approach of good, portable code.

The ebuild maintainers are very responsive. Rather than consult the project's roadmap to determine when a fix can go out, they just do it. They put the "early and often" shine on the most glacial of development styles.

The Success of Gentoo

Posted Nov 20, 2003 10:05 UTC (Thu) by ulajamb (subscriber, #16976) [Link]

This is one of the best side effects of portage I can think of. Updating, restoring, and adding new packages is very easy when I need it to be but also very configurable when I need it to be. The term 'meta-distribution' has not been brought up here, that is how some speak of Gentoo because of the ability to control so much.

First, one thing left out of the article here is that Portage does not yet support reverse dependencies. That is, when I remove package A, the packages B,C,etc. that were installed soley to allow me to install package A are not automatically removed. That is ok for me, because I keep track of things like that. Servers don't change much (well, mine don't), so in my eyes this is only a problem for desktop users that are trying out a lot of packages. My advice: when you install something that requires more than a couple packages (like installing X or Gnome), just do "emerge __ -p" (p means pretend) and copy/paste the list of packages to a file. All in all, this really only wastes me a few minutes (setting up xargs to unmerge the packages in that file).

Otherwise portage is great, you can choose default 'USE' flags to keep unnecessary features from being compiled into your downloads. You can tweak this for a particular package very quickly at install time.

A key point - I will repeat my sentence above. When I want something quickly it is there. When I want to tweak, secure and customize there are a lot of ways to do this past conf files (you can force portage to use your own ebuild for instance).

Also, one thing often left out of discussions of Portage is that there are several large packages that have a pre-compiled choice (like who needs to optimize openoffice?) . Also, there is something called the GRP cd for several architectures that has a lot of packages precompiled for those computers. I'm not even using it to have 'optimized' packages, you can get into those arguments with some people who bash Gentoo for what they see as its glaring folly. There are speed contests and too many arguments to keep track of. In the end, O2 and a couple flags for gcc are great for me. If I ever needed to optimize something far past this it would be a special case and I'd look into it for that application.

There are many other small things I appreciate about the whole distribution (not just portage).

------

Mainly, I like the control it offers. This is a high priority for the maintainers.

Another key point for me is the good will, collaboration, and knowledge found in the user forums. It's always the first place I look for a solution to particular problems -- it's usually already been discussed.

If you have a fast internet connection especially, why not try it out on a small partition?

(I've been using Gentoo for about 8 months now, and do NOT have much experience with other distributions, so you might take that into account when considering my opinions)

The Success of Gentoo

Posted Dec 6, 2003 17:54 UTC (Sat) by irf (guest, #17606) [Link]

ulajamb, actually reverse dependency is there, you may have to emerge
gentoolkit and do:
revdep-rebuild
and it does it for you
do revdep-rebuild --pretend
and it will let you know what needs to be re-emerged without actubally
re-emerging the packages, try it
HTH

The Success of Gentoo

Posted Dec 2, 2003 17:18 UTC (Tue) by bc90021 (guest, #17401) [Link]

Ah, but the beauty of Gentoo is that while your portage tree may have grown, that doesn't mean that you have to take advantage of it.

It's like buying a car... it used to be that you could buy a car with no AC, no radio, and no power-anything. Nowadays, those things are just standard. Why? Because they're so popular, and don't detract in any way. If you want extras (navigation, DVD, etc.) those are available too. The concept is choice, and having a "bigger minimum" doesn't detract from that at all.

Gentoo is that car, only today you can still get it without AC. (Try getting that from Ford or Chevy!) If you want minimal, install minimal, and run with it. If you prefer otherwise, that's your choice too. Your portage tree being bigger only means that "emerge sync" is going to take a tiny bit longer, but there may come a time when you want some obscure piece of software, and that larger portage tree might not seem so bad... ;)

Community

Posted Nov 21, 2003 1:20 UTC (Fri) by mmealman (guest, #9223) [Link]

I think a big part of Gentoo's growth has been because of the user community.

To communicate with other users you have mailing lists, irc just like everyone else AND the web forums. I have no idea why other major distributions haven't rallied around web forums. I can't count how many times a quick search through the Gentoo forum archive of posts has turned up an answer to any question I might have.

I'd love to see other distributions spend as much time building their community as they do working on the distribution. Fedora could easily take off just as well as Gentoo has if they had a decent system of communication in place for their users.

Mailing lists and IRC alone just don't cut it anymore. You need a casual place to shoot the breeze with other users, post screen shots and generate anticipation and excitement about your distribution.

Bug testing

Posted Dec 3, 2003 0:38 UTC (Wed) by butters (guest, #17430) [Link]

A major side effect of Portage and how easy it is to stay on the cutting edge of OSS developments is that there are now MANY more linux users with convenient access to beta and candidate releases. This makes the process of identifying bugs in new software quicker, especially ones that only effect certain hardware configurations.

I bet that a completely disproportionate number of people running a 2.6.0-testN kernel right now are running Gentoo. Same thing goes for GNOME 2.5 and many other testing packages.

Gentoo was developed for developers that constantly update their software as changes are made. Gentoo has turned a lot of casual (and often clever) linux users into developers, or at least into masses of bug wranglers. Portage is a very powerful tool for open source software testing and "validation."

Community

Posted Dec 6, 2003 18:13 UTC (Sat) by irf (guest, #17606) [Link]

mmealman, I agree with you, gentoo's owes it growth because of the
wonderfull support of the user's community, especially the gentoo forums.
I have been on gentoo for over a year now, and never had to post a
problem which I had on the forum (until today), chances are that someone
did that already and the problem has been resolved! now that is a hell of a
community!

Gentoo rc scripts

Posted Dec 3, 2003 16:16 UTC (Wed) by stalione (guest, #17457) [Link]

The article didn't mention that awesome job Gentoo developers did with the rc scripts. In Gentoo by default there are three run levels, boot, nonetwork and default and it also gives you the flexibility to create your own run levels (see Gentoo Linux 1.0 Init System). The other nice thing about their rc script system is the use of keywords like need, depend, use that lets you specify dependencies. No longer do you have to worry about assigning rc-scripts a particular number to make sure it starts after/before some other script. Simply specify something like "need net" to make sure the script only starts if network is present.

I have been using Linux for a very long time and I have never come across a more polite and helpful community than the Gentoo user community. What good is it to a newbie if you answer their questions with "RTFM"?

Also this article did not mention that there are binary packages available to use by specifying the -k when emerging (only valid if the binary package exists).

"--usepkg (-k) Tells emerge to use binary packages (from $PKGDIR) if they are available, thus possibly avoiding some time-consum-ing compiles. This option is useful for CD installs; you can export PKGDIR=/mnt/cdrom/packages and then use thisoption to have emerge "pull" binary packages from the CD in order to satisfy dependencies."