Leading items

EGLIBC: Not a fork, but a glibc distribution

The GNU C library (or glibc) is a fundamental component of the Linux operating system. It provides much of the user-space interface to the kernel as well as a sizable portion of the utility routines that are used by virtually all Linux applications. A variant of glibc—known as Embedded glibc or EGLIBC—is not very well known outside of the embedded space, but that looks to be changing with the announcement that Debian will switch from glibc to EGLIBC.

The project is quick to point out that it is not a fork of glibc as it aims for compatibility—both source and binary—with glibc. Instead, EGLIBC is like a distribution of glibc, adding patches to fix problems as well as making it easier to use for embedded applications. It also adds support for embedded architectures that are not part of the focus of core glibc development.

The project—started in 2006—is being spearheaded by CodeSourcery, which provides compilers and other development tools based on GCC to its customers. Chief Sourceror Mark Mitchell says that CodeSourcery provides administrative support for the project and "leadership, but not control". Its customers are embedded Linux distributors along with hardware manufacturers and EGLIBC arose from those customers looking for smaller C library solutions.

Mitchell said those customers were looking at uClibc as an alternative to glibc, but using uClibc has some disadvantages because it takes "real work getting applications that work with glibc to work with uClibc". Because of the universal use of glibc, "the Linux that you run on a supercomputer and the Linux that you run on a cell phone are very similar". One of the key advantages of uClibc, though, is its customizability, so that embedded developers can turn off features—Mitchell's classic example is NIS—that are not needed for the product.

That led to adding "option groups" to EGLIBC, which can be enabled or disabled, more-or-less independently to reduce the size of the library. There are roughly 30 option groups currently defined, but the project hopes that new contributors will add new ones. By using the option groups, one can disable things like networking (or just IPv6 networking), math libraries, Sun RPC, Streams, various internationalization/localization options, and more.

Working with glibc maintainer Ulrich Drepper can be difficult due, at least in part, to his blunt manner. While Mitchell is not one to contrast the EGLIBC development community with that of glibc, it is apparent that EGLIBC is trying to be a more inclusive upstream. Others, including Debian's Aurélien Jarno, who announced the switch to EGLIBC, are less reticent, pointing to the difficulty of working with Drepper as a main advantage of EGLIBC.

For his part, Mitchell would like to see a "vibrant open-source community" spring up around EGLIBC, noting that "we welcome all comers". While the project is funded by a consortium of companies: Freescale Semiconductor, MIPS Technologies, MontaVista, and Wind River; one does not need to work for those companies in order to contribute. Nor do the patches need to be focused only on embedded concerns; bug fixes and features that are not accepted by glibc will be considered. Mitchell said that EGLIBC is looking forward to contributions from anyone and, in particular, "are hopeful that Debian will contribute" to the project.

So that any contributions can be rolled back into glibc, EGLIBC requires contributors to assign their copyright to the Free Software Foundation as with a glibc patch. The intent is that the EGLIBC code never stray very far from glibc; in fact, changes to glibc are merged back into EGLIBC frequently. Core glibc development is "focused on x86 and other server-class hardware", Mitchell said, but if that changes, EGLIBC could be merged back into the upstream project. "We'd be happy to see FSF glibc pick up anything we've done, even to the point of obviating EGLIBC", he said.

Jarno is clearly frustrated with how glibc development is done, quoting from the EGLIBC mission statement as part of the announcement: "Encourage cooperation, communication, civility, and respect among developers." But that is not the only advantage to EGLIBC, he lists a number of others including the configurability, a better test suite, and better stable branch handling that makes important bug fixes available for earlier versions.

In a follow up posting to debian-devel, Jarno lists a few more reasons along with answers to some of the questions he has heard. For Debian in particular, supporting many different architectures, including multiple embedded architectures, is very important. He notes that EGLIBC is essentially a drop-in replacement for glibc in keeping with that project's compatibility goals. The common request to add strlcat() and strlcpy()—BSD string handling functions that avoid buffer overruns—will likely not be satisfied for just those compatibility reasons.

EGLIBC clearly offers some things that the standard glibc lacks—not just for embedded developers. The focus on being inclusive will likely expand the pool of C library hackers, which is something that the current glibc development community is not actively encouraging. With luck, it will also act as a test bed for fixes and features—especially as it gets more exposure and testing via Debian—that may lead to a better glibc down the road. It will be interesting to see if other distributions take the same step that Debian has, and what that might mean for glibc in the future.

Comments (42 posted)

Transmageddon and Arista pursue simple transcoding

Christian Schaller's Transmageddon and Daniel Taylor's Arista are both easy-to-use video file conversion tools for GNOME, but they share more than just a vision for simple file transcoding. Rather than competing head-on (or attempting a merge), the two developers are collaborating in the middle; sharing information and utilizing the similar aims of their projects to strengthen the underlying GStreamer multimedia framework on which both code bases depend.

Schaller is a longtime GStreamer contributor; he started Transmageddon in March of 2009 for several reasons. First, as a personal exercise in developing in Python, but also as an attempt to find real-world areas of improvement for application developers using GStreamer, and to scratch a personal itch — the lack of an easy-to-use, GStreamer-based video transcoder. Said itch also prompted Taylor to start work on Arista, as a personal project in 2008, with its first public release in April of 2009. He discovered Transmageddon through Schaller's blog, and the two began discussing how best to proceed.

A variety of motives

That two developers independently decided to write simple GUI transcoding applications should come as no surprise. Video transcoding is increasingly in demand as more and more portable devices have the graphics hardware to support it and digital content distribution skyrockets. Command line encoders like FFmpeg and MEncoder can perform almost any conversion, but require the user to know exactly which options to enable, and, in most cases, demand that every codec switch be specified or else they fail to produce a usable result. That leads to obtuse (and extremely long) commands that are difficult to get right and even more difficult to troubleshoot. Several graphical transcoding applications have appeared for Linux desktops in recent years — Thoggen, OGMRip, K9Copy, AcidRip, dvd::rip, HandBrake, and WinFF, to name a few — but as both Taylor and Schaller commented, all have their shortcomings.

Some, like Thoggen, OGMRip, and Oggconvert, are limited to specific output codecs. Others, like dvd::rip, attempt to offer every option under the sun, making them difficult for casual users to use. "There are dozens of GUIs that are essentially the same confusing options with a nice graphical label. To me that doesn't really help," Taylor observed. "The point of Arista is not to be everything to everyone, just to be a simple way to transcode your media for various devices. That's it."

In addition, many of the existing transcoders are built as stand-alone front-ends, and offer little in the way of desktop integration. "Most of the other GUIs I've seen directly use FFmpeg (libavformat, libavcodec, etc)," said Taylor. "I wanted to make use of GNOME technologies to make a real GNOME app, which means using GTK+, GStreamer, Gconf, GIO, etc. Arista should fit into the desktop." Taylor said he has a few features in mind for future releases (such as subtitle and DVD menu support), but also hopes to add a D-Bus interface that will enable other applications to control Arista.

For Schaller, building a GUI transcoder is also a way to field test GStreamer. He mentioned the possibility in a 2008 blog post, speculating that such a small-scale application could be used to "test and make sure our muxers and encoders keep working" better than a more complex tool like a video player or editor. So far, he said, a few bugs may have been discovered, but the more important benefits are from testing GStreamer features that have not been used before — such as property presets, which allow the application to set bundles of individual encoder settings by name (either for encoder profiles like H.264 Baseline, or to target specific hardware devices).

The benefits extend beyond the existing code to other applications, Schaller added:

Collaboration and divergence

After Schaller and Taylor touched base, they decided to stay in close touch and share code whenever possible, but to take their respective applications on slightly different paths. Taylor would take the conservative, end-user-oriented approach, making sure that Arista would always run on the current release of Ubuntu. Schaller would try to more closely track the development version of GStreamer, including reliance on GStreamer features not yet available downstream. That way, there will always be a transcoder available for and usable by "non-techies" in Arista, and Transmageddon can serve as a test bed for new features.

The cross-project collaboration is already reaping rewards. Schaller says that he has incorporated Arista's existing preset support into Transmageddon, and is extending it using some bleeding-edge changes from the development version of GStreamer. Once the Linux distributions are shipping the next update of GStreamer, the plan is to roll Transmageddon's improved preset support back into Arista.

The projects are also helping GStreamer. They are accumulating a collection of presets that will be valuable to other applications, have uncovered bugs in the capabilities system, and exposed that GStreamer needs a better encoder and decoder for the Adaptive Multi-Rate (AMR) audio format.

Schaller said he is also planning on turning his notes on the development process into a guide for others:

Test drive

You can download both applications and try them out to compare. Transmageddon is available from Schaller's web site as a source code package and as an RPM package. The latest release is 0.9, from May 3, 2009. It does require Python and a development branch of GStreamer, though, which may place it out of reach for most users.

Arista is available for download from the project's Launchpad site. The most recent version is 0.9.1, also from May 3. A source code package is provided, as is an ArchLinux build through the ArchLinux User-community Repository (AUR), and those running Ubuntu 9.04 can add Taylor's Personal Package Archive (PPA) to their Apt repositories list and install the application through the package manager. 0.9.1 also requires Python and GStreamer, but only GStreamer 0.10, which is widely available.

Arista allows the user to select source videos in any encoding and file type readable by GStreamer (including DVD content), and select an output format — including presets configured for a range of portable hardware devices. Beyond that, there is little to the interface but a progress bar estimating the time remaining in the transcoding job and a two-frame-per-second preview window useful for monitoring the output.

Other transcoders offer far more options, but at the cost of far more complexity. Users must often know the audio and video codecs, output width and height, frame rate or bit rate, and many other options. Arista and Transmageddon make the job drastically simpler merely by abstracting away all of those details in device presets — the correct settings are still chosen, but the confusion and chance for human error are not. It will be interesting to watch the projects evolve in the months ahead; the notion of a simple to use video transcoder is appealing, but even more appealing is what can happen when that functionality is available to other desktop applications through D-Bus.

Comments (15 posted)

GNOME debates Javascript bindings

Which Javascript binding should the GNOME desktop support? This is a question that the GNOME desktop-devel was debating last week. Since GNOME-Shell, the foundation of next year's GNOME 3.0, is built on Javascript, the answer is a major policy decision. However, the difficulties of making the decision are compounded because the contenders, Seed and Gjs, are both in rapid development and their final form is difficult to predict. However, that does not stop the push for a decision from the Seed community.

Both Seed and Gjs are GNOME projects. They are interpreters that allow users to write code in Javascript and have the core GNOME code use it. One of the major differences between them is the flavor of Javascript that they use, with Seed designed to work with WebKit, a Javascript engine favored by Apple and KDE, and Gjs with SpiderMonkey,a Javascript engine designed for Mozilla's applications. Since GNOME-Shell is written in Javascript (currently with Gjs) the choice of engines is a decision with far-reaching consequences. Also at stake is the prospect of tying future GNOME development to a particular dialect of Javascript whose development and adoption is still uncertain.

What makes the decision particularly hard is that, with Seed at release 0.7 and Gjs at 0.2, both are likely to change considerably before their final releases, so judging the suitability of one over the other is next to impossible. All the same, an effort at objective comparison was written by Owen Taylor, the GNOME-Shell maintainer.

Taylor notes that only Gjs currently supports "let" statements, which create variables scoped to the local block, and "const," which is used to declare a constant variable. He also refers to the fact that switching to Seed would currently mean a loss of destructuring assignment, a more flexible, multiple-assignment syntax. But he is largely unconcerned about these losses, viewing them as only temporary. He is more concerned about the choice of the widely used SpiderMonkey as opposed to the more untried JavascriptCore module behind WebKit, although he notes that Epiphany, the default GNOME browser, and Yelp, the GNOME help system, already use WebKit. Without giving particulars, Taylor also characterizes Gjs as having a limited set of well-implemented features, as opposed to Seed, which has more features. In fact, Gjs seems to exist largely to serve the needs of GNOME-Shell as they arise, while Seed is more active and independent project.

At this stage, he is not concerned with the speed of either, reasoning that it will change in coming months, but he does express concern at garbage collection — the reclamation of memory by the removal of unused objects — arguing that both SpiderMonkey's and WebKit's garbage collection is more geared to web pages than desktops. In the end, Taylor seems to be relatively indifferent to which one is used by GNOME so long as it has the tools that GNOME-Shell will require. In other words, the choice is too uncertain for even those most closely affected to have strong opinions on it.

The most that Taylor can provide is a general criterion for making a choice: "Javascript standardization is highly politicized and affected by concerns like what Microsoft can implement in IE and in general by considerations of cross-browser compatibility," Taylor wrote. "We should use whatever dialect of Javascript is supported by our engine that makes our life better."

The push for a choice

Nobody involved with Gjs has been active in this discussion, while those involved with Seed have been raising the possibility of choosing one over the other for six months. Last November, Frederick Peters, in the GNOME Commit-Digest, summarized some differences between Seed and Gjs as described by Robert Carr, Seed's maintainer. According to Peters, Carr suggested that, for the time being, "it's best for both projects to continue development at this point."

By January, as plans for GNOME 2.26's March release were being finalized, Carr was ready to propose that GNOME standardize on Seed. The proposal sparked a discussion about the relative merits of Seed and Gjs as they existed then. But the majority opinion was that it was still too early to make a choice, and the proposal was not accepted.

Four releases of Seed later, with plans being discussed for the release of GNOME 2.28 , the intended beta for GNOME 3.0 that is due for release in November, Carr renewed his proposal in May. He pointed out that Seed would require no new external dependencies, since WebKit/GTK+ and GObject Introspection, with which it interacts, are already scheduled to be included in the upcoming release. Furthermore, Seed already uses standard parts of GNOME, such as gnome-git and Bugzilla. Carr also cited the flexibility of using Seed, arguing that "at this point, most Gjs code could be prettily easily ported to Seed" while porting the other way "might be a bit more difficult" and suggesting that "it would be possible to port GNOME-Shell to Seed with no more than a few days' work." Similarly, he suggested that writing an extension to support Seed would be "around a day's work" for most applications. In passing, he also raised the possibility of a common module for both Gjs and Seed, which would preserve flexibility.

In response to a direct question from Hubert Figuiere about "what makes Seed more suitable than Gjs?" Carr admitted that, "For 2.28, it may make sense to have both Gjs and Seed as modules, and try and keep code somewhat compatible. It's still not entirely clear which Javascript engine is going to end up being better long term, so we might not want to completely commit to one yet."

Still, Carr favored Seed for a number of technical reasons. He argued that Seed's use of WebKit and GObject was more in keeping with existing tendencies in GNOME, and the Seed C API is far more intuitive than the SpiderMonkey API used by Gjs. Moreover, he described Seed as having more example code and documentation, which would make it easier to use.

What would happen to Gjs if Carr's proposal is accepted is uncertain. However, with Seed officially endorsed, Gjs would undoubtedly become a side line within GNOME if it continued at all. So far as most developers are concerned, it would probably survive largely in the code that Seed borrowed from it.

The discussion

The initial response to Carr's proposal was that having two Javascript engines was unnecessary. However, most people on the mailing list gradually seemed to agree with Vincent Untz, who argued that "we need both engines and we need them in our platform sooner rather than later," on the grounds that which one was preferable was still uncertain.

After a few remarks, the discussion turned to a question of whether GNOME-Shell's developers were prepared to use Seed, and what effort that decision would require. Carr replied that he was prepared to write a patch that would allow Gnome-Shell to use Seed. Others, though, such as Xan Lopez suggested that he was underestimating the size of the task. "Do you have any idea of how big a task adding 'let' [statements] support to JSC [Javascript Core] would be?" Lopez asked, referring to a one of Seed's greatest lacks, and implying that the task would be a big one. To this question, Carr replied that he was looking into the issue, and would report back when he had more information.

After discussing the idea of adding let statements to Javascript with the JSC project, Xan Lopez disagreed with Taylor on the grounds that the upstream JSC team was opposed to non-standard extensions. Instead, he urged that GNOME should "just stick to a [Javascript] defined in some standard widely used for all GNOME code, in order to avoid future headaches."

Carr offered to alleviate concerns by maintaining a Seed version of GNOME-Shell separately from the main branch for the next two releases. But, at this point, the unspoken consensus seems to have been that a decision could not be made.

Instead, talk turned to how to implement a common module for both Seed and Gjs. In a long post, Havoc Pennington strongly advocated this position. Pennington's "practical path" was to have both engines use the same modules and test suite, and to add support for let statements. As he saw it, the common module would also need to support disabling features that one or the other didn't support. This path was extra work, Pennington acknowledged, but he thought that the common module "could be the right decision even if GNOME includes only one of the runtimes, just to get more people hacking on the suite of [Javascript] modules."

For now, then, GNOME seems to have chosen to develop both modules together as much as possible, and to delay a final decision yet again. This approach may seem untidy and unsatisfying to many, but, it is hard to see what other decision could be made. Both the engines and Javascript itself are too unsettled for a clear choice to be made. Moreover, the undoubted pressure to advance GNOME 3.0 means that the time to make such decisions is minimal. By compromising, GNOME has allowed itself to progress while keeping as much flexibility as possible.

Comments (28 posted)