Two approaches to Open gadgets

One device which has begun to attract attention is the GP2X, which will be shipping soon. This device has a superficial resemblance to the Sony PSP; it has a central screen with a set of buttons on the right and a joystick-button on the left. Unlike the PSP, however, it is an open device. The specifications are available (this Wikipedia page has the most comprehensive information), and the device runs Linux. It is clearly meant to be hacked on, and it could be the source of no end of interesting applications.

On the other hand, some details are scarce, and there appears to be no place to download the Linux distribution used on the device. An earlier version of the product page contains the ominous words "copyright protection by certified DRM." The device will remain vaporware for a little longer; once it is in circulation, the world will see if it is truly a Linux-friendly (as opposed to simply Linux-using) gadget or not.

A more interesting project, one which could certainly benefit from more development help, is Rockbox. The Rockbox developers are creating a free system for portable music players; the primary target is the Archos product line, but work is proceeding on iRiver 1xx and 3xx players as well. This project (which will be releasing version 2.5 "soon") is a demonstration of why free software is such a nice thing to have on these devices.

A partial list of advantages to the Rockbox software would include:

• A much wider range of translations than the original manufacturer provides.

• Numerous features for blind users, including a voice mode which reads out menu entries as the user moves over them.

• Gapless playback.

• A wider range of codecs, enabling the use of audio formats not supported by the manufacturer.

• A user-configurable "while playing" screen, enabling scarce display space to be used for exactly what the user wishes to see.

• A plugin architecture for adding new features. The plugin list appears heavily biased toward games, but it also includes image file viewers, clocks and stopwatches, and more.

• On the iRiver: faster booting and the ability to boot into USB storage mode when the filesystem is corrupted. So filesystem problems which would turn a stock iRiver into a brick are recoverable with Rockbox.

The list goes on, but the point should be clear: Rockbox allows the owner of a music player to do away with no end of annoyances, add new features, and generally get the most out of a nice piece of hardware. The freedom to make changes like this is what drew many of us to free software in the first place.

The sad thing is that the Rockbox developers have had to put considerable work into figuring out how the hardware works and developing firmware patches. Had the vendors simply opened up their hardware in the first place, that effort could have gone into making the software better. This situation should eventually change: Rockbox already looks better than what a number of manufacturers are installing onto their players. As Rockbox develops and that gap widens, there will come a time when some manufacturer will realize that the ability to run Rockbox will be a positive selling point for a media player. Then, maybe, we'll have a truly open gadget to play with.

Comments (14 posted)

Reiser4 and kernel inclusion

Reiser4 is an interesting filesystem. It comes with claims of improved speed and space utilization; those are welcome, but they are beside the real point. Reiser4 includes a "wandering log" mechanism which provides journaling capability without the need for a separate journal. The ability to perform multi-step transactions is built into the filesystem, though not yet completely exposed to user space. Multi-stream files (including file-like access to file metadata) are supported, though this feature is turned off for the moment as well. A flexible plugin architecture makes it easy to add new features (such as encryption, compression, different formats, etc.) to the filesystem. And so on.

Hans Reiser and his developers at Namesys are trying to change how people work with filesystems - and with computers as a whole. The underlying vision is one where the filesystem implements the entire namespace used by the system; everything truly is a file. In the Reiser view of the future, applications like relational database managers need not exist; such tasks will be handled in the filesystem itself.

What it comes down to is that reiser4 represents some of the most innovative work being done with filesystems for Linux - or for any other system. So one might well wonder why inclusion is proving to be such a challenge. Some of the reasons are straightforward: there were genuine issues with the code. The "files as directories" capability opened the door to trivial, user-initiated kernel deadlocks - a feature which can absolutely ruin those performance benchmarks. The multiplexed sys_reiser4() system call - to be used for managing transactions, among other things - is just the sort of call that the Linux developers are trying to get away from (and its use of an in-kernel command interpreter did not help). A number of other things needed to be fixed; the Namesys hackers have been working through the list, but a few items remain.

The real point, however, is that getting code into the kernel is an increasingly hard thing to do. In the early days of Linux, almost any code which made things work or added new features was welcome - we needed it all. In more recent times, it is often hard to argue that new features are truly needed, especially at the kernel level. So each new addition must be weighed against the costs that will be incurred when it is added.

The result is that the standards for new kernel code have gone up considerably over the years. Reiser4 has run into these standards, and objections have been raised to code which duplicates features found elsewhere in the kernel, is hard to read, violates the layering rules, has unclear locking schemes, or which uses obsolete interfaces. The point is that, in order to be merged, the reiser4 code must be understandable by people other than its original developers. As Alan Cox put it:

"What happened later on" is that the reiser3 developers moved on to reiser4; not only did they stop maintaining the code, but they actively opposed updates made to the code by other developers. At this point, reiser3 is almost entirely maintained by non-Namesys developers. In the future, the same thing may well happen with reiser4.

The crux is this: the Linux kernel has been around for 14 years, and is expected to last for quite a few more. The kernel hackers understand that, if they are insufficiently careful about what they merge now, they will have a big mess to deal with five years down the road. Many developers, working in all areas of the kernel, have had seemingly good code turned away because the development community was worried about maintaining the code in the future. The process is most frustrating for the people involved, but it is absolutely essential if we want to continue to use Linux into the future. To many, the difficulties encountered by reiser4 (and FUSE, and the realtime LSM, and class-based kernel resource management, and ...) represent the kernel development process at its worst, but the opposite is true.

That said, reiser4 has had a harder time and more microscopes applied to its code than many other developments. Mr. Reiser's approach to community relations, which strikes many as occasionally belligerent and paranoiac, certainly has not helped here. This issue has been discussed often, but there is another issue which deserves airing: some people are clearly uncomfortable with the vision behind the ongoing Reiser filesystem effort. It doesn't quite look like the Unix systems we grew up with. Linux is not an experimental or research-oriented system, so the inclusion of radically different ideas of how the system should work must be carefully considered. But Linux must also evolve, or risk irrelevance in the future. Hans Reiser's efforts to push that evolution are a good thing; the community discourages such work at its peril. So perhaps the time has come to let reiser4 in; the wider community can then get to work dealing with any remaining issues.

Comments (53 posted)

The relevance of the Linux Standard Base

More than seven years later, version 3.0 of the LSB specification has been released. With this release, the LSB requires the system to have a relatively new compiler and toolchain (gcc-3.4 or newer), adds some libraries and interfaces, and cleans up some obsolete interfaces. There are two core variants of the LSB specification, depending on whether the target system is expected to have graphics capability or not. Sample implementations are available for eight different architectures. The release notes have more details, for those who are curious.

In many circles, however, the LSB 3.0 release is being greeted with a big yawn. Most Linux users probably have a hard time seeing how the LSB benefits them. Ulrich Drepper, who, as maintainer of glibc is faced with a wide range of LSB compliance issues, has recently claimed that the LSB lacks value and should be dropped. It is a rare Linux user who chooses a distribution or application based on its adherence to the LSB.

The stated purpose of the LSB was to encourage the availability of applications - both free and proprietary - for Linux. So it is telling that, among the available Linux applications, very few claim to be targeted at LSB-compliant systems. In fact, your editor found just one beyond the special versions of free applications in the LSB's own application battery: it's Appgen's MyBooks, which works on bleeding-edge LSB-compliant systems like SUSE 8.1 and SCO OpenLinux. In general, application vendors are not targeting the LSB; they are, instead, certifying specific distributions.

Not everybody feels a need for wide availability of proprietary applications for Linux. But, for those who do, the certification of individual distributions is exactly the sort of situation that the LSB was created to prevent. From that standpoint, the LSB would appear to have failed.

That said, the LSB effort has certainly had a positive effect in bringing Linux distributions closer, and in raising awareness among distributors of how their offerings diverge from the standard. Even if an application is not specifically aimed at LSB compliance, the fact that it probably just works on non-certified systems is, at least in part, attributable to the LSB. There is value in separating the core part of a distribution (that which, in some sense, makes it "Linux") from the additional features and services distributors throw in to add value to their offerings. The LSB helps to bring that separation about.

From the moment that Linux started to attract attention outside of the development community, detractors have grumbled that it would fragment in the same way that Unix did. Yet, despite the existence of hundreds of distributions, several of which are widely used, this fragmentation has not happened. Linux applications remain portable, and, just as importantly, switching from one distribution to another is (usually) nearly painless. Linux is Linux, regardless of the distributor. The reasons for this state of affairs include the use of a (more or less) common kernel and application base, and the fact that free licensing makes it easy for good ideas to move quickly from one distribution to another. But there is a place for standards as well. As long as the LSB continues to codify current expectations of what a Linux system should be, it will help to keep the Linux universe coherent.

Comments (5 posted)

Another look at response times

Vulnerability	Distributor
	Debian	Fedora	Gentoo	Red Hat	SUSE	Ubuntu
Apache mod_ssl	14	9	21	11	14	12
clamav	22	3	3	n/a	5	--
evolution	--	1	13	19	6	1
fetchmail	22	0	4	4	7	5
PCRE	13	4	14	18	16	3
PHP XML-RPC	9	4	5	6	7	4
PHP XML-RPC 2	18	10	9	4	15	5
ProFTPd	35	--	4	n/a	n/a	n/a
vim modeline	--	16	n/a?	28	n/a?	1

In the above table, numbers which are underlined reflect alerts issued after the previous version. Those which are, instead, bold are corrections for erroneous entries published two weeks ago.

As one can see, a number of corrections were required. One might conclude from this that your editor was being even more clueless than usual when compiling the previous version of the table. One would probably be right, but there is a little more to it than that. It turns out that putting together a table like this is a hard thing to do.

The previous version stated that Fedora had not issued an advisory for clamav. That is, in fact, true; no advisory ever came out. The clamav package in Fedora Extras was quietly replaced, however, shortly after the vulnerability was disclosed. In the presence of silent fixes, it is hard for users to know if they are vulnerable or not; this is doubly true in cases where security fixes are backported to previous releases of the affected package. Fedora Extras does not do backporting, but it still requires an alert administrator to know that, while clamav has been fixed, ProFTPd in Extras remains vulnerable.

Speaking of ProFTPd, your editor had seen that package in a SUSE distribution he had at hand, and assumed it was still distributed. That turns out not to be the case.

Both SUSE and Gentoo claim to not be affected by the vim modeline vulnerability because they ship versions with the modeline feature turned off by default. Turning off a possibly insecure feature is a good thing to do; it reflects a concern by the distributor for the security of its users. Some of those users, however, will certainly turn the feature back on. Others will be concerned by the fact that they are running software with a known, unpatched vulnerability, even if that vulnerability does not directly affect them. In such cases, it would make sense for the distributor to, at a minimum, issue an advisory explaining the situation. Putting out a fix would be better.

Other corrections above reflect simple screwups on your editor's part. Sorry.

The corrected table still shows some real patterns in the relative response times for security updates. There is value in this information. As time permits, LWN will be making changes to its security database to make the generation of this sort of table an easier and more accurate process. But a task which, in the presence of nice things like CVE numbers, should be relatively straightforward is likely to require a fair amount of time (and iterations) for the foreseeable future.

Comments (9 posted)

Mozilla Linux Command Line URL Parsing Security Flaw Reported (MozillaZine)

Comments (6 posted)

clamav: multiple vulnerabilities

Comments (none posted)

Mailutils: format string vulnerability in imap4d

Comments (none posted)

masqmail: input sanitizing and symlink vulnerabilities

Comments (none posted)

Py2Play: remote execution of arbitrary Python code

Comments (none posted)

turqstat: buffer overflow

Comments (none posted)

Zebedee: Denial of Service vulnerability

Comments (none posted)

Kernel release status

The current 2.6 prepatch is 2.6.14-rc2, released by Linus on September 19. "Not a whole lot o' excitement, ye scurvy dogs, but it has t' ALSA, LSM, audit and watchdog merges that be missed from -rc1, and a merge series with Andrew." Some specific additions which came in after -rc1 include a new virtual filesystem for security modules, some DCCP additions, a number of audit subsystem patches, some netfilter enhancements, and an ALSA update. See the long-format changelog for the details.

Linus's git repository currently contains a SCSI update, some netfilter patches, an InfiniBand update, and various fixes.

The current -mm tree is 2.6.14-rc1-mm1. Recent changes to -mm include per-task write throttling, a conversion of the input subsystem to sysfs (includes some driver model changes which will need reworking prior to merging), a big reiser4 update meant to address various review comments, the removal of the perfctr patches (the maintainer is moving on and recommending perfmon instead), and some page allocator tweaks.

Comments (2 posted)

Quotes of the week

-- Linus Torvalds

	if (is_key_possessed(keyref)) {

-- Linus Torvalds

Comments (none posted)

A new approach to kernel timers

Ingo Molnar's recently-released 2.6.13-rt6 tree, which contains the realtime preemption patch set, brought a surprise in the form of a new high-resolution timer implementation by Thomas Gleixner. Ingo has stated his intention to merge this new code ("ktimers") upstream, so it merits a look.

The ktimer implementation starts with the view that there are two fundamentally different types of timers used in the system. They are (using the terms adopted by the patch):

• Timeouts. Timeouts are used primarily by networking and device drivers to detect when an event (I/O completion, for example) does not occur as expected. They have low resolution requirements, and they are almost always removed before they actually expire.

• Timers are used to sequence ongoing events. They can have high resolution requirements, and usually expire.

The current kernel timer implementation is heavily oriented toward timeouts. To see how, consider the following diagram which, with sufficient imagination, can be construed as a model of the data structure used inside the kernel to manage timers:

At the right side of the diagram is an array (tv1) containing a set of 256 (in most configurations) linked lists of upcoming timer events. This array is indexed directly by the bottom bits of a jiffies value to find the next set of events to execute. When the kernel has, over the course of 256 jiffies, cycled through the entire tv1 array, that array must be replenished with the next 256 jiffies worth of events. That is done by using the next set of jiffies bits (six, normally) to index into the next array (tv2), which points to those 256 jiffies of timer entries. Those entries are "cascaded" down to tv1 and distributed into the appropriate slots depending on their expiration times. When tv2 is exhausted, it is replenished from tv3 in the same way. This process continues up to tv5. The final entry in tv5 is special, in that it holds all of the far-future events which do not otherwise fit into this hierarchy.

This structure has some distinct advantages. It can retrieve all of the events to execute with a simple array lookup. Insertion of events is cheap, since their location in the structure is easy to calculate. Importantly, the removal of events is also cheap; there is no need to search through a long list of events to find a specific one to take out. Since most timeouts are removed before they expire, quick removal is a useful feature.

On the other hand, this data structure is firmly tied to jiffies values, and cannot easily cope with timers with sub-jiffies resolution. The cascade process, which moves events from the higher arrays to the lower ones, can be expensive if there are a lot of events to work with. Events which are removed prior to expiration will often not have to be cascaded at all, while those which survive through to expiration will have to work their way through the structure. If the clock interrupt frequency is raised (to get better timer resolution), these cascades will happen more often, and the cost of the data structure goes up.

The ktimers patch makes no changes to the existing API or data structure, which are deemed to be adequate and efficient for use with timeouts. Instead, it adds an entirely new API (and internal implementation) aimed at the needs of high-resolution timers. So ktimers are described entirely with human time units - nanoseconds, in particular. They are kept in a sorted, per-CPU list, implemented as a red-black tree. This structure provides for relatively quick insertion or removal, though it will be slower than the timeout structure shown above - but there is no need for the cascade operation.

The core structure for ktimers is, unsurprisingly, struct ktimer. They must be initialized before use with one of the following functions:

    void init_ktimer_mono(struct ktimer *timer);
    void init_ktimer_real(struct ktimer *timer);

Internally, each ktimer is tied to a "base," being the clock by which it is run. The ktimer patch provides two such clocks. The "monotonic" clock is similar to jiffies in that it is a straightforward, always-increasing count. The "realtime" clock, instead, tries to match time as known outside of the system; that clock can be corrected by the kernel or by the system administrator. A ktimer with a 5ms expiration will, if initialized with init_ktimer_mono(), expire 5ms in the future (with the usual proviso that delays can happen). That same timer, if initialized with init_ktimer_real(), will expire when the realtime clock says that 5ms have passed. But, since the realtime clock may be adjusted in the meantime, the actual elapsed time could differ.

There are some caller-accessible fields in struct ktimer:

    void (*function)(void *);
    void *data;
    nsec_t expired;
    nsec_t interval;

When the timer expires, function() will be called with data as its argument. The expired field will contain the time at which the timer actually expired, which might be later than requested. Interestingly, the high-resolution version of the ktimers patch does not set this field. Finally, interval is used for periodic timers.

A timer is set with a call to:

    int start_ktimer(struct ktimer *timer, nsec_t *time, int mode);

Here, time is the expiration time in nanoseconds, and mode describes how that time is to be interpreted. The possible mode values are:

• KTIMER_ABS: the timer will expire at an absolute time.
• KTIMER_REL: the given time value is a relative time, which must be added to the current time to get an absolute expiration time.
• KTIMER_INCR: for timers which have been used before, the time value is added to the previous expiration time.
• KTIMER_FORWARD: like KTIMER_INCR, except that the time value will be added repeatedly, if necessary, to obtain an expiration time in the future.
• KTIMER_REARM: like KTIMER_FORWARD, except that the interval value stored in the timer is added.
• KTIMER_RESTART: the expiration time of the timer is not changed at all.

For KTIMER_FORWARD and KTIMER_REARM, the ktimer code also maintains an integer overrun field in the ktimer structure. If a timer is started after the next expected expiration time (in other words, the system fell behind and did not restart the timer soon enough), overrun will be incremented to allow the calling code to compensate.

The return value will be zero, unless the timer is already expired, in which case the timer will not be started and the return value will be negative. If, however, the mode argument contains the bit KTIMER_NOCHECK, the timer will be started and executed normally, regardless of whether it is already expired.

Most of the other ktimer functions are reasonably self-explanatory for those who have seen the current timer API:

    int modify_ktimer(struct ktimer *timer, nsec_t *time, int mode);
    int try_to_stop_ktimer(struct ktimer *timer);
    int stop_ktimer(struct ktimer *timer);

There is also a convenience function to make a process sleep on a ktimer:

    nsec_t schedule_ktimer(struct ktimer *timer, nsec_t *time, 
                           int state, int mode);

The additional argument here (state) should be TASK_INTERRUPTIBLE or TASK_UNINTERRUPTIBLE, depending on whether the sleep should be interrupted by signals or not. The return value is the number of nanoseconds remaining in the requested sleep time; it will be zero except when the sleep is ended prematurely.

The standalone ktimers patch posted by Thomas is the version most likely to be merged. This patch runs ktimers from the normal clock interrupt, with the result that it provides no better resolution than the existing timer API. All of the structure is there to do better, however, once the low-level timer code and architecture specific support is in place. A separate patch exists which enables ktimers to provide high-resolution timers on the i386 architecture.

So far, the largest objection to the ktimer implementation is the use of nanoseconds for time values. Nanosecond timekeeping requires 64-bit variables, which will slow things down a little on 32-bit systems. The response from the developers is that the additional overhead is almost zero and not worth worrying about. So, unless some other surprise turns up, ktimers could find their way into the kernel not too long after 2.6.14 comes out.

(See also: this posting from Thomas, which describes the motivation behind ktimers and its relation to other timing patches in detail).

Comments (5 posted)

ZONE_DMA32

Not all systems implement all of these zones. Some newer architectures do not support ancient peripherals and leave out ZONE_DMA. In general, 64-bit systems have no addressing problems and do not need ZONE_HIGHMEM. The ia64 architecture settled on a different implementation of ZONE_DMA, defining it to cover all memory addressed below 4GB.

As it turns out, there are uses for a 4GB zone. Quite a few devices have trouble accessing memory which cannot be addressed with 32 bits. Drivers for such devices have been forced to use ZONE_DMA, the I/O memory management unit (on systems which have one), or bounce buffers. None of those solutions is ideal: ZONE_DMA is a small and scarce resource, IOMMU space can also be scarce, and bounce buffers are slow. All of these problems could be avoided if DMA memory could be reliably allocated below the 4GB boundary.

Andi Kleen has decided that the time has come for the x86-64 architecture to support a 32-bit DMA zone. So his patch adds a new zone (ZONE_DMA32) and an associated GFP flag (GFP_DMA32) for allocations. According to Andi, the reason which prevented the addition of this zone in the first place (the fact that the virtual memory subsystem had a very hard time balancing memory between zones) has gone away. Meanwhile, the lack of this zone is causing real problems.

This patch does not actually add the new zone for any architecture except x86-64. For ia64, it causes GFP_DMA to mean the same thing as GFP_DMA32, with the idea that GFP_DMA should, once again, be restricted to the older, 24-bit meaning. The patch also causes the generic DMA code to use the new zone when it makes sense, making it available to properly-written drivers with no additional work required.

This patch has come too late for inclusion into 2.6.14, but expect to see it in a mainline kernel shortly thereafter.

Comments (none posted)

Predictive per-task write throttling

The Linux VM subsystem attempts to address this problem with a simple form of write throttling. When the number of dirty pages gets too large, a process caught in the act of dirtying a page will be sent off to write out a few pages before being allowed to proceed. This technique slows the dirtying of pages while simultaneously helping to reclaim pages which have already been written to. This write throttling code makes no attempt to penalize any specific process, however; it will happily throttle any process which dirties a page at the wrong time.

Andrea Arcangeli has decided to improve the situation with a per-task predictive write throttling patch, currently found in the -mm tree. The patch is surprisingly simple - especially after noting that the bulk of it is involved with setting up the /proc and sysctl control interfaces.

At its core, the patch adds a simple accumulator which keeps an approximate count of the number of pages dirtied by each process over the last five seconds. It then assumes that each process will continue to dirty pages at about the same rate into the future. The "are there too many dirty pages?" calculation is then changed to take this rate into account. The code, thus, is making a guess at what the dirty memory situation will be like in the future, based on what each process is doing. Any process which looks like it will cause too much memory to be dirtied gets to perform writeback for a while, while processes which are not writing to lots of pages are not given that particular chore.

Andrea's preliminary results show that, with this patch in place, small, interactive tasks run in competition with a large copy task will run more quickly. Since the copy operation is being made to perform writeback (when it would have otherwise been dirtying more pages), more memory is available for the other tasks in the system. The interesting part of the result is that the copy task runs no slower with this patch in place. A process which is bound by the system's ability to write pages to disk will not benefit from being allowed to dirty the bulk of the system's memory, and it will not suffer by being throttled. So this little patch looks like it could be a winner for everybody involved.

Comments (6 posted)

A Look at Slackware Linux 10.2

The above paragraph summarizes why Slackware, which had as much as 90% market share of all Linux installations in the mid-nineties, has slowly and painfully become a niche distribution, catering mostly to die-hard Linux geeks. A good case in point is the kernel in Slackware 10.2. Although the default kernel is version 2.4.31, version 2.6.13 is also provided in the /testing directory for the more adventurous users. This kernel can be selected during installation. Once you do that, however, the system will boot into the new kernel without loading any kernel modules, disregarding any hardware detection that might have taken place during the installation. Users are then left to their own devices (no pun intended) to set up and load any kernel modules they might require.

The situation is somewhat better if the user chooses one of the standard binary kernels - either the bare one, or one of the specially prepared kernels with support for certain less common hardware. This type of installation will result in a functional system, with kernel modules for sound cards, USB devices, and network cards loaded and working properly. But the installer does nothing to set up the graphical part of the system; although it provides a functional xorg.conf file with a VESA driver and a decent screen resolution and color depth, it does not extract information from the graphics card, let alone create a proper configuration file with the parameters supported by the card. Configuring X, together with adding non-root users, is a manual task left entirely to the person performing the installation.

Virtually all major distributions available today do an excellent job setting up not only graphics cards and monitors - even more exotic devices, such as scanners, wireless cards or digital cameras, can often be detected and configured without any user intervention. Of course, any such interference with the kernel might introduce bugs and even serious instability, and this is something that Slackware is trying to avoid at all costs. As such, there is little wonder that Slackware is considered to be one of the most stable and bug-free distributions - without taking any risks and without introducing even remotely troublesome code into the product, Slackware is indeed rock solid. And if a user decides to load a kernel module and things go wrong, then it's the user's problem, not Slackware's.

The above attitude means that Slackware is a great product for deployment on servers, but much less exciting as an operating system on workstations - at least until the distribution is painfully set up to support all the peripherals. Even so, some users might be disappointed with the new Slackware release, which, for the first time in years, ships without the GNOME desktop. Although not everybody likes GNOME, there are useful GTK+ and GNOME applications that many might choose to run while logged into KDE or one of the other available desktops. Those users will now have to get GNOME from independent sources, perhaps from Freerock GNOME or GWARE, thus adding a layer of complexity to the process of security updates. And if you think about using the popular Dropline GNOME packages on Slackware, then think again - due to the project's insistence of adding PAM and replacing large system packages, Patrick Volkerding does not recommend it as a suitable option.

Security and system updates provide further cases in point to illustrate how much more convenient most modern distributions have become over the last few years. Although Slackware issues security advisories and provides timely security updates, the process of patching holes is as cumbersome as ever - it entails downloading the updated package manually, then checking its signature, before firing up Slackware's pkgtool to upgrade the vulnerable package. Similarly, a highly manual method awaits any user who decides to upgrade from an older version of Slackware Linux to a newer one - a complicated 10-step process that starts with dropping to runlevel 1, then updating glibc, pkgtool and sed before proceeding with the rest of the software and before bravely refreshing all the configuration files and clean up the resulting mess. Suddenly, you wish that you were running Ubuntu, which can be upgraded with a single command, or SUSE, where a similar task can be achieved from within a nice graphical application.

Before I get reminded about it - yes, I know that Slackware can be extended to include various third-party tools and applications that make security and system upgrades so much more convenient. It also enjoys a large number of community sites that package extra software for Slackware. With their help, Slackware can indeed be extended into a more complete and user-friendly distribution that can do anything that other modern distributions do out of the box. But will it be still Slackware? Or will it be a new distribution where only the base is Slackware, while the reminder is a mix of third-party tools and applications where stability and security are no longer guaranteed?

And that's really what Slackware Linux is today: a base system with the Linux kernel, GNU, pkgtool and a fairly bare collection of the most common open source applications. As such, it gets very high marks for being an extremely clean, stable, reliable and secure operating system. On the other hand, it scores very low in terms of user-friendliness, hardware setup, upgrade convenience and features. A perfect system for many web or file servers, for the geeks who need to have total control, and for those who wish to build a new distribution on top of it.

Comments (11 posted)

Slackware 10.2 is released

Comments (2 posted)

Preview of Linux DCC 3.0 Released (LinuxElectrons)

Comments (none posted)

Giving Desktop/LX users a Mandriva Club discount

Full Story (comments: none)

DebCentral.org launches users group for Debian based distributions

Full Story (comments: none)

Debian Project news

DVD videos of the Debconf5 sessions (plus Debian Day and some extras) are now available in PAL format. NTSC format discs and downloadable images will be coming soon.

If you have been having problems getting recently released security updates, you may just have to be patient. "The recently released security update of XFree86 in DSA 816 for sarge and woody has caused the host security.debian.org to saturate its 100MBit/s network connection entirely. Due to the large number of X packages, the gross size of these packages and the high number of users who need to install the update, the server is busy sending out updates which exhaust its total outgoing bandwidth."

There has been ongoing discussion of an architecture-specific release criteria. Some architectures will need to re-qualify to be included with 'etch'.

Comments (1 posted)

Order SUSE Linux 10 now

Full Story (comments: none)

AspisOS Linux

Comments (none posted)

Debian Weekly News

Full Story (comments: none)

Fedora Weekly News

Comments (none posted)

Gentoo Weekly Newsletter

Comments (none posted)

Mandriva Community Newsletter #108

Full Story (comments: none)

Red Hat Magazine

Comments (none posted)

DistroWatch Weekly, Issue 118

Comments (none posted)

New primary Quantian mirror available

Full Story (comments: none)

Fedora updates

Updates for Fedora Core 3: gnupg (update to 1.2.7), mc (bug fixes), openmotif (fixed mrm initialization error), termcap (new termcap-description for rxvt-unicode-terminal-emulator), xorg-x11 (bug fix).

Comments (none posted)

Mandriva MDKA-2005:040

Full Story (comments: none)

Trustix Secure Linux Bugfix Advisory TSL-2005-0048

Full Story (comments: none)

ISP Server Setup - OpenSUSE 10 RC 1 (HowtoForge)

Comments (none posted)

First Look at Ubuntu 5.10 Preview (Mad Penguin)

Comments (none posted)

Opinion: Make mine a Lite, a MEPISLite (Linux-Watch)

Comments (none posted)

Back Up to CD with Cedar Backup

The code is a second-generation effort, according to the project history. It started out as a Perl application, and was later changed to Python and renamed.

Unlike more traditional tape-based backup systems, Cedar Backup is squarely aimed at the use of common and inexpensive CDR media. A big advantage of CD-based backups is the ability to read the backup media on just about every computer that one can buy today.

The online manual describes the numerous Cedar Backup features:

• Supports master/client machines on a network.
• Uses ssh-based encryption for moving backup data between machines.
• Runs with a four-stage backup process.
• The backups are fired off from a series of cron scripts.
• Writes backups to CDR and CDRW media types.
• Supports multi-session disks.
• Writable DVD support is planned for a future release.
• Performs daily, weekly and incremental backup types.
• Backups are initiated from a command line interface.
• Configuration information is stored in an XML-formatted file.
• Sends error messages via email.
• Stores directories as tar files with optional compression.
• Comes with extensions for backing up subversion and MySQL data.
• Allows user-supplied extensions for backing up other types of data.
• Restore operations work on any machine.

Version 2.6.0 of Cedar Backup was released last week, it adds minor feature enhancements and bug fixes: "This release is focused around a wide-ranging set of enhancements, bugfixes, and documentation updates. The list of changes is fairly large, although not much of the core functionality was touched."

The software is available as a Debian package, or an easily installed Python script. Installation on a Fedora Core 3 system was simply a matter un-tarring the source and running the install script. The dependencies of the package include the Python language on all machines, and a number of CD-specific utilities on the master machine.

If you need to back up a single machine, or a group of machines, Cedar Backup is worth investigating.

Comments (1 posted)

Release 2.0.1 of Linux-HA is available

Full Story (comments: none)

PostgreSQL 8.1 Beta 2 available

Full Story (comments: 1)

PostgreSQL Weekly News

Full Story (comments: none)

Sendmail 8.13.5 released

Comments (none posted)

Setting up a repository server with yum-pull

Full Story (comments: none)

John the Ripper 1.6.39 released

Full Story (comments: none)

SSL-Explorer 0.1.13 released (SourceForge)

Comments (none posted)

Mod_python 3.2.2b (Beta) is out

Comments (none posted)

NAJAX 0.4.0.0 released (SourceForge)

Comments (none posted)

LASH 0.5.0 released

Full Story (comments: none)

GNOME Software Announcements

• GARNOME 2.12.0.1 (new features and bug fixes)
• Serpentine 0.6.2 (new features and bug fixes)
• Sysprof 1.0 (new features)

Comments (none posted)

The GNOME Journal, September Edition (GnomeDesktop)

Comments (none posted)

Preliminary Gnome 2.14 Schedule

Full Story (comments: none)

Scribes 0.1 Released (GnomeDesktop)

Comments (7 posted)

KDE Software Announcements

• amaroK 1.3.2 (new features and bug fixes)
• eric3 3.7.2 (bug fixes)
• iList 0.4.7 (new features and bug fixes)
• KAlgebra 0.3 (new features and bug fixes)
• KDE Asciiquarium 0.3.2 (new features)
• KEnigma 0.5 (new features and bug fixes)
• Kalva 0.8.0 (new features)
• Konstruct 20050921 (installs KDE 3.5 Beta)
• kradview 0.5.3 (bug fixes)
• nokuake 0.1 (kuake replacement)
• xik 0.5 (initial release)

Comments (none posted)

KDE 3.5 beta 1 released

Comments (2 posted)

Don't Install, Just Copy with klik (KDE.News)

Comments (none posted)

Gadgetboard 1.0 announced

Comments (3 posted)

XCircuit 3.3.36 released

Comments (none posted)

Block Rage version 0.2.1 Released (SourceForge)

Comments (none posted)

Wine Traffic

Comments (none posted)

CMS Releases Beta VOE (LinuxMedNews)

Comments (none posted)

Jackbeat 0.5.3

Full Story (comments: none)

KMiditracker 0.5.13 released

Full Story (comments: none)

Patchage 0.2.3 Released

Full Story (comments: none)

PyVISA 0.9.6 released

Comments (none posted)

Mozilla Firefox 1.0.7 Released (MozillaZine)

Comments (none posted)

SeaMonkey 1.0 Alpha Released (MozillaZine)

Comments (none posted)

Boost C++ library 1.33.0 released

Comments (none posted)

Caml Weekly News

Full Story (comments: none)

AndroMDA 3.1 Release Candidate 1 has been released (SourceForge)

Comments (none posted)

This week on harmony-dev

Full Story (comments: none)

What Is a Portlet (O'ReillyNet)

Comments (none posted)

This week in perl6-compiler (Perl.org)

Comments (none posted)

Understanding MVC in PHP (O'Reilly)

Comments (none posted)

Dr. Dobb's Python-URL!

Full Story (comments: none)

PyInstaller 1.0 released

Comments (none posted)

Ruby Weekly News

Comments (none posted)

Dr. Dobb's Tcl-URL!

Full Story (comments: none)

Processing Atom 1.0 (XML.com)

Comments (none posted)

The More Things Change (XML.com)

Comments (none posted)

Eric3 3.7.2 released

Comments (none posted)

Mercurial v0.7 released

Full Story (comments: 9)

XPlanner 0.7 beta 1 released (SourceForge)

Comments (none posted)

Microsoft Challenges Massachusetts on Open-Format Plan (eWeek)

Comments (27 posted)

Are potential legal liabilities holding back Linux adoption? (TechRepublic)

Comments (2 posted)

Chart of SCO's Answer to Novell's Counterclaims (Groklaw)

Comments (15 posted)

Red Hat and IBM to take Linux to emerging markets (PCPro)

Comments (none posted)

Bid to trademark the word 'Linux' rejected (News.com)

Comments (22 posted)

Politicos call for music copyright reform (News.com)

Comments (3 posted)

U.S. Patent Reform Bill: An Interview with Mark Webbink (O'ReillyNet)

Comments (none posted)

A Glimpse of SUSE Linux 10.0 and Other Things Brewing at Novell (MozillaQuest)

Comments (3 posted)

An Interview with Chia-liang Kao (O'ReillyNet)

Comments (4 posted)

A Brief Introduction to Multi-Category Security (MCS) (LiveJournal)

Comments (none posted)

What Is Open Source (O'ReillyNet)

Comments (4 posted)

The Daemon, the GNU and the Penguin, Chapter 18 (Groklaw)

Comments (none posted)

Peter van der Linden's Guide to Linux: A Lesson in Encryption, Part 1 (Linux Journal)

Comments (2 posted)

OOo Off-the-Wall: Back to School with Bibliographies (Linux Journal)

Comments (none posted)

Five Pitfalls of Linux Sockets Programming (developerWorks)

Comments (30 posted)

CLI Magic: CDargs (Linux.com)

Comments (7 posted)

'Firefox Secrets' and 'Hacking Firefox' Published (MozillaZine)

Comments (none posted)

Wireless made easy with Netapplet (NewsForge)

Comments (15 posted)

Book Review: Perl Best Practices (Linux Journal)

Comments (none posted)

Using Qpsmtpd (O'ReillyNet)

The return of Citizens Against Government Waste

Comments (22 posted)

Nokia Joins Eclipse Foundation as Strategic Developer and Board Member

Comments (none posted)

OSV to States: Follow Massachusetts in Open Document Standards

Full Story (comments: none)

Dell releases its first Linux consumer product with Mandriva

Comments (6 posted)

Linbox improves the Linbox Rescue Server

Full Story (comments: 1)

Mandriva revamps partner program

Comments (none posted)

MontaVista Linux on blades

Artesyn Communication Products has announced that its PICMG 2.16, AdvancedTCA, AdvancedMC, ProcessorPMC blades and modules, will come with MontaVista Carrier Grade Linux.

Comments (none posted)

Opera now ad banner free

Full Story (comments: 9)

Red Hat GFS Now Supported by Oracle, EMC and NetApp

Full Story (comments: none)

Polycom Chooses Red Hat to Manage Storage Clusters

Full Story (comments: none)

Versora and Win4Lin Announce Linux Desktop Migration and Virtualization Toolkit

Full Story (comments: none)

Zend Technologies Announces Zend Platform 2

Comments (none posted)

No Starch Press Releases "The Debian System"

Full Story (comments: none)

RT Essentials - O'Reilly's Latest Release

Full Story (comments: none)

No Starch Press Releases: "The TCP/IP Guide"

Full Story (comments: none)

The LDP Weekly News

Full Story (comments: none)

Linux training materials now available free (NewsForge)

Comments (none posted)

Samba PDC and Kolab with OpenLDAP: integration

Comments (none posted)

Clarkson University wins first TuxMasters invitational (NewsForge)

Comments (none posted)

The Libre Graphics Meeting

Full Story (comments: none)

LPI Offers Free Certification Exams at LinuxWorld UK

Full Story (comments: 1)

MySQL Users Conference 2006 - Call for Participation

Full Story (comments: none)

Open Source Desktop Workshop, San Diego

Full Story (comments: none)

@System Security Conference, Pisa

Full Story (comments: none)

Univa Grid Technology Pioneers to Present at Multiple Industry Conferences

Comments (none posted)

Zend/PHP Conference and Expo 2005 Program Announced

Comments (none posted)

Events: September 22 - November 17, 2005

Comments (none posted)

ODBMS.ORG Launches Portal on Object Databases

Full Story (comments: 4)

Beta Broadcast 001: 'Is this thing on?' (O'ReillyNet)

Comments (none posted)

Response to "TOE performance" letter in Sept 8 edition

Jonathan
 
In his "TOE performance" letter in the September 8th issue, Dave S.
Miller asked for some further TOE performance information, and I'd like
to provide the following response.
 
Regards,
 
Asgeir Eiriksson
CTO
Chelsio Communications Inc.
 
----
 
 
We welcome the chance to respond to the concerns that David Miller has
about TOE cards.
 
First, let me remark that it seems to me that he's been badly burned in
the past by over-hyped TOE cards, but I maintain that we at Chelsio have
learned from these prior mistakes by other people, and we should have a
fresh look at TOE cards at 10GE speeds. I'll mention some of the reasons
in the following:
 
>
> > You might want to ask the Chelsio guys to provide some performance
> > metric other than their "land speed record" that, as Linux
> > networking stack maintainer, I'm frankly sick of hearing about over
> > and over again.
  
Considerable number of HPC folk pay close attention to the LSR, and it
also demonstrates the resiliency of the TOE implementation to different
topologies (not all TOEs were created equal) and applications for remote
back-up (e.g. for single connection data transfers such as FTP).
Admittedly, the LSR has in fact gotten too much press.
 
>
> > What's more interesting to me is an area I know TOE is poor in, and
> > that is TCP connection rates. It's all too easy to make one sole
> > connection pump a lot of data, but it's hard to make a web or
> > database server serve hundreds of thousands of connections per
> > second. TOE cards generally cannot do that because each connection
> > setup/teardown requires setting up and tearing down state on the
> > network card, which subsequently kills TCP connection rates.
>
 
I agree with your list of important performance corners so the Chelsio
TOE is designed from the ground-up with these in mind.
  
I believe, your observations on the connection setup process might be
valid for the way Microsoft Chimney currently sets up connections (time
will tell), but this is not how Chelsio is proposing to do TOE on Linux.
 
In the proposed Linux patch the connection setup and teardown is
offloaded to the NIC, and a SYN that hits an offloaded listening server
triggers a request-to-host/response-from-host to "ASK" the host if the
connect request should be accepted or not (this allows full integration
with linux access controls, etc.). The response-from-host triggers the
sending of the SYN+ACK so I would maintain that this flavor of
connection setup integrates well with the Linux access controls, and you
will see some benefit in setup/teardown performance. The hardware is
capable of processing SYN packets at line rate (some millions/sec but
the setup loop includes socket creation/destruction, kernel checks and
controls on acceptance, making the overall rate lower). In the proposed
patch the active open is also offloaded, and this will lead to a
significant benefit in performance. Finally, the FIN processing is
offloaded for both types of close.
  
The Chelsio TOE does not have any on-chip caches and therefore has a
flat performance profile as the number of connections is increased. We
have measured the performance up to 14000 connections with linux 2.6.*
(about 6Gbps aggregate BW on an Opteron) and our profiling of the code
indicated to us that we were running into linux bottlenecks (select()
de-multiplexing, etc.) at that point and not hitting TOE issues (yet).
The veritest report Figure 3 and 4 at the following location
http://www.chelsio.com/technology/Chelsio10GbE_Fujitsu.pdf shows the BW
from 1 up to 1000 connections, but the 14000 connection measurement is
unpublished internal data.
  
We also included low end-to-end latency in the list of design
objectives, and traffic shaping and pacing capabilities. The first
requirement is to enable going toe-to-toe with the IB and FC crowd. The
second requirement is useful for media pumping applications of 10s of
thousands of audio, or thousands of MPEG streams at one extreme, and on
the other end of the scale to throttle and give-priority to connections
that are going to e.g. storage, etc.
 
These issues are just a sampling of the issues that we've encountered in
our TOE integration work, and I have no doubt that the linux community
at large is capable of improving the integration, utility, and
performance of the TOE even further, and this frankly is one of the
motivations behind open sourcing our TOE software and submitting the
patch.
 
>
> > So if you're a scientist trying to break the land speed record
> > between Stanford University in California and some place in the
> > middle of Europe on the other side of the planet, yeah TOE is
> > probably a great toy to play with.
>
> > TOE users are niche, always have been, and always will be. It is no
> > mistake that the Chelsio guys do not delve into this aspect of their
 
> > technology.
>
> > And the study they mentioned in their mail to you of course will be
> > full of accolades for their approach. If you read only the documents
 
> > posted on their web site, you might think that TOE is the best thing
 
> > since sliced bread.
>
 
 
The publications on the Chelsio website fall into three broad
categories: a) PR by marketing people, b) white-papers by our engineers,
and c) published papers by some of the top names in the HPC field, and
you're no doubt referring to a) and maybe b) above in your remarks. The
papers in category c) are by independent researchers in the HPC field,
and in their papers they've chosen the applications to benchmark, and
they've chosen what to measure and how to measure it.
 
The following is the list of such publications at this writing:
 
[1] "Head to TOE Evaluation of High-Performance Sockets over Protocol
Offload Engines", by Dr. Wu Feng of Los Alamos National Labs, Dr. DK
Panda of Ohio State University, et al., that will appear at Cluster
2005, Boston. Available at
http://www.chelsio.com/technology/Cluster_2005_Techical_R...
ANL.pdf
 
[2] "Performance Evaluation of a 10-Gigabit Ethernet TOE", by Dr. Wu
Feng of Los Alamos National Labs, et al. that appeared last month at Hot
Interconnect 2005. Available at
http://www.chelsio.com/technology/HotInterconnect_2005.pdf
  
[3] "Infiniband and 10-Gigabit Ethernet for I/O in Cluster Computing",
by Helen Chen of Sandia National Labs, et al. that appeared in July at
the Cluster Symposium 2005. Available at
http://www.chelsio.com/technology/Cluster_Symposium_2005_...
ndi.pdf
 
 
When I look through [1], [2], and [3] I observe the following:
 
- the performance for the applications chosen, and presumably the
applications that the researchers care about show TOE outperforming NIC
- the end-to-end latency for socket API and TOE is less than for SDP
over IB
- there are various traffic profiles in the benchmarks and TOE does well
on every single one.
 
There's of course a lot more there in the papers, and I invite people to
look through the results for themselves and reach their own conclusions.
  
As an aside: it is also interesting to note that TCP+Ethernet flow
control does great against the supposedly superior IB flow control in
all the above experiments.
  
> > The TOE folks are frankly between a rock and a hard place. They need
 
> > some support in upstream Linux for their solution to really be far
> > reaching and viable, yet the negative aspects of their technology
> > are such that this is likely not going to happen.
>
> > They also refuse to actively consider stateless offloads, which are
> > much better for long term maintainability and do not bypass the
> > Linux TCP networking stack we've been tuning for 10+ years. Doing so
 
> > would at least make these guys appear less anti-social and I would
> > certainly pay more attention to their concerns if they at least made
 
> > some efforts in this area. But they'll never do something so open
> > minded because their whole buisness model surrounds TOE.
>
> > With that in mind I applaud folks like Lenoid Grossman who are
> > working on stateless TCP receive offloads for highspeed networks on
> > the products they work on.
>
> > Take care.
>
 
Our NIC in addition to having TOE support, also offloads iSCSI, and
iWARP (RDMA), and has support for stateless offload technology such as
TSO and checksum generation/checking, and supports MSS of 1500B up to
jumbo frame size for each of these traffic types.
  
So, we are obviously for customer choice, and if I were to extract one
NIC vs. TOE observation from all our performance comparison work to
date, it would be the following:
 
- NIC with jumbo frames can fill a 10GE wire in the Tx or Rx directions,
but the NIC gets into trouble as the average packet size goes down
(packet frequency goes up) or the connection count goes up.
- TOE with a traffic mix anywhere from 500B average frame size to jumbo
frame size will fill a 10GE wire, i.e. no performance corners at any
packet size or connection count
 
Finally, not all applications are data mover applications that can use
jumbo frames, and there are applications with smaller packet sizes that
clearly benefit from TOE at 10GE speeds, so high performance TOE
integration into Linux clearly deserves to be considered without any
preconceived notions, similar to all other new technologies that Linux
considers for inclusion.
 
Regards,
 
Asgeir Eiriksson
CTO
Chelsio Communications Inc.

Comments (1 posted)