Sponsored link
Vyatta –
Linux & Open Source
Alternative to Cisco –
Advanced Routing,
Firewall, VPN, QoS..
Free Download ->
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
LWN.net Weekly Edition for September 13, 2007LinuxConf.eu wrapup The very first LinuxConf Europe event was held in Cambridge, UK, in the first week of September. This conference is the result of a cooperation between the UK Unix User Group and the German Unix User Group; it is, in a sense, a combination of the UKUUG and Linux-Kongress events held in previous years. Talks by Dirk Hohndel and Michael Kerrisk were published last week. Here is a summary of some other LCE events.
Power management remains the focus of a great deal of attention. Arjan van
de Ven started off a set of power-related talks with an overview of where
the problems are. His biggest point is that software is a critical part of
the power consumption picture; contemporary hardware provides a number of
In the case of the CPU, idle periods of 20ms to 50ms are needed for effective power savings. Past kernels have rather defeated that goal, though, by receiving a clock interrupt every 1-10ms. The dynamic tick patches have finally fixed that problem, making it possible for longer sleeps to happen. But then user space comes along and ruins things. Since the advent of PowerTop, though, improvements have been coming quickly. Many distributions now consume at least 30% less power in typical laptop use. Things may be getting better, but Matthew Garrett started the following session by noting that Linux still sucks - at least, it sucks power. This is a problem, he says, because getting half the battery lifetime as Windows on the same hardware is really embarrassing. Systems are still waking up far too much; the problems exist in both kernel and user space.
User-space applications remain a problem. People tracking down wakeups often blame the X server, but the real trouble is usually the applications which are causing X to wake up. There is a tool in the works which will identify the real source of X wakeups; this is a good thing: once problems are identified they are usually fixed pretty quickly. Polling for vertical retrace periods (so that the display can be updated without artifacts) seems to be a particular problem; some API work is being done to make it easier to avoid this polling. Evidently there are also some applications which repeatedly ask the server if a particular extension is available; since the set of extensions does not change while the server is running, there is little point in doing this. There are some interesting things which can be done to better use the power-saving features of the hardware. For example, some framebuffers can compress the video data into a dedicated memory area, then drive the video from the compressed data. This technique reduces video memory bandwidth, saving power (up to half a watt) in the process. An interesting consequence is that the amount of power saved is dependent on how well the screen's contents compress - a user's choice of background wallpaper will affect their power usage. Finally, there is a lot to be gained if device drivers can communicate more information to user space, making polling unnecessary. Applications which poll for changes to the audio volume are an example here; if the sound system simply told them that the volume had been adjusted, they could update their displays and go back to sleep.
Flash makes life easier in a number of ways; the lack of seek delays, for
example, means that much of the trouble the kernel goes to in scheduling of
block I/O operations can be eliminated. On the other hand, flash has
challenges of its own: it is not quite the random-access array of blocks
that one would like. In particular, writing to flash requires dealing with
wear-leveling issues, erase operations, and more.
Manufacturers have done their best to paper over these issues through the
use of translation layers which make a flash array look like a simple disk
drive. These layers make it easier to use flash with existing software,
but there are problems: performance is not always what one would like, and
there can be hidden caches which delay the persistent storage of data. So
Jörn has a request to the flash manufacturers: give us direct access
to the flash array, without translation layers, and let us figure out how
to best support it.
Chris Mason is not waiting for flash to take over; instead, he is working
on the next-generation Linux filesystem for rotating disks. The result, Btrfs, was the subject of
Chris's talk at LCE. LWN covered
Btrfs last June.
The directories are actually indexed twice. One index is there for fast
filename lookup; the other one, instead, lets the readdir() system
call return files in inode-number order, speeding filesystem traversals.
Extended attributes are stored as directory entries. Every file has a
backpointer to its containing directory - and, yes, multiply-linked files
have backpointers to all of the directories in which they are found.
Perhaps the most fun part of the talk was the plots Chris has generated
from various benchmark runs. The limiting factor on filesystem performance
is generally disk seeks; it is important to minimize disk head movement.
In general, ext3 tends to move the disk head all over the platter during
benchmark runs while Btrfs and XFS do better. Chris noted that better
writeback clustering in the virtual memory subsystem would help ext3.
More benchmark plots (some animated) can be found in the Btrfs
benchmark and Seekwatcher pages.
Toward the end, Chris was asked whether performance slows down when the
disk gets full. The answer was "no" because the system crashes instead.
That's a good reminder that Btrfs remains an early-stage development; the
on-disk format has not even been finalized yet. But the production version
of Btrfs is certainly something to look forward to.
Back in 2000, the British Computer Society awarded its Lovelace Medal to
Linus Torvalds. In 2007, the society finally caught up with him to deliver
the medal - though, as speaker Dr. David Hartley noted, they probably were
almost as quick as the post office would have been. As is typically the
case, Linus seemed somewhat embarrassed by the attention.
LinuxConf Europe intends to be a conference on a truly European scale. To
that end, next year's event will likely move to Germany; the details were
not yet finalized to the point that the location could be announced at this
year's conference, though. LCE, helped by the kernel summit, has gotten
this institution off to a good start; your editor is looking forward to
next year's edition.
Changes ahead for Python
With its first
alpha just released, Python 3.0 (aka Python 3000 or Py3k) is
making progress, though a final release is still a year off. Py3k overhauls
the language core, removing inconsistencies and other "warts", without
maintaining compatibility with the 2.x version. Various standard Python
idioms go by the wayside and it will take some getting used to.
One of the driving forces for Py3k is to handle unicode strings in a uniform
way. In the 2.x series, unicode handling has bugs, especially when mixing
encoded and unencoded text. The Py3k solution is to separate strings,
which contain decoded text, and byte-strings which are binary data into two
distinct types, str and bytes. Those types cannot be
combined without converting one via the encode() and decode()
methods. The drawback to this change is explained in the
What's New in
Python 3.0 document:
This also leads to a distinction that needs to be made when handling files. Files are either binary or text files, with text files requiring an encoding to be specified when they are opened. If the wrong type or encoding is given, I/O to the file may fail. One very visible change – perhaps the most controversial – is eliminating the print statement, moving it to a function. The change is being made mostly for consistency, as there is no other language statement like print, but it also adds additional features. One can now specify a separator, line ending, and file directly, there is no need for the print >>sys.stderr, "error" syntax, instead that becomes print("error", file=sys.stderr). As the "What's new" document points out:
Initially, you'll be finding yourself typing the old print x a lot in
interactive mode. Time to retrain your fingers to type print(x) instead!
Another area that has changed significantly is the dict methods. The keys(), items(), and values() methods no longer return lists, so code that treats them that way will fail. They now return something called a "view" that references the dict directly, producing values as they are needed, much like an iterator. In addition, the has_key() boolean method has been removed, the in operator should be used instead. There are lots of smaller changes that will catch the unwary. Many of the features removed have been deprecated for some time, but, for programmers who don't follow Python language development closely, they may surprise. The raise statement has different syntax, integer division no longer truncates, instead it returns a float (with // used to get the old behavior), xrange() has been removed, and so on. It adds up to a substantial pile of things to deal with when moving existing code to Python 3. The migration from 2.x is being assisted by the development of Python 2.6, which is slated for release in April 2008. It will provide a Py3k warnings mode that complains at runtime when a feature is being used in a way that is incompatible. It will also have many of the new features enabled, either as __future__ imports or just added into the language if it doesn't conflict with 2.x syntax. The 2to3 tool is also being developed to translate 2.6 constructs into their 3.0 equivalents. The Python Enhancement Proposal (PEP) governing the Py3k plan (PEP 3000) gives an overview of how code can be maintained to run on both 2.6 and 3.0. It sounds somewhat painful, but incompatible language changes are never easy. There is still plenty of work to be done, the final release of 3.0 is currently scheduled for August 2008. One of the bigger remaining chunks is a reorganization of the standard library namespace. PEP 3108 lays out the changes to be made, including removing older, unsupported, or rarely used modules, renaming modules to conform to the naming standard, merging the C and Python implementations of modules (i.e. cPickle goes away and is replaced with pickle). It cleans up what had become a bit of a mess over time. All of these changes have not come about without some objections, both from those who think another incompatible "upgrade" is not warranted to those who think Py3k doesn't go far enough. One area that is not being changed, but is a source of frustration for some, is the "global interpreter lock" (GIL), which only allows one thread at a time to operate on any Python objects or call out to C language extensions. Especially with the advent of multi-core and multi-CPU systems, the lock is very restrictive, serializing most of the core language processing. Guido van Rossum, Benevolent Dictator for Life (BDFL) of the Python language has been very open about addressing these concerns on his All Things Pythonic weblog. That doesn't mean he plans to change things, especially with regards to the GIL, but he puts together a well reasoned defense, mostly concerning the performance of the language with finer-grained locks. He is clearly not much of a fan of multi-threaded programming with its attendant race conditions, deadlocks, and other issues, but he is not opposed to efforts to remove the GIL either. As he points out, it is not inherent in the Python language, but is an attribute of the current language implementation, other implementations (Jython, IronPython) do not have the GIL. There are fundamental changes in Python 3, it will be interesting to see how quickly it is adopted after being released. People learning Python won't need to learn Py3k for another two years or so, according to van Rossum, and should, instead, concentrate on 2.x (which means 2.5 until April). The unicode handling rework will probably be enough to get the increasing number of localized programs updated, but the rest of the changes are not terribly compelling. It is likely that there will be Python 2.x programs around for a long time to come.
Fedora reaching out to new niches Purpose-built Fedora distributions, called "spins", are a recent addition to that community in an attempt to reach additional users. The idea is to use tools like Revisor to create a custom collection of software that work well together for a particular set of tasks. This collection can then be installed or run from a live CD, providing an easy means to have the right collection of tools immediately, rather than after a lengthy yum install pass. The concept itself is not new, there are many distributions targeted at a particular subset of users. Typically, other popular distributions (Debian and Ubuntu in particular) have been used as the basis for them. The Fedora project is embracing the idea, pulling together a list of the spins and elevating at least two to the status of "official spins". The idea is to appeal to those who don't want to be bothered with tracking down, installing, and configuring the tools needed for their task; instead it is all packaged for them. Starting with Fedora 7, two official releases of the distribution are available, one for each of the dominant desktops. For Fedora 8, there will also be a developer spin, which has the explicit goal of attracting more Fedora developers. It will include Eclipse, perhaps other integrated development environments (IDEs), gcc and friends, emacs, SystemTap, and other developer tools. Other ideas, such as a working Xen virtual machine and targeting web developers, have been discussed as well. The other official spin for Fedora 8 is the Fedora Electronic Lab (FEL). This project pulls together the tools for electronic design and configures them to work well together. A wide variety of software for circuit simulation, hardware development in VHDL and Verilog, Very Large Scale Integration (VLSI) design, and embedded systems development are included. Universities are high on the list of target audiences, with the FEL website claiming 250 universities already using Fedora; attracting more is one of the goals. Several other spins are being worked on as well, not "officially", but there does seem to be some serious work going into them. The Security LiveCD is a Fedora 7 based spin for security auditing and testing. It contains all of the tools that an administrator or security researcher might need to do forensic analysis of a rooted machine, check a network for vulnerable hosts, or do penetration testing. Since it can be booted directly from a read-only device, risks of infection from any malware are eliminated. Any machine can be quickly turned into a security workstation by using a distribution like this. Another ambitious project is the Fedora Art Studio. This spin not only collects the tools into one package, it also pulls in content likely to be useful to artists, desktop publishers, animators, and other creative folks. There are collections of clip art, fonts, textures, brushes, and so on, all with free licenses. There are also tutorials included to get people up to speed on the various packages. Plans are to include default Firefox bookmarks for useful sites as well. Other spins are listed on the site, ranging from the Creative Commons LiveContent spin (covered by LWN here) to a SystemTap live CD. The Fedora wiki has various Howtos on remixing and rebranding Fedora, as well as using the Live CD tools. Most people who want to build a custom spin will start by using the Revisor GUI tool, which provides options for installation, live or virtualization (for Xen or KVM virtual machines) media for CDs, DVDs, USB thumb drives and more. The project has clearly put a lot of time and effort into making it as easy as possible to create new spins from the large repository of Fedora software. It remains to be seen if any of these spins become popular, but it may be a good way to introduce new users to Fedora. It is unlikely that power users will find a spin that covers all of what they use, but they just might find one that serves as a good starting point. They can either customize their own spin from there or use the usual repository tools to grab whatever extras they need. For a distribution that, until recently, had a reputation for not working with the community, this effort may go a long way towards erasing that history.
LWN advertising update II LWN recently tried a new (for us) form of advertising, known as "in-text" advertising – ads that pop up from highlighted keywords in an article. When we announced the change, it was obvious from the comments that it was a tad unpopular. Truth to tell, they started getting on our nerves more as time went on; they didn't seem quite so annoying when running it on our development systems. We have discontinued the ads; they will not be coming back. A lot of good points were made in the comments, we appreciate the time you took to make them. Our readers are (obviously) very important to us; your opinions on what works and what doesn't are always carefully considered. There were also several interesting suggestions made, we will be pondering those as we make plans. We do want to dispel one concern that we heard. We are not under an imminent threat of going under. We are proceeding with the plan we laid out in May: working on the revenue side of the business while producing the same quality of content you have come to expect. There will be other experiments along the way; some will fail, hopefully some will succeed as well.
Page editor: Jonathan Corbet Security Eavesdropping on Tor traffic A Swedish security researcher, Dan Egerstad, recently highlighted a flaw in the way many folks are using Tor, a tool for internet anonymity. He said that he had captured user names and passwords for at least 1000 email accounts, posting the details for 100 of those. Ten days after the initial disclosure, he followed up with information on how he captured the data. Tor (aka The Onion Router) is a system designed to hide the source and destination of internet traffic by routing it through a few intermediate nodes. Software is available for most operating systems and can run in either client or server mode. The Tor network consists of many server nodes that can route this traffic, but it also has special nodes, called "exit nodes" that are the endpoints for traffic within the Tor network. Exit nodes are the ones that actually talk to the server the client was trying to reach, thus they see any traffic exactly as it will be presented to the destination. A Tor client picks a random path through the network, using a directory server to get a list of active nodes. For each hop along that path, it negotiates a separate session key. It encrypts the packet data, along with a destination address, once per node in the path, building up a packet with multiple layers of encrypted information. Each layer can only be decrypted by the proper intermediate node. Each intermediate node only knows about its predecessor, the destination, and the key, so with more than a few nodes, the source and ultimate destination are hidden. The exit node is the last layer of the onion, what it decrypts is the data bound for the destination. Running an exit node for Tor has some risks associated with it, as all traffic that goes to a destination site appears to originate from the exit node host. If the destination gets attacked by a denial of service or other exploit, the exit node operator would seem to be the guilty party. For this reason, Tor servers can determine whether or not they are willing to be exit nodes. What Egerstad did was to volunteer five servers as exit nodes and monitor the traffic that went by. What his exit nodes saw was the traffic bound for various servers, much of it in the clear. He collected authentication for email servers from many users, with the ones he released being embassy workers and members of human rights organizations. He monitored the POP3 and IMAP protocols, specifically looking for keywords associated with governments. By looking at those two protocols, he not only was able to capture passwords, his exit nodes also saw all of the email stream by as it was delivered to the users. This should come as no real surprise, unencrypted email protocols are a security hazard; they should probably go the way of telnet, and be banished from internet usage. What is more surprising, but perhaps shouldn't be, is that people are using Tor to retrieve their email. Tor is not supposed to be a complete privacy solution, and it is not presented that way, but the difference between anonymity and privacy seem to have gotten lost. It is a near certainty that others are doing just what Egerstad did. Governments and criminals – though it can be hard to distinguish between the two at times – both have an interest in monitoring this kind of traffic. Egerstad lists a number of suspicious exit nodes in the Tor network, any or all of which could be scanning the cleartext traffic that streams by. In some ways, Tor is really no different than the myriad routers that internet traffic passes through; each of those presents a point where traffic could be intercepted. Tor is better in that regard, perhaps, because all but the last leg (which, of course, traverses any number of routers) are encrypted. If an encrypted protocol, SSL or an ssh tunnel for example, were used end-to-end, Egerstad's monitoring would not have worked. With proper certificate/key handling, no intermediate node, Tor or router, can decrypt the traffic. It is a bit ironic that one would use a service meant to provide anonymity to log in to a system using credentials that are intended to restrict access to a particular user. It is a bit like renting a room at the No-Tell Motel using your credit card. Presumably, the users had Tor installed and running for other reasons and either didn't know or forgot to turn it off when retrieving their email. Perhaps their email client helpfully retrieves their email every few minutes without their intervention. It should be noted that Tor does not do anything above the protocol level to anonymize traffic. Cookies, browser identification strings and other information can be used to identify who is using the connection to anyone with access to the traffic. Obviously, logging in makes that even easier. Another known threat to anonymity using Tor, even with end-to-end encryption, is timing analysis. If someone can monitor the timing of the packets at the client and those at the server, they can make a statistical correlation between the two. Tor achieved another kind of notoriety, recently, as some of the storm worm spam started pushing it as a solution for internet anonymity. Unfortunately, users who followed the link landed on a fake Tor download page. Downloading the software did not result in any increase in their privacy, it simply installed one of the storm worm variants. It is certainly not the publicity that Tor wanted, but it could, perhaps, lead a few users to the real Tor. It is a dubious honor, but the storm worm herders must believe that the Tor name has some credibility in order to use it this way. Tor is an excellent tool for what it does, but it certainly is not a solution to all internet communication privacy issues. As with most things, users need to understand what they are doing before they can gain the benefits of Tor. By managing the higher level identifying information correctly (perhaps by using something like Privoxy), one can use internet services anonymously with a reasonable level of comfort. Using end-to-end encryption makes it that much better.
New vulnerabilities eggdrop: stack-based buffer overflow
gforge: missing input sanitizing
jffnms: multiple vulnerabilities
lighttpd: buffer overflow
openssh: inappropriate use of trusted cookies
phpmyadmin: multiple vulnerabilities
qgit: arbitrary code execution
samba: incorrect group assignment
wordpress: privilege bypass
xorg-server: local privilege escalation
Updated vulnerabilities aide: checksum errors
apache2: information disclosure
apache: multiple vulnerabilities
apache: cross-site scripting
avahi: denial of service
bochs: buffer overflow
centericq: buffer overflows
clamav: denial of service
clamav: multiple vulnerabilities
cups: denial of service
gpdf: integer overflow
dovecot: privilege escalation
dovecot: directory traversal
evolution: format string error
evolution-data-server: malicious server arbitrary code execution
fetchmail: denial of service
file: integer overflow
firebird: buffer overflow
firefox: multiple vulnerabilities
firefox, thunderbird, seamonkey: multiple vulnerabilities
flac123: arbitrary code execution
freetype: integer overflows
gallery2: multiple unspecified vulnerabilities
gcc: file overwrite vulnerability
gd: buffer overflow
gd: multiple vulnerabilities
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[Arjan van de Ven]](/images/conf/lce2007/arjan-sm.jpg)
![[Matthew Garrett]](/images/conf/lce2007/mgarrett-sm.jpg)
![[Jörn Engel]](/images/conf/lce2007/jengel-sm.jpg)
Jörn Engel gave a talk on the death of hard disks. His core point is
that flash-based storage is faster, requires less power, makes less noise,
and is more robust than rotating storage. It is also more expensive, for
now, but flash is getting cheaper much more quickly. Jörn projects
that flash-based drives will become more economical than hard drives
between 2012 and 2019, depending on which drives one looks at.
![[Chris Mason]](/images/conf/lce2007/cmason-sm.jpg)
![[Seek counts plot]](/images/conf/lce2007/seek-counts.png)