Sponsored link Serve your customers, not your servers, with VERIO Linux VPS. Full-access test-drive here. |
LWN.net Weekly Edition for March 20, 2008Python gears up for 2.6 and 3.0 Things are heating up in the Python world in advance of two major synchronized releases of the language. As it heads towards Python 3000 (aka Py3k or Python 3.0), alongside the transitional version 2.6, the development team is narrowing its focus to just those items that are required for the releases. Along the way, the conversations taking place on python-devel provide a look inside the development and release process decisions that a project needs to make as releases loom. Py3k is the next-generation version of Python, as we described last September. It will not be backward compatible with programs written for Python 2.x in a wide variety of ways. Python 2.6 is an effort to bridge the gap, enabling much of the 3.0 functionality so that new programs can start using it. It can also provide warnings for code that will not work with Py3k. Python 2.6 was originally scheduled for an April 2008 release, in advance of the August 2008 release planned for Py3k. Now the two are slated for synchronized releases, roughly monthly, until the final release now scheduled for early September 2008. The synchronization is seen as important for two reasons as Python's Benevolent Dictator For Life (BDFL) Guido van Rossum outlines:
Not only could
this potentially save the release manager and his assistants some
time, doing the final releases together sends a clear signal to the
community that both versions will receive equal support.
Because Py3k is such a radical change, the 2.x series will continue for a long time. van Rossum's recent PyCon keynote (PDF slides) mentions five years as the time frame for 2.6 to be supported, with 2.7 and 2.8 releases possible. A stable development platform for the next few years is very important for current Python users as is giving them a long time to migrate their code. The third alpha of Py3k was released at the end of February along with the first alpha of 2.6. Additional alpha releases of each are slated for April and May as laid out in Python Enhancement Proposal (PEP) 361. Those are to be followed by betas in June and July with the final release planned for September 3. All of that adds up to a fairly aggressive schedule, but the team seems confident—at least so far. One of the issues that the Python hackers are trying to figure out is how to track the items still left to be done. van Rossum describes the scope of the problem:
In order to make such a tight release schedule we should try to come
up with a list of tasks that need to be done, and prioritize them.
This should include documentation, and supporting tools like 2to3. It
should include features, backports of features, cleanup, bugs, and
whatever else needs to be done (e.g. bugbot maintenance).
No one had any major objections to van Rossum's suggestion of using the bug tracker to track the tasks, with Christian Heimes pointing out:
Despite the url bugs.python.org it's an issue tracker and not a bug
tracker. We track patches, feature requests, ideas and bugs in the same
tracker.
The bug tracker allows for different priorities to be set on bugs (or tasks) that are entered into it, which led van Rossum and others to wonder about the proper usage of that field. One of the problems is distinguishing between issues that must be addressed before the next release versus those that must be addressed sometime before the final release. In some sense, both are "critical" and "show-stopping" (depending on which show you are focused on). Brett Cannon reported the scheme they came up with:
So "release blocker" blocks a release. "Critical" could very easily
block a release, but not the current one. "High" issues should be
addressed, but won't block anything. "Normal" is normal. And "low" is
for spelling errors and such.
This can elevate bugs that are relatively minor, but need to be handled before a final release, into a category that inflates their importance. But, not elevating the bugs can lead to them incorrectly being set aside for a later release. van Rossum wondered about this bug priority "inflation", but it is the way that 2.6/3.0 release manager Barry Warsaw wants to handle things:
Critical is the right one to use.
Neal and I will basically be moving
issues between 'release blocker' and 'critical' with the former
meaning this issue blocks the upcoming release.
Other projects or project managers might make different decisions on how to handle bug priorities, but the important thing is to make a reasonable decision quickly. Once that was done, the tasks were added to the tracker and could be prioritized correctly within the framework and without a lot of hand-wringing about which way is "best". It is an important skill for project managers of all kinds to learn. Things are progressing rapidly on python-devel these days—not surprising with two major releases due in less than six months. There is a lot of work to be done, but the Python hackers aren't shrinking from those tasks. In addition, the team has also been able to change their processes as needed to support their tight schedule. With hard work and a bit of luck that should put Py3k and its 2.6 sibling on our development machines by autumn.
Who maintains dpkg? The Debian project is known for its public brawls, but the truth of the matter is that the Debian developers have not lived up to that reputation in recent years. The recent outburst over the attempted "semi-hijacking" of the dpkg maintainership shows that Debian still knows how to run a flame war, though. It also raises some interesting issues on how packages should be maintained, how derivative distributions work with their upstream versions, and what moral rights, if any, a program's initial author retains years later.Dpkg, of course, is the low-level package management tool used by Debian-based distributions; it is the direct counterpart to the RPM tool used by many other systems. Like RPM, it is a crucial component in that it determines how systems will be managed - and how much hair administrators will lose in the process. And, like RPM, it apparently causes a certain sort of instability in those who work with it for too long. Ian Jackson wrote dpkg back in 1993, but, by the time a few years had passed, Ian had moved on to other projects. In recent times, though, he has come back to working on dpkg - but for Ubuntu, not for the Debian project directly. One of his largest projects has been the triggers feature, which enables one package to respond to events involving other packages in the system. This feature, which is similar to the RPM capability by the same name, can help the system as a whole maintain consistency as the package mix changes; it can also speed up package installations. Triggers have been merged into Ubuntu's dpkg and are currently being used by that distribution. The upstream version of dpkg shipped by Debian does not have trigger support, though, and one might wonder why. If one listens to Ian's side of the story, the merging of triggers has been pointlessly (perhaps even maliciously) blocked for several months by Guillem Jover, the current Debian dpkg maintainer. So Ian concluded that the only way to get triggers into Debian in time for the next release ("lenny") was to carry out a "semi-hijack" of the dpkg package. By semi-hijack, Ian meant that he intended to displace Guillem while leaving in place the other developers working on dpkg, who were encouraged to "please carry on with your existing working practices." Ian also proceeded to upload a version of dpkg with trigger support, and without a number of other recently-added changes. It is worth noting that all of this work went into a separate repository branch, pending a final resolution of the matter. So when the upload was rejected (as it was) and Ian was deprived of his commit privileges (as he was), there was no real mess to clean up. Those wanting a detailed history of this conflict can find it in this posting from Anthony Towns. It is a long story, and your editor will only be able to look at parts of it. One of the relevant issues here is that Guillem Jover appears to be a busy developer who has not had as much time to maintain dpkg as is really needed. Since the beginning of the year, he has orphaned a number of other packages (directfb and bmv, for example) in order to spend more time on dpkg. But, as a result of time constraints, a number of dpkg patches have languished for too long. While this was happening, Guillem put a fair amount of the time he did have into reformatting the dpkg code and making a number of other low-level changes, such as replacing zero constants with NULL. Ian disagrees strongly with the reformatting and such - unsurprisingly, the original code was in his preferred style. And this is where a lot of the conflict comes in, at two different levels. Ian disagrees with the coding style changes in general, saying:
Everyone who works on free software knows that reformatting it is a
no-no. You work with the coding style that's already there.
Many developers will disagree on the value of code reformatting; some projects (the kernel, for example) see quite a bit of it. Judicious cleaning-up of code can help with its long-term maintainability. All will agree, though, that reformatting can make it harder to merge large changes which were made against the code before the reformatting was done. This appears to be a big part of Ian's complaint: unnecessary (to him) churn in the dpkg code base makes it hard for him to maintain his trigger patches in a condition where they can be merged. Code churn is a part of the problem, but Ian's merge difficulties are also a result of doing the trigger work in the Ubuntu tree rather than in Debian directly. Ian did try to unify things back in August, but that was after committing Ubuntu to the modified code. Ubuntu's dpkg is currently significantly different from Debian's version, and, while one assumes that, sooner or later, Debian will acquire the trigger functionality, there is no real assurance that things will go that way. Dpkg has been forked, for now, and the prospects for a subsequent join are uncertain. Ian also asserts that, as the creator of dpkg, he is entitled to special consideration when it comes to the future of that package. His semi-hijack announcement makes that point twice. But one of the key features of free software is this: when you release code under a free license, you give up some control. It seems pretty clear that Ian has long since lost control over dpkg in Debian. So who does control this package, and how will this issue be resolved? Certainly Ian's hijack attempt found little sympathy, even among those who think that dpkg has not been well maintained recently. There are some who say that the disagreement should be taken to the Debian technical committee, which is empowered to resolve technical disputes between developers. But faith in this committee appears to be at a low point, as can be seen in this recent proposal to change how it is selected:
It's been pretty dysfunctional since forever, there's not much
that can be done internally to improve things, and since it's
almost entirely self-appointed and has no oversight whatsoever the
only way to change things externally is constitutional change.
Meanwhile, the discussion has gone quiet, suggesting that, perhaps, it has been moved to a private venue. The dpkg commit log, as of this writing, shows that changes are being merged, but triggers are not among them. It is hard to imagine that the project will fail to find a way to get the triggers feature merged and the maintenance issues resolved, but that does not appear to have happened yet.
Installfest generates 350 Linux computers for schools On Saturday March 1st, Untangle and the Alameda County Computer Resource Center (ACCRC) organized the first of what is hoped to be many "Installfest for Schools" events. It took place at four San Francisco Bay area locations (San Francisco, Berkeley, San Mateo and Novato) and refurbished 350 older computers with Ubuntu for northern California schools. The primary goal of the installfest was to give children in disadvantaged neighborhoods the same access to technology that students in wealthy school districts grow up with. However, the event was also about curbing waste. 132 million PCs were bought in the year 2000 alone and none of them can run Vista. But older hardware works great with GNU/Linux and extending the life of these PCs will keep thousands of tons of toxic electronic waste out of the landfill. And let's not forget about budgetary waste. With many states facing budget crises that will inevitably force deeper classroom spending cutbacks, why should our schools to spend their scarce resources on proprietary software licenses? In fact, cutbacks may create an incredible window of opportunity for the GNU/Linux desktop movement to establish itself within schools. The installfest drew approximately 130 free and open source software community volunteers across the four locations. We started with over 1,000 older, discarded computers that had been collected by ACCRC through donations from the general public, local businesses and municipal governments. Some of the computers were smooth sailing: they met the hardware specification, had all of the necessary components and installed without any problems. Other computers had software install problems, but those were easy to solve because so many of the Bay Area's most hardcore free and open source software gurus participated and with their combined expertise, no error message went unattended to. The rest of the computers required a little more care, as many of them were missing a hard drive, NIC or enough RAM to run Ubuntu. Yet, by disassembling problematic boxes it was easy to form a pool of spare parts that could then be stitched back together to create working computers. The week after the installfest, ACCRC put the finished systems through a 72-hour burn-in test and we now have 350 computers that have already started being donated to schools. The Ascend School in Oakland received the first batch of nine computers. Other schools that have received open source computers from the ACCRC include:
Computer hardware and software specifications
The minimum specifications for each computer were an 800mhz processor (PIII or AMD), 256MB Ram and a 20 GB hard drive, but we were pleasantly surprised to find a handful of P4 processors in the mix as well. One location even received a batch of 6 dual core systems with elegant slim cases—who throws those out and what else are they looking to get rid of?—but ironically we couldn't install them during the event because they were only equipped with DMS-59 DVI ports that required special monitor cables. Each system received a fresh copy of Ubuntu 7.10 desktop with the latest apt-get upgrade applied as of February 27, 2008. Because the computers were going into schools with little or no GNU/Linux expertise, it was important to try and create a positive first experience so we worked with Creative Commons to package samples of pictures from Flickr and music from Jamendo to show off the fun side of the donated computers. No Starch Press also donated PDF copies of Ubuntu for non-Geeks that were loaded on to each computer so that help for common support questions was never more than a click away. Install specifications
Each location was set up with 10 to 40 workstations that had permanent
keyboards, mice, monitors and cables so that the volunteers only had to
move the desktops themselves back and forth. The process was started by
booting from custom install CDs and the packages were applied over the
network via apache http web servers. The custom CDs were optimized to make
the Ubuntu OS installation as fast and easy as possible. Physically
placing the CD into the drive and booting from disc was really all that was
required because the additional content from Creative Commons and No Starch
Press were bundled as Debian packages that were automatically installed via
the network just like the other Ubuntu updates and patches.
The installfest networks were based on dual Pentium III servers with a RAID array and Gigabit network cards plugged into a 24-port Gigabit switch. It was important to have a fast setup because updating as many as 40 systems at once placed a heavy load on drives and network connections. Electricity was also a concern as most of the outlets available had 15 or 20 Amp circuits. Given the intensity of the installation/reboot workload and the relatively power inefficient CRT monitors, we drew the line at 5 workstations per 15 Amp circuit because an extra machine might have fit, but blowing the circuit breaker would have caused a big disruption—especially if the breaker happened to be in a locked closet. Community goes the extra mileWith 130 volunteers showing up, Untangle and ACCRC really had a lot of help in pulling the Installfest for Schools off. However, the community did far more than just show up, our volunteers really went the extra mile to save the day on as we stumbled across a handful of unexpected hiccups. One particularly inspirational moment was when the San Mateo location ran out of computers, our volunteers drove their own cars across the Bay to pickup extra hardware rather than close the location early! We also owe a debt of gratitude to 3 members of the San Francisco Linux Users' Group (Christian Einfeldt, Jim Stockford and Daniel Mizyrycki), who worked long hours to set up and clean up that location. We also received lots of help from free and open source software related organizations. Mozilla in particular really stepped up to the plate by blogging about the event and then bringing schwag and pizza for all 130 volunteers! But Mozilla wanted to get their hands dirty as well and Mozilla team members showed up to lend a hand at each location. Creative Commons and the No Starch Press helped put together content. Also, O'Reilly, OSI, the Linux Foundation, Sun and Canonical really helped get the word out with supportive blog mentions that encouraged participation as well. Future plansMoving forward, Untangle and ACCRC hope to continue organizing bigger and better Installfests for Schools. Our goal is to turn the one-time regional event into a distributed national event occurring on a regular basis. If we're able to find some friendly organizations to help out, we may even be able to go international. Stay tuned because you'll be hearing from us sooner rather than later about the next Installfest for Schools. Anyone wishing to help should stay informed by signing up for the installfest mailing list. As we move more into a distributed national event, we need all of the help that we can get identifying local schools, old computer donors and feet on the street volunteers to make sure everything goes smoothly. That work will be coordinated on the mailing list. [ Andrew Fife, of Untangle, is one of the organizers of the project. ]
Page editor: Jonathan Corbet Security Breaking CAPTCHA Perhaps someday it will be considered discrimination against a sentient, but these days a way to distinguish between programs and humans is required for many web-based applications. Keeping spambots from posting comments in weblogs or other bots from signing up for a web service are two of the most common applications for separating humans and bots. As has often been the case in the past, though, when the stakes are high enough, attackers will find ways to circumvent barriers like this. The most common means of testing for humans in web site sign-ups and the like is a CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart). Typically these are images that contain some text that has been mangled so that it is still recognizable by humans, but not by programs—at least that is the theory. Variations on the theme include asking math or "common sense" questions that programs will supposedly not be able to figure out—more likely no attacker has had enough interest breaking them. Serious CAPTCHAs tend to use images that can be created on the fly, giving nearly infinite variety. Some of the most sophisticated CAPTCHAs are those used by various free web mail services: Hotmail, Yahoo, and Gmail. These services provide quite a bit of storage that might be of use to an attacker, but they also lend their reputation to mail that gets sent from those accounts. Domains like yahoo.com and gmail.com are very unlikely to be blacklisted. Mail coming from those domains may also score lower in various spam testing rules, which may be exactly what an attacker is looking for. Various techniques have been tried in the past to circumvent CAPTCHAs, with the most successful ones using humans. It seems that many folks will happily solve CAPTCHAs in order to view pornography or for cash. Over the last year, though, CAPTCHA-breaking programs have started to appear. In a very detailed report, Websense presents evidence that Gmail's CAPTCHA has been cracked. Earlier reports indicate that attackers have cracked Yahoo, Windows Live, and Hotmail CAPTCHAs as well. Cracked does not mean 100% success rate—humans cannot even achieve that—it just needs to work often enough to provide the attackers with the accounts they want. These programs use some image processing and optical character recognition (OCR) techniques to decipher the puzzle, removing humans from the equation entirely. Typical success rates are in the 20-35% range. For attackers with botnets available to spread out the work, this could yield an amazing number of accounts in relatively short order. CAPTCHAs have a number of bad characteristics: they are annoying to most and unusable by those who are visually impaired. Yet they are pervasive. Alternate techniques using audio have so far been found wanting; a more interesting method is Asirra from Microsoft Research. Asirra uses 3 million images of dogs and cats from animal shelters that have been categorized. The test then shows a dozen random images from the database and asks the "human" to select all the cat photos. This would seem much more difficult for a program to handle. The picture database would need regular updates to thwart attackers just collecting all the images and doing their own categorization—perhaps with help from porn viewers or poor folk. Also, computer recognition systems will someday be able to recognize dogs and cats. It is a difficult problem to solve, but one that needs to be addressed. Systems like OpenID are not enough—it is not what they were designed for—as there is nothing stopping bots from having OpenIDs. Some mechanism that would allow reputation or trust to accumulate on a given ID might help prove that its holder is a human—or at least a well-behaved bot. Designing a reputation service that is decentralized will also be difficult, but it is the right direction for solving these kinds of problems.
Security news CERT C Secure Coding Standard: last call for reviewers CERT is working put put together a set of secure coding standards - essentially a long set of rules on how to avoid common mistakes which can create security vulnerabilities. The standard for C is headed for a 1.0 release around mid-April. Before that happens, CERT is asking for one last round of comments; see the associated web site for more information.
New vulnerabilities backup-manager: password disclosure
cups: heap overflow
dovecot: multiple vulnerabilities
horde3: file execution and disclosure via directory traversal
ikiwiki: cross-site scripting
krb5: multiple remote vulnerabilities
krb5: double-free flaw
krb5: denial of service
ldapscripts: password disclosure
lighttpd: information disclosure
LIVE555 Media Server: denial of service
phpMyAdmin: cross-site scripting
roundup: unspecified vulnerability
sarg: arbitrary code execution
silc-toolkit: stack-based buffer overflow
smarty: arbitrary code execution via crafted search string
unzip: arbitrary code execution
xine-lib: buffer overflow
Updated vulnerabilities cairo: integer overflow
MySQL: privilege escalation
SDL_image: buffer overflows
Sun JDK/JRE: multiple vulnerabilities
Xorg: multiple vulnerabilities
apache: cross-site scripting
apache: several vulnerabilities
asterisk: possible SQL injection
audacity: insecure tmpfile handling
bind: off-by-one error
boost: denial of service
cacti: multiple vulnerabilities
clamav: arbitrary code execution
clamav: arbitrary file overwrite
clamav: heap corruption
cups: denial of service
cups: multiple vulnerabilities
dbus: privilege escalation
debian-goodies: privilege escalation
evolution: format string vulnerability
exiftags: multiple vulnerabilities
firebird: multiple vulnerabilities
firebird: buffer overflow
firefox: multiple vulnerabilities
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||