By Jonathan Corbet
May 4, 2011
Your editor does not normally need any computerized assistance when it
comes to knowing what needs to be done next; it's usually just a matter of
dealing with whatever is on fire at the moment. Still, there are times
when it would be nice to have a task management tool with a longer view, where a
different set of priorities can be expressed. Recently it became time to
work on one of those longstanding todo items and examine
Astrid as a possibility for that role.
A task manager which works with the phone obviously makes sense; one cannot
always be at the keyboard when trying to figure out what to do next.
Anybody who has an Android-based phone has a long list of task management
applications to
choose from; simply working through the list is the sort of todo item that
can make one decide to just drink beer and mess around on the net instead.
So why pick Astrid out of the pile of applications? It comes down to two
reasons: it comes well recommended, and it's free software - Astrid is
licensed under GPLv2 and can be downloaded from github.
Astrid provides the kinds of features that one might expect from this type
of application. One can enter tasks and assign an importance and a
deadline to each. This is the 21st century, so, naturally, tasks can be
tagged as well. A timer mechanism allows the user to track how much time
goes into the accomplishment of each task - a useful feature, perhaps, if
that time will subsequently be billed for. And, naturally, one can mark
tasks as being done.
There are various options for how obnoxiously Astrid
should nag as deadlines approach and just how hard it should be to
make the notifications go away. A true procrastinator with a masochistic
streak can configure the application to a phone-smashing level of
annoyance. For the rest of us, simple notifications and, possibly, a
desktop widget will do.
In general, Astrid works as advertised. Entering new tasks is easy, and
the notifications work as expected. There is a whole set of options for
filtering the display of tasks which must be useful to some people;
work-oriented tasks can be separated from shopping lists, for example. The
desktop widget works but, like a number of other Android applications, it
makes poor use of the space which is given to it. About half of the
allotted space contains task information - truncated to the point of near
uselessness. Some thought put into the organization of that widget might
make it worth dedicating scarce desktop space to. As it is, your editor
couldn't justify deleting something else to make space for it.
One of the sticking points with phone-based task managers is the simple
inconvenience of entering tasks through an on-screen keyboard. The
motivation to turn an "oh, yeah, I have to do X" moment into a documented
task fades when it requires using a small screen and a tiny keyboard. The
obvious answer to this problem is synchronization with some sort of online
service, preferably one which enables access over the web from real
computers as well. To this end, Astrid will synchronize with the Google
mail task list or with an online service called Producteev. Gmail provides simple
online storage and synchronization, while Producteev has (for a fee) all
kinds of options for shared task lists, report generation, and more. The
fact that use of SSL is a paid option on Producteev is not particularly
encouraging, but so it goes.
The online documentation talks about synchronization with Remember the milk, but version
3.7.3.1 does not appear to actually support that service.
Your editor only tried out the Google-based synchronization. It works as
expected; tasks move nicely between Google and the application. Of
course, using this feature requires providing one's Google password to
Astrid and providing one's task lists to Google, but, after all, it's not
like Google knows anything else about us anyway.
This kind of synchronization is useful, and, honestly, worrying about storing
some todo items on Google's server seems pretty silly if one is willing to
take the privacy-reducing step of walking around with an Android phone in
one's pocket. Still, it would be nice to be able to synchronize
applications like this to a more private backing store - preferably one
which does not require a great deal of pain to set up. We need a
mechanism by which one can install a simple backup appliance onto a server
anywhere on the net; otherwise we'll always be dependent on services
provided by companies whose interests will not always align with our own.
(Along these lines, it's worth noting that Astrid, in its default
configuration, phones home to a service called Flurry with "anonymized" information about
how the application is used. This reporting can be turned off in the
settings menu.)
While Astrid is free software, it supports add-ons which are not. The Astrid store offers a couple of
paid add-ons. One of which offers timers, "bigger better widgets,"
voice-based features, and "No Ads!" A fair amount of time playing with
Astrid has not turned up an advertisement yet, but, one assumes, that could
happen at some future time. There is also a "Locale add-on" which allows
tasks to be keyed on the phone's location.
Some of these features, seemingly, could be added by anybody willing to dig
into the Astrid source. Certainly, any sort of advertising antifeature
could easily be subtracted that way. One can only assume that patches to
that end would not be accepted upstream. Some quick searching failed to
turn up anybody who has rebuilt Astrid with a different feature set; it is
nice to know that a fork is a possibility, though, should Astrid's masters
take an overly user-hostile direction.
There could be changes in store for Astrid; recently its developers announced that
they had "brought in some external funding" and will be
focusing more on Astrid in particular. There will be a new synchronization
service at Astrid.com, an iPhone version,
and more. Whether this money - and the revenue expectations that certainly
come with it - will push Astrid in a less friendly direction remains to be
seen. Hopefully the open-source nature of the code will help to keep the
developers focused on keeping users happy.
For now, there seems to be little to worry about; the only visibly
commercial aspect of current Astrid builds is the "add-ons" button in the
main menu. Astrid's developers have been successful in creating a task
management application which is appealing to a wide range of people. With
luck, it will just get better now that more developer time will be made
available.
Comments (11 posted)
May 2, 2011
This article was contributed by Josh Berkus
The first beta release of PostgreSQL 9.1, the next annual release of the well-known open source relational database, came out on May 2nd. The new version contains many new features sure to please the hearts of database geeks around the world. There are enough new features, in fact, that we can't cover them all in this article. Here's four of the more interesting ones:
Transaction-controlled synchronous replication
Last year, PostgreSQL 9.0 introduced asynchronous binary replication in PostgreSQL for the first time. Building on that foundation, version 9.1 includes an option for synchronous replication as well. Synchronous replication means that transactions do not return to the application until they have been received both on the master server and on a replica server. This ensures that data from any committed transaction cannot be lost even if the master server goes offline permanently.
This feature culminates a three-year-long development effort by NTT Open Source and 2ndQuadrant, which will allow Japanese telecommunications giant NTT to finally replace most of their Oracle servers with PostgreSQL. The PostgreSQL clustering project PostgresXC is the remaining part of this replacement effort.
There is also a substantial penalty in response time for synchronous
replication, since the transaction needs to be written to disk on two
servers before returning. To alleviate this, PostgreSQL goes beyond the
synchronous replication offered by other database systems, and offers the
ability to control whether commits are synchronously or asynchronously
replicated on a per-transaction basis. This allows application developers
to distinguish between critical data which must not be lost (like financial
transactions), and less critical data for which response time is more
important, to optimize the performance of the system.
Each replica (or "standby") gives itself a name in its
recovery.conf file when it connects to the master:
primary_conninfo = 'host=master01 user=replicator application_name=replica1'
Then the master sets a priority list of synchronous replicas in its postgresql.conf file:
synchronous_standby_names = 'replica1'
Then you can commit a transaction as synchronous or asynchronous:
-- commit synchronously to the standby
SET synchronous_commit = 'on';
UPDATE user_balance SET balance = 573.29 WHERE user_id = 40021;
-- messages are not important, send asynchronously to the standby
SET synchronous_commit = 'local';
INSERT INTO user_messages VALUES ( 40021, 57843, 'HI!' );
PostgreSQL also supports having a priority list of synchronous replicas,
thus supporting higher-availability configurations. The new
pg_stat_replication database view allows database administrators (DBAs) to monitor the status of database replicas, and other administrative settings control what to do if the standby goes offline.
In future versions, PostgreSQL plans to support different synchronization modes, as well as the ability to have "quorum" synchronization. The latter is for "synchronize to two out of these five servers" replication, for improved response time without lowering data integrity.
Unlogged tables
The burgeoning popularity of non-durable "NoSQL" databases like Redis and
MongoDB, along with the well-established Memcached, have demonstrated that there is a class of data for which data loss after a crash is less important than response time. EnterpriseDB developer Robert Haas has added a feature to PostgreSQL to handle this kind of data: unlogged tables.
The word "unlogged" refers to the transaction log, which guarantees durability of writes. With an unlogged table, if the database server shuts down, on restart the table will be empty. However, by not syncing to disk, you can write to unlogged tables up to 20 times as quickly as to durable tables. They can also be thought of as global temporary tables or "in memory" tables.
Unlogged tables are intended to hold data which is high-volume but not very valuable. For example, many PostgreSQL users currently log website user session activity to Memcached. Unlogged tables are also useful for holding raw bulk load data during batch processing in data warehouses. Both of these uses will simplify life for PostgreSQL users by allowing them to reduce the number of single-purpose databases in their application stack.
SELinux integration
Another multi-year development effort that is being first released in
PostgreSQL 9.1 is the unification of data access control in PostgreSQL and
SELinux. NEC programmer Kaigai Kohei has spent the last three years
developing SE-PostgreSQL, which integrates the concept of label-based data access rules into PostgreSQL. PostgreSQL is the first SQL database system to have this level of integration.
This feature is intended for high-security environments where even the
DBA may have limited access to data, such as for national security databases. Security policies established at the system or network level through SELinux may now be applied to table, view, function, and column permissions. This enables the establishment of a single consistent security policy regardless of whether data is in files or inside the database.
Regular readers of LWN will be aware that SE-PostgreSQL had a lot of trouble getting integrated into mainstream PostgreSQL. Indeed, the SE-PostgreSQL which appears in 9.1 beta is almost completely rewritten from the original version. First, with the help of other hackers, Kohei rewrote all of the calls to SELinux permissions lookups as hooks which are enabled at PostgreSQL compile time, or not, in order to remove any impact on non-SE-enabled installations. Second, all of the management and tools for SE-PostgreSQL were moved to an optional loadable module.
Finally, certain actions had to be exempted from Mandatory Access Control because of difficulty of implementation. Chief among these is row-level access control, since there are a number of unresolved theoretical issues about how to implement it. This means that, of course, Kohei still has work to do.
Extensions and PGXN
Speaking of optional modules, one of the primary strengths of PostgreSQL has always been its extensibility. Users and developers have defined data types, operators, functions, and aggregates to support a variety of types of specialty data, including genomic, inventory, mathematical, astronomical, time series, and geological data. Other plug-ins support integration with queues, compatibility with other database systems, and experimental new features. The most famous plug-in to PostgreSQL is probably the PostGIS geographic database.
However, most of these "plug-ins" to PostgreSQL have been difficult to find, install, back-up, and upgrade, limiting their use or even causing users to reinvent them several times. That's changing with version 9.1.
First, Dimitri Fontaine of 2ndQuadrant has created a packaging concept in
PostgreSQL 9.1 called Extensions. By packaging a collection of database
objects as an Extension, the administration of adding and removing plug-ins, and most importantly upgrading them, becomes easily scriptable. Users can even create their own extensions for in-house database customizations which need to be versioned.
Second, David Wheeler of PostgreSQL Experts created PGXN, "The PostgreSQL EXtension Network". PGXN is patterned on online repositories for programming languages like Perl's CPAN and Ruby GEMs. PGXN will give PostgreSQL users a central place to find and download PostgreSQL plug-ins not distributed with the core database. Currently in development is a unified download and installation tool with dependency tracking, like CPAN or apt-get.
PostgreSQL development
The PostgreSQL project has been experimenting for the last couple of years with changes to its development cycle, patch review, and release processes. The biggest problem the project struggles with is the constant torrent of patches and feature ideas being submitted, which has been growing at a much faster rate than the pool of reviewers and committers has. 9.1, for example, will contain more new major features than 9.0 did. 9.0, in turn, had more features than 8.4.
As a result, contributors to the project frequently discuss tinkering with
the development cycle in order to optimize reviewer, contributor, and
developer time. A recent
discussion grappled with those topics.
Should the project branch off version 9.2 right after beta or wait until the release candidate? Historically, PostgreSQL has refused to begin development on a new version before development on the prior version was completed, and that seems likely to continue to be the policy. PostgreSQL developers have found that working on more than one version at a time causes a great deal of backporting, as well as lack of attention to fixing bugs.
Should PostgreSQL continue to have CommitFests (periods of
intensive focus on patch review and merging rather than
development) every other month or move to a one-week-a-month cycle? Doing a week-a-month would give committers more of their own time as well as giving developers faster feedback. However, there are some serious doubts as to whether it's even possible to implement given the distributed, part-time nature of most of PostgreSQL's contributor base.
What should the schedule be for 9.2 development? This depends, of course, rather strongly on answers to both of the above. 9.0 and 9.1 both began development in July and released 12 to 14 months later. Some contributors feel that a regular annual schedule is a benefit while others don't find it important. This discussion is far from over, and may even result in changes which are tried and then reverted if they don't work.
In related news, the PostgreSQL code base has passed 1 million lines of code with version 9.1. And long-time project contributor Magnus Hagander has been elected to the PostgreSQL Core Team.
There are, as mentioned, many other features in this beta release, which
you can read about in the release
notes. Several of these features are innovations which will appear in
PostgreSQL before any other relational database. Since this is a beta version, it is considered not yet ready for production use. The PostgreSQL developers expect the final version to be out in two to four months, depending on testing and bug-fixing.
Comments (13 posted)
By Jonathan Corbet
May 3, 2011
Back in 2006, LWN looked at the rather
complicated story around the maintainership of the RPM package manager.
Given the importance of this tool for any RPM-based distribution, the
lack of a clear story on how it was being maintained was somewhat
discouraging. Later that year, the Fedora project
announced the creation of a new,
community-oriented project around RPM. Since then, things have been on the
quiet side, but recent events show that the RPM story has not yet run its
course.
The above-mentioned RPM project lives on at rpm.org; the 4.9.0
release was announced at the beginning of March. The code is actively
maintained and sees the addition of some small features, but the project does not
show any real signs of having big plans for the future. Only a
handful of committers show up in the repository; almost all of them
work for Red Hat. By all appearances, RPM is at least halfway into
maintenance mode.
But rpm.org is not the only RPM out there; a fork exists at rpm5.org. This version is maintained primarily
by Jeff Johnson, a former Red Hat employee who has set out to create a
better RPM outside of the company. This version has focused on wider portability,
has added features like new compression formats, and a number of other
things; the full feature list is not exactly easy to find, and searching for the
significant entries in the changelog is not
a project
for anybody who is short on time. One of the key features, though, seems
to be transactional
package management (also called "RPM ACID"); the idea here is to ensure
that any RPM operation either succeeds or fails fully, regardless of what
may happen in the middle. According to Jeff, killing an rpm4 operation in
the middle can
leave a corrupted system behind; rpm5 is meant to eliminate that
possibility.
The rpm5 fork arguably has more development activity and some interesting
new features. That said, it has remained a relatively obscure project; it
has been picked up by distributions like Alt Linux, ArkLinux, and Unity
Linux, but has not been seriously considered by the larger distributors.
That has changed, though, in recent months; the Mandriva 2011 plans
include a switch to rpm5. That has brought a lot more attention to this
fork, with some interesting outcomes.
It has not always been clear to everybody in the Mandriva camp that this
switch is a good idea; back in November, Eugeni Dodonov started an extensive discussion by asking about the
reasoning behind this change:
As far as I know, rpm4 is supported by Fedora/RedHat/CentOS and
SUSE/OpenSUSE among the major vendors. Rpm5, on its hand, is used
by AltLinux and Unity Linux. RPM4 is mostly stable, with long-time
pending issues and its behavior is well-known. RPM5 is under
constant development, with many features and without any large
install base.
Many others spoke up on both sides of the issue; the discussion grew rather
heated and unpleasant at times. The decision to go with rpm5 seemed to be
motivated by a number of considerations. Some of the new features were
attractive to Mandriva. The development community for
rpm5 was seen as being more active and open than for rpm4, which was seen
as dominated by Red Hat and relatively closed. Mandriva was carrying a
long list of patches against rpm4 which, evidently, could be upstreamed
into rpm5 but not into rpm4. The notion of compatibility with the other
large RPM-based distributions was seen as illusory at best; each
distribution was seen as heavily patching rpm4 for its own needs and it has
always been difficult to do non-native installations of RPM packages. And,
importantly, Mandriva's RPM maintainer, Per Øyvind Karlsen, wanted to go that way:
I've made it clear several times that I've planned on going with
rpm5, and given that I'm the maintainer, have the best knowledge of
and this change really shouldn't affect people in a negative way if
you look beyond the reckless updating from testing. As I am the one
to maintain it, and it shouldn't have inconveniences pushed on
others, I find it my own decision to make whether I want to
maintain a totally messed up rpm.org version which is bogged down
with patches and pain to regenerate for every release.
So that is how it went. Per Øyvind's assertion that the change "really
shouldn't affect people in a negative way" turned out not to be quite the
case, though; Mandriva's "cooker" mailing list has been dominated by
discussions of rpm5-related issues ever since. Other small transitions - to
systemd, for example - have not generated a fraction of the traffic.
A transition of this type was never going to be easy; it has been further
complicated by the facts that integration with Mandriva's
higher-level tools has been a big job and rpm5 has had some growing
pains in the process of scaling up to a distribution of the size and
complexity of Mandriva.
It must be said that, through this process, Jeff has clearly put in a
massive amount of time supporting Mandriva. One could have easily gotten
the impression
that he had been hired by the company as part of this project. As problems
were reported, Jeff worked to solve them. It is a rare development project
that will put that much effort into supporting its users - even
high-profile users; Jeff has performed a real service for Mandriva.
That support faltered a bit at the end of April, when Jeff abruptly announced that the rpm5 mailing list archive
(and much of the rest of the site) had been taken down. His responses to
queries about why this happened are best described as abusive; by his own
admission, he wanted to stop the discussion as quickly as possible. The
site has since returned; there is also a new rpm6.org which, for now,
redirects to rpm5.org.
What is going on is far from clear, but there are hints in this message from Jeff where he
accuses an ex-Mandriva developer of being "responsible for destroying the
rpm5 brand." He is clearly
tired of dealing with problem reports and complaints about rpm5, many of
which, he says, are not really related rpm5 problems. But much of the
trouble seems to come from the fact that Mageia - a fork of Mandriva - has
not followed Mandriva into the rpm5 transition. Mageia's position, as described by Anne Nicolas, seems reasonable:
the Mageia developers are busy enough just trying to get a releasable
distribution together. Given all that needs to be done, throwing in a
low-level package management system change just didn't seem like the best
idea.
It would appear that there is some bad blood remaining between the Mandriva
and Mageia camps. There are hints of non-public discussions that have been
rather less friendly than what can be seen on the public lists; Jeff and
rpm5 may well have been dragged into some of that. At least, that's what
one might conclude from comments like:
You've had *MONTHS* to make peace with rpm5 and mostly
haven't. rpm5 is being cited as a "fiasco" and is blamed for most
anything you don't like. What you don't like has nothing
whatsoever to do with this "tourist with camera" wandering the
dungeons hacking on RPM.
I can tell you that a "fork" is a huge waste of time & energy from
first hand experience. I can also say that the M&M distros are
more similar than different, and strongly suggest that you make
peace somehow and move on.
Since then, the tempest seems to have mostly passed, and Jeff is back to
talking about making rpm5 work better with Mandriva. But this incident has
been a wakeup call for some people in the Mandriva community who now feel
they have cause to worry about the foundation on which the 2011 release is
being built.
It is going to be interesting to watch what happens from here. Mandriva
appears to be well committed to the rpm5 transition at this point, and,
seemingly, things are beginning to stabilize. If rpm5 performs well in the
final Mandriva 2011 release, it could motivate questions from users of some
of the other distributions on why they are stuck with the "older" version.
Alternatively, users could see the pain Mandriva has been through and, if
the result doesn't appear to be worth it, they may decide that they're
happier with their relatively boring rpm4. Either way, it seems that this
particular drama has not played itself out yet.
Comments (89 posted)
Page editor: Jonathan Corbet
Security
Seth Schoen began his Getting HTTPS Everywhere talk at
Linuxfest Northwest (LFNW) with an optimistic take. All that's required to
have HTTPS Everywhere is
that a few million more sites deploy HTTPS, the ones that have deployed
HTTPS fix their implementations, find a way to fix the problems with the Certificate Authorities (CAs), and it's done. Piece of cake.
Perhaps it's not that simple after all. Schoen, senior technologist for the Electronic Frontier Foundation (EFF), explained early in the talk that the EFF isn't pursuing HTTPS adoption everywhere for grins — Schoen talked about Wireshark and Firesheep, and the ease with which people could snoop on others' Web traffic. He used examples of sniffing conversations over VoIP and other traffic, and said that it's "just out of convenience and courtesy" that most of the traffic that goes over a network isn't sniffed and viewed by someone else. However, convenience and courtesy only go so far — there are always those who are willing to go the extra mile to violate others privacy for fun, profit, or other nefarious purposes.
Thus the need for encryption over all connections, and not just for
e-commerce sites, online banking, etc. The EFF and Tor Project released a Firefox extension called HTTPS
Everywhere last year to help make it easier for users to enforce the
use of HTTPS where it's supported. Schoen says that HTTPS adoption is much
better than a year ago, particularly with popular sites like Google and
Facebook. Even the US Federal Trade Commission (FTC) has called on Web services to start using HTTPS. Many sites now offer HTTPS as an option, though few offer HTTPS as the default.
So far, Schoen says that they estimate 500,000 users of the extension
— though that is merely a drop in the bucket when you consider the
number of people using Firefox (which passed 100 million downloads a few
weeks ago). The extension now supports more than 700 sites, which may sound
paltry until one realizes what's involved. It is not as simple as simply
adding "s" to the "http" in a request, but actually requires users to
verify that the same content is available at the URL if it is requested as
"https" instead.
In some cases, like Wikipedia, it is not. For instance, requesting "http://www.wikipedia.org" will call up (as one might expect) the front page of Wikipedia. Requesting "https://www.wikipedia.org" gives an error. Users who want secure access to Wikipedia want "secure.wikipedia.org" instead. Requesting the Mozilla homepage without the "www" gives an error for an untrusted certificate, though requesting the HTTPS version of "www.mozilla.org" works fine. In short — too many sites on the Internet do not allow the user to simply assume that HTTPS will work with all links.
So the EFF is looking for more users to help. Schoen called on users to
install HTTPS Everywhere, send bug reports when it doesn't work properly or
sites have changed, and to help write rules for it. Naturally, it would
also help if everyone responsible for a Web site would actually turn on HTTPS.
Users of Chrome and Chromium will be able to take advantage of the HTTPS
Everywhere extension soon. Schoen said that Chrome/Chromium was not
originally targeted because Chrome lacked the APIs necessary for HTTPS
Everywhere. There's also an effort afoot to provide an HTTPS Everywhere Web
proxy. He also gave a shout-out to the DuckDuckGo search engine, which has an
option for rewriting searches so that users will be sent to the secure
version of the resulting sites if available.
HTTPS Now
It doesn't help much to have the HTTPS Everywhere extension if sites don't have a secure version to redirect to. To that end, the EFF is working with Access on a program called HTTPS Now.
This effort includes resources for correctly
deploying HTTPS and the ability to search for sites and see how (or if)
they've deployed HTTPS. It also has a reporting system for users to explain how sites use HTTPS. For instance, users can report the name of the site, whether it only uses HTTPS on some pages or all pages, whether it uses secure cookies, has a valid SSL certificate, the key size of the certificate, and more. The reporting page has a lot of help to guide users who might not understand what a technology is, or how to determine if it's used. For instance, the help page for HTTPS Strict Transport Security (HSTS) explains HSTS and guides users to Qualys SSL Labs which has a SSL Server Test page which will examine a site and provide much of the information they want. (LWN, by the way, gets an overall rating of B from the service.) [ Editor's note: it would seem that accepting weak ciphers is the main thing dragging down our grade, which is something we plan to look into and fix in the near future. ]
SSL Observatory
Part of turning HTTPS on everywhere requires having a certificate — preferably not self-signed if one expects much traffic from users who have no way of verifying the veracity of a self-signed certificate. Not that certificates from CAs are always reliable. Schoen also talked about the SSL Observatory, another effort from the EFF to investigate certificates.
This is no small feat. According to Schoen the effort is trying to
examine all publicly visible SSL certificates on the Internet. This
has required making TLS connections to every IPv4 address. The EFF has
found that certificates are signed by about 650 organizations that are
trusted directly or indirectly by Mozilla and/or Microsoft as CAs.
Schoen says that the CA system has been subject to "a lot of little
scandals," that are worrisome. For example, signing
unqualified domain names like "exchange" instead of
"exchange.host.tld", which is what the CAs are supposed to do. Then there's
the recent Comodo incident where a reseller
of Comodo certificates was compromised and an intruder obtained
certificates for a number of targets. Though the certificates were almost
immediately revoked, it demonstrated a potential problem with the CA and reseller structure.
Schoen noted that the system as it stands is rather fragile — not
surprising given that it was invented by Netscape as a Band-Aid to calm fears about online credit card transactions.
For now, the EFF has been gathering data and examining it on its own. Schoen says that eventually the HTTPS Everywhere plugin would allow users to submit data to the Observatory. He also noted a few other efforts along the same lines, like the Perspectives Firefox extension and Google's certificate catalog.
The combined HTTPS efforts from the EFF and its partner organizations are enormous undertakings. Having all sites on the Internet (or even most) providing secure connections, and helping to reform the current CA mess, could take quite a few years. Pushing the awareness of the need for secure connections outside the tech community that understands the issues at hand will take quite a bit of effort, not just at the user level, but also at the site level. For instance, while Google and Microsoft have HTTPS for their Webmail offerings, Yahoo only offers HTTPS at login — when one logs into Yahoo Mail using HTTPS, they're immediately shunted to HTTP after login.
This will not be an easy fix, but the EFF's efforts are already
bearing fruit. While a half-million users is a drop in the bucket,
it's an impressive uptake for one year's effort. The EFF (and tools
like Firesheep) have helped drive awareness over the last year and
encouraged some major sites to push their users to secure connections,
which is a good start — but not enough. Users would do well to
check out the resources offered by the EFF, to participate in the
Observatory and other efforts as time allows, and push their own
organizations to offer HTTPS everywhere as well.
Comments (22 posted)
Brief items
The problem as I see it is the slippery slope. Because next, the RIAA is
going to want to remotely disable computers they feel are engaged in
illegal file sharing. And the FBI is going to want to remotely disable
computers they feel are encouraging terrorism. And so on. It's important to
have serious legal controls on this counterattack sort of defense.
--
Bruce
Schneier
Time over target can get expensive when aircraft are involved although
it can be kept down to as low as $50/hr or maybe even less so it
wouldn't take much to discover every AP in a whole metro area. A
smallish haul of card numbers resulting from the flights would easily
cover it: I always consider how much an attacker would stand to gain
when considering how likely they are to do something as outlandish as
aerial wireless recon.
--
Tracy
Reed on the Dailydave mailing list (thanks to Mattias Mattsson)
Comments (none posted)
The Tor project has
announced
that it is moving away from its Firefox extension and toward the
maintenance of its own fork of the browser. "
The Tor Browser bug
[fixes] on the other hand are more directly usable by Firefox in its own
Private Browsing Mode, which makes them more likely to merge quicker, and
be maintained long-term. Also, because we are releasing our own
Firefox-based browser, we will also have more control over experimenting
with them and deploying these fixes to our users rapidly, as opposed to
waiting for the next major Firefox release."
Comments (2 posted)
New vulnerabilities
firefox: arbitrary code execution
| Package(s): | firefox |
CVE #(s): | CVE-2011-0079
|
| Created: | May 2, 2011 |
Updated: | May 5, 2011 |
| Description: |
From the Ubuntu advisory:
Boris Zbarsky, Gary Kwong, Jesse Ruderman, Michael Wu, and Ted Mielczarek
discovered multiple memory vulnerabilities. An attacker could exploit these
to possibly run arbitrary code as the user running Firefox. |
| Alerts: |
|
Comments (none posted)
firefox: multiple vulnerabilities
| Package(s): | firefox |
CVE #(s): | CVE-2011-0065
CVE-2011-0066
CVE-2011-0067
CVE-2011-0069
CVE-2011-1202
|
| Created: | April 29, 2011 |
Updated: | October 4, 2012 |
| Description: |
From the Red Hat advisory:
Two use-after-free flaws were found in the Firefox mObserverList and
mChannel objects. Malicious content could use these flaws to execute
arbitrary code with the privileges of the user running Firefox.
(CVE-2011-0066, CVE-2011-0065)
A flaw was found in the way Firefox displayed the autocomplete pop-up.
Malicious content could use this flaw to steal form history information.
(CVE-2011-0067)
A flaw was found in the way Firefox handled certain JavaScript cross-domain
requests. If malicious content generated a large number of cross-domain
JavaScript requests, it could cause Firefox to execute arbitrary code with
the privileges of the user running Firefox. (CVE-2011-0069)
A flaw was found in the Firefox XSLT generate-id() function. This function
returned the memory address of an object in memory, which could possibly be
used by attackers to bypass address randomization protections.
(CVE-2011-1202)
|
| Alerts: |
|
Comments (none posted)
kernel: multiple vulnerabilities
| Package(s): | kernel |
CVE #(s): | CVE-2011-0191
CVE-2011-1013
CVE-2011-1016
CVE-2011-1093
CVE-2011-1180
CVE-2011-1573
|
| Created: | April 28, 2011 |
Updated: | August 19, 2011 |
| Description: |
From the SUSE advisory:
CVE-2011-0191: A information leak in the XFS geometry calls could be
used by local attackers to gain access to kernel information.
CVE-2011-1013: A signedness issue in drm_modeset_ctl() could be used
by local attackers with access to the drm devices to potentially
crash the kernel or escalate privileges.
CVE-2011-1016: The Radeon GPU drivers in the Linux kernel did not
properly validate data related to the AA resolve registers, which
allowed local users to write to arbitrary memory locations associated
with (1) Video RAM (aka VRAM) or (2) the Graphics Translation Table
(GTT) via crafted values.
CVE-2011-1093: A bug in the order of dccp_rcv_state_process() was fixed
that still permitted reception even after closing the socket. A Reset
after close thus causes a NULL pointer dereference by not preventing
operations on an already torn-down socket.
CVE-2011-1180: In the IrDA module, length fields provided by a peer
for names and attributes may be longer than the destination array
sizes and were not checked, this allowed local attackers (close to
the irda port) to potentially corrupt memory.
CVE-2011-1573: Bounds checking was missing in AARESOLVE_OFFSET, which
allowed local attackers to overwrite kernel memory and so escalate
privileges or crash the kernel.
|
| Alerts: |
|
Comments (none posted)
kernel: multiple vulnerabilities
| Package(s): | kernel |
CVE #(s): | CVE-2011-1160
CVE-2011-1577
CVE-2011-1581
|
| Created: | April 29, 2011 |
Updated: | November 28, 2011 |
| Description: |
From the openSUSE advisory:
CVE-2011-1160: Kernel information via the TPM devices could by used by local attackers to read kernel memory.
CVE-2011-1577: The Linux kernel automatically evaluated partition tables of storage devices. The code for evaluating EFI GUID partitions (in fs/partitions/efi.c) contained a bug that causes a kernel oops on certain
corrupted GUID partition tables, which might be used by local attackers to crash the kernel or potentially execute code.
CVE-2011-1581: Doing bridging with devices with more than 16 receive queues could crash the kernel.
|
| Alerts: |
|
Comments (none posted)
mediawiki: multiple vulnerabilities
| Package(s): | mediawiki |
CVE #(s): | CVE-2011-0047
CVE-2011-0003
CVE-2010-2787
CVE-2010-2788
CVE-2011-1578
CVE-2011-1579
CVE-2011-1580
|
| Created: | May 2, 2011 |
Updated: | December 19, 2011 |
| Description: |
From the CVE entries:
Cross-site scripting (XSS) vulnerability in MediaWiki before 1.16.2 allows remote attackers to inject arbitrary web script or HTML via crafted Cascading Style Sheets (CSS) comments, aka "CSS injection vulnerability." (CVE-2011-0047)
MediaWiki before 1.16.1, when user or site JavaScript or CSS is enabled, allows remote attackers to conduct clickjacking attacks via unspecified vectors. (CVE-2011-0003)
api.php in MediaWiki before 1.15.5 does not prevent use of public caching headers for private data, which allows remote attackers to bypass intended access restrictions and obtain sensitive information by retrieving documents from an HTTP proxy cache that has been used by a victim. (CVE-2010-2787)
Cross-site scripting (XSS) vulnerability in profileinfo.php in MediaWiki before 1.15.5, when wgEnableProfileInfo is enabled, allows remote attackers to inject arbitrary web script or HTML via the filter parameter. (CVE-2010-2788)
Cross-site scripting (XSS) vulnerability in MediaWiki before 1.16.3, when Internet Explorer 6 or earlier is used, allows remote attackers to inject arbitrary web script or HTML via an uploaded file accessed with a dangerous extension such as .html at the end of the query string, in conjunction with a modified URI path that has a %2E sequence in place of the . (dot) character. (CVE-2011-1578)
The checkCss function in includes/Sanitizer.php in the wikitext parser in MediaWiki before 1.16.3 does not properly validate Cascading Style Sheets (CSS) token sequences, which allows remote attackers to conduct cross-site scripting (XSS) attacks or obtain sensitive information by using the \2f\2a and \2a\2f hex strings to surround CSS comments. (CVE-2011-1579)
The transwiki import functionality in MediaWiki before 1.16.3 does not properly check privileges, which allows remote authenticated users to perform imports from any wgImportSources wiki via a crafted POST request. (CVE-2011-1580) |
| Alerts: |
|
Comments (none posted)
php5: multiple vulnerabilities
| Package(s): | php5 |
CVE #(s): | CVE-2011-1072
CVE-2011-1144
CVE-2006-7243
CVE-2011-0420
|
| Created: | May 2, 2011 |
Updated: | April 13, 2012 |
| Description: |
From the CVE entries:
The installer in PEAR before 1.9.2 allows local users to overwrite arbitrary files via a symlink attack on the package.xml file, related to the (1) download_dir, (2) cache_dir, (3) tmp_dir, and (4) pear-build-download directories, a different vulnerability than CVE-2007-2519. (CVE-2011-1072)
The installer in PEAR 1.9.2 and earlier allows local users to overwrite arbitrary files via a symlink attack on the package.xml file, related to the (1) download_dir, (2) cache_dir, (3) tmp_dir, and (4) pear-build-download directories. NOTE: this vulnerability exists because of an incomplete fix for CVE-2011-1072. (CVE-2011-1144)
PHP before 5.3.4 accepts the \0 character in a pathname, which might allow context-dependent attackers to bypass intended access restrictions by placing a safe file extension after this character, as demonstrated by .php\0.jpg at the end of the argument to the file_exists function. (CVE-2006-7243)
The grapheme_extract function in the Internationalization extension (Intl) for ICU for PHP 5.3.5 allows context-dependent attackers to cause a denial of service (crash) via an invalid size argument, which triggers a NULL pointer dereference. (CVE-2011-0420) |
| Alerts: |
|
Comments (none posted)
python: information leak
| Package(s): | python |
CVE #(s): | CVE-2011-1521
|
| Created: | May 3, 2011 |
Updated: | October 18, 2012 |
| Description: |
From the Pardus advisory:
A security flaw was found in the way handlers for ftp:// and file:// URL
schemes in the Python urllib and urllib2 extensible libraries processed
the urllib open URL request. A remote attacker could use this flaw to
access sensitive information or cause a denial of service (excessive CPU
and memory use) of a Python web application, processing URLs, via a
specially- crafted urllib open URL request.
|
| Alerts: |
|
Comments (none posted)
qemu-kvm: privilege escalation
| Package(s): | qemu-kvm |
CVE #(s): | CVE-2011-1750
|
| Created: | May 2, 2011 |
Updated: | July 7, 2011 |
| Description: |
From the Debian advisory:
The virtio-blk driver performed insufficient validation of
read/write I/O from the guest instance, which could lead to
denial of service or privilege escalation.
|
| Alerts: |
|
Comments (none posted)
seamonkey: arbitrary code execution
| Package(s): | seamonkey |
CVE #(s): | CVE-2011-0072
|
| Created: | April 29, 2011 |
Updated: | June 7, 2011 |
| Description: |
From the Red Hat advisory:
A use-after-free flaw was found in the way SeaMonkey appended frame and
iframe elements to a DOM tree when the NoScript add-on was enabled.
Malicious HTML content could cause SeaMonkey to execute arbitrary code with
the privileges of the user running SeaMonkey. |
| Alerts: |
|
Comments (none posted)
spip: denial of service
| Package(s): | spip |
CVE #(s): | |
| Created: | May 2, 2011 |
Updated: | May 4, 2011 |
| Description: |
From the Debian advisory:
A vulnerability has been found in SPIP, a website engine for publishing,
which allows a malicious registered author to disconnect the website
from its database, resulting in denial of service.
|
| Alerts: |
|
Comments (none posted)
thunderbird: multiple vulnerabilities
| Package(s): | thunderbird |
CVE #(s): | CVE-2011-0070
CVE-2011-0071
CVE-2011-0073
CVE-2011-0074
CVE-2011-0075
CVE-2011-0077
CVE-2011-0078
CVE-2011-0080
CVE-2011-0081
|
| Created: | April 29, 2011 |
Updated: | July 19, 2011 |
| Description: |
From the Red Hat advisory:
Several flaws were found in the processing of malformed HTML content. An
HTML mail message containing malicious content could possibly lead to
arbitrary code execution with the privileges of the user running
Thunderbird. (CVE-2011-0080, CVE-2011-0081)
An arbitrary memory write flaw was found in the way Thunderbird handled
out-of-memory conditions. If all memory was consumed when a user viewed a
malicious HTML mail message, it could possibly lead to arbitrary code
execution with the privileges of the user running Thunderbird.
(CVE-2011-0078)
An integer overflow flaw was found in the way Thunderbird handled the HTML
frameset tag. An HTML mail message with a frameset tag containing large
values for the "rows" and "cols" attributes could trigger this flaw,
possibly leading to arbitrary code execution with the privileges of the
user running Thunderbird. (CVE-2011-0077)
A flaw was found in the way Thunderbird handled the HTML iframe tag. An
HTML mail message with an iframe tag containing a specially-crafted source
address could trigger this flaw, possibly leading to arbitrary code
execution with the privileges of the user running Thunderbird.
(CVE-2011-0075)
A flaw was found in the way Thunderbird displayed multiple marquee
elements. A malformed HTML mail message could cause Thunderbird to execute
arbitrary code with the privileges of the user running Thunderbird.
(CVE-2011-0074)
A flaw was found in the way Thunderbird handled the nsTreeSelection
element. Malformed content could cause Thunderbird to execute arbitrary
code with the privileges of the user running Thunderbird. (CVE-2011-0073)
A directory traversal flaw was found in the Thunderbird resource://
protocol handler. Malicious content could cause Thunderbird to access
arbitrary files accessible to the user running Thunderbird. (CVE-2011-0071)
A double free flaw was found in the way Thunderbird handled
"application/http-index-format" documents. A malformed HTTP response could
cause Thunderbird to execute arbitrary code with the privileges of the user
running Thunderbird. (CVE-2011-0070)
|
| Alerts: |
|
Comments (none posted)
tiff: integer overflow
| Package(s): | tiff |
CVE #(s): | CVE-2010-4665
|
| Created: | April 29, 2011 |
Updated: | June 21, 2011 |
| Description: |
From the openSUSE advisory:
Directories with a large number of files could cause an integer overflow in the tiffdump tool. |
| Alerts: |
|
Comments (none posted)
udisks: loads arbitrary LKMs
| Package(s): | udisks |
CVE #(s): | CVE-2010-4661
|
| Created: | April 29, 2011 |
Updated: | August 30, 2012 |
| Description: |
From the openSUSE advisory:
This update of udisks improves input validation. Before it was possible to load arbitrary LKMs. |
| Alerts: |
|
Comments (none posted)
usb-creator: restriction bypass
| Package(s): | usb-creator |
CVE #(s): | CVE-2011-1828
|
| Created: | May 2, 2011 |
Updated: | May 4, 2011 |
| Description: |
From the Ubuntu advisory:
Evan Broder discovered that usb-creator did not properly enforce
restrictions when performing privileged disk operations. A local attacker
could use this flaw to perform certain disk operations, such as unmount
arbitrary mountpoints.
|
| Alerts: |
|
Comments (none posted)
vino: denial of service
| Package(s): | vino |
CVE #(s): | CVE-2011-0904
CVE-2011-0905
|
| Created: | May 3, 2011 |
Updated: | January 22, 2013 |
| Description: |
From the Ubuntu advisory:
Kevin Chen discovered that Vino incorrectly handled certain client
framebuffer requests. A remote attacker could use this flaw to cause Vino
to crash, leading to a denial of service.
|
| Alerts: |
|
Comments (none posted)
vlc: heap corruption
| Package(s): | vlc vlc-firefox |
CVE #(s): | CVE-2011-1684
|
| Created: | May 3, 2011 |
Updated: | May 4, 2011 |
| Description: |
From the Pardus advisory:
When parsing some MP4 (MPEG-4 Part 14) files, insufficient buffer size
might lead to corruption of the heap. |
| Alerts: |
|
Comments (none posted)
Page editor: Jake Edge
Kernel development
Brief items
The current development kernel is 2.6.39-rc6,
released on May 3. Linus said:
We're still chasing some stuff down, but I think we're ok for
-rc6. This isn't going to be the final -rc, but it doesn't seem to
be in bad shape, and people who have posted regression reports
please check them again, and people who haven't, please do give it
a test.
See the
full changelog for all the details.
Stable updates: 2.6.38.5
(May 2), 2.6.35.13 (April 28),
and 2.6.27.59 (April 30). Each
contains a long list of important changes.
Comments (none posted)
Let ARM rot in mainline. I really don't care anymore.
--
Russell King
Nah, I'll leave it in so that they can have a blast from the past
and can boast on the slashdot of the future or on ((LWN++)..)++
that they've fixed the oldest bug in Linux when booting it on
quantum x86 computers and tried running their RAS daemon.
--
Borislav Petkov
Yes, open source programming is a team sport, but finding the right
people is really the killer feature (the same is obviously true in
the kernel too - I really do think we have a great set of
maintainers. I may complain about them and I'm somewhat infamous
for my flames when things don't work, but at the same time I'm
convinced there's some of the best people out there working on
maintaining the kernel).
--
Linus Torvalds
Comments (none posted)
In response to the EFF's
call
for owners of wireless networks to allow open access, Luis Rodriguez has
put up
a wiki
page to gather information on how providing open access can be made
safer. Contributions of ideas and comments are welcome.
Comments (none posted)
LinuxFR has an
interview with
Linus Torvalds on a wide range of subjects. "
LinuxFR : What is your opinion about Android ? Are you mostly happy they made cellphones very usable or sad because it's really a kernel fork ?
Linus Torvalds : I think forks are good things, they don't make me sad. A lot of Linux development has come out of forks, and it's the only way to keep developers honest - the threat that somebody else can do a better job and satisfy the market better by being different. The whole point of open source to me is really the very real ability to fork (but also the ability for all sides to then merge the forked content back, if it turns out that the fork was doing the right things!)"
Comments (65 posted)
By Jonathan Corbet
May 4, 2011
As was discussed at the
2011 filesystem,
storage, and memory management summit, there is an increasing level of
interest in restricting the amount of kernel memory which can be used by
groups of processes. One area of special interest is the directory entry
(dentry) cache; a
malicious program can, by creating a deep enough directory hierarchy, run
the kernel out of memory with an explosion of the size of the dentry
cache. So limiting dentry use has some real appeal, especially for those
working to ensure that containers running on a Linux system cannot
interfere with each other.
Pavel Emelyanov's per-container dcache
management patches are a first attempt at limiting dentry use. This
patch works by organizing dentries into "mobs," being groups of dentries
all of which represent names in a specific subtree of the filesystem. If
the root of a mob were the root of a container's filesystem namespace, all
dentries created by that container would be contained within that mob. At
that point, a simple sort of resource control can be applied: adding a
dentry to a mob which has hit its maximum size would require the removal of
another dentry to compensate. If no dentries an be removed, attempts to
add others will fail.
The patch set adds three new ioctl() calls: FIMOBROOT to
create a new mob at a given point in the filesystem, FIMOBSIZE to
set the maximum size of a mob, and FIMOBSTAT to query the current
usage of a mob. Pavel is somewhat apologetic about this interface; he
seems to think it will have to change before the work could be considered
upstream. But the first step is get some discussion of the concept; so
far, there have been no responses to Pavel's patches.
Comments (none posted)
Kernel development news
By Jonathan Corbet
May 3, 2011
The perf events subsystem often looks like it's on the path to take over
the kernel; there is a great deal of development activity there, and it has
become a sort of generalized event reporting mechanism. But the original
purpose of perf events was to provide access to the performance monitoring
counters made available by the hardware, and it is still used to that end.
The merging of perf was a bit of a hard pill for users of alternative
performance monitoring tools to swallow, but they have mostly done so. The
recent discussion on "offcore" events shows that there are still some
things to argue about in this area, even if everybody seems likely to get
what they want in the end.
The performance monitoring unit (PMU) is normally associated with the CPU;
each processor has its own PMU for monitoring its own specific events.
Some newer processors (such as Intel's Nehalem series) also provide a PMU
which is not tied to any CPU; in the Nehalem case it's part of the "uncore"
which handles memory access at the package level. The off-core PMU has a
viewpoint which allows it to provide a better picture of the overall memory
behavior of the system, so there is interest in gaining access to events
from that PMU. Current kernels, though, do not provide access to these
offcore events.
For a while, the 2.6.39-rc kernel did provide access to these
events, following the merging of a
patch by Andi Kleen in March. One piece that was missing, though, was
a patch to the user-space perf tool to provide access to this
functionality. There was an attempt to merge that piece toward the end of
April, but it did
not yield the desired results; rather than merge the additional
change, perf maintainer Ingo Molnar removed
the ability to access offcore events entirely.
Needless to say, that action has led to some unhappiness in the perf user
community; there are developers who had already been making use of those
events. Normally, breaking things in this way would be considered a
regression, and the patch would be backed out again. But, since this
functionality never appeared in a released kernel, it cannot really be
called a regression. That, of course, is part of the point of removing the
feature now.
Ingo's complaint is straightforward: the interface to these events was too
low-level and too difficult to use. The rejected perf patch had an example
command which looked like:
perf stat -e r1b7:20ff -a sleep 1
Non-expert readers may, indeed, be forgiven for not immediately
understanding that this command would monitor access to remote DRAM -
memory which is hosted on a different socket. Ingo asserted that the
feature should be more easily used, perhaps with a command like (from the
patch removing the feature):
perf record -e dram-remote ./myapp
He also said:
But this kind of usability is absolutely unacceptable - users
should not be expected to type in magic, CPU and model specific
incantations to get access to useful hardware functionality.
The proper solution is to expose useful offcore functionality via
generalized events - that way users do not have to care which
specific CPU model they are using, they can use the conceptual
event and not some model specific quirky hexa number.
The key is the call for "generalized events" which are mapped, within the
kernel, onto whatever counters the currently-running hardware uses to
obtain that information. Users need not worry about the exact type of
processor they are running on, and they need not dig out the data sheet to
figure out what numbers will yield the results they want.
Criticism of this move has taken a few forms. Generalized events, it is
said, are a fine thing to have, but they can never reflect all of the
weird, hardware-specific counters that each processor may provide. These
events should also be managed in user space where there is more flexibility
and no need to bloat the kernel. There were some complaints about how some
of the existing generalized events have not always been implemented
correctly on all architectures. And, they say, there will always be people
who want to know what's in a specific hardware counter without having the
kernel trying to generalize it away from them. As Vince Weaver put it:
Blocking access to raw events is the wrong idea. If anything, the
whole "generic events" thing in the kernel should be ditched.
Wrong events are used at times (see AMD branch events a few
releases back, now Nehalem cache events). This all belongs in
userspace, as was pointed out at the start. The kernel has no
business telling users which perf events are interesting, or
limiting them!
Ingo's response is that the knowledge and
techniques around performance monitoring should be concentrated in one
place:
Well, the thing is, i think users are helped most if we add useful,
highlevel PMU features added and not just an opaque raw event
pass-through engine. The problem with lowlevel raw ABIs is that the
tool space fragments into a zillion small hacks and there's no good
concentration of know-how. I'd like the art of performance
measurement to be generalized out, as well as it can be.
Vince, meanwhile, went on to claim that perf was a
reinvention of the wheel which has ignored a lot of the experience built
into its predecessors. There are, it seems, still some scars from that
series of events. Thomas Gleixner disagreed with
the claim that perf is an exercise in wheel reinvention, but he did say
that he thought the raw events should be made available:
The problem at hand which ignited this flame war is definitely
borderline and I don't agree with Ingo that it should not made be
available right now in the raw form. That's an hardware enablement
feature which can be useful even if tools/perf has not support for
it and we have no generalized event for it. That's two different
stories. perf has always allowed to use raw events and I don't see
a reason why we should not do that in this case if it enables a
subset of the perf userbase to make use of it.
It turns out that Ingo is fine with raw events
too. His stated concern is that access to raw events should not be the
primary means by which most users gain access to those performance
counters. So he is blocking the availability of those events for now for
two reasons. One of those is that he wants the generalized mode of access
to be available first so that users will see it as the normal way to access
offcore events. If there is never any need to figure out hexadecimal
incantations, many user-space developers will never bother; as a result,
their commands and code should eventually work on other processors as well.
The other reason for blocking raw events now is that, as the interface to
these events is thought through, the ABI by which they are exposed to user
space may need to change. Releasing the initial ABI in a stable kernel
seems almost certain to cement it in place, given that people were already
using it. By deferring these events for one cycle (somebody will certainly
come up with a way to export them in 2.6.40), he hopes to avoid being stuck
with a second-rate interface which has to be supported forever.
There can be no doubt that managing this feature in this way makes life
harder for some developers. The kernel process can be obnoxious to deal
with at times. But the hope is that doing things this way will lead to a
kernel that everybody is happier with five years from now. If things work
out that way, most of us can deal with a bit of frustration and a one-cycle
delay now.
Comments (5 posted)
By Jake Edge
May 4, 2011
Sandboxing processes such that they cannot make "dangerous" system calls is
an attractive feature that has already been implemented in a limited way
for Linux with seccomp. Two years ago, we looked at a proposal to expand seccomp to
allow more fine-grained control over which system calls would be allowed.
That proposal has been mostly dormant since then, but was recently
resurrected after incorporating some of the suggestions made at that time.
The reaction to the current proposal so far seems positive, and it might
just be gaining some traction that the previous patchset lacked.
Seccomp (from "secure computing") is enabled via a prctl() call
and, once enabled, restricts the process from making any further system calls beyond
read(), write(), exit(), and
sigreturn()—any other system call will abort the
process. That creates a pretty secure sandbox, but it is also
extremely limited as there are other things that developers might want to
do from within such a sandbox. In fact, the Chromium browser has gone to great lengths to implement its own
sandbox that uses seccomp, but expands the range of legal system calls
through some contortions.
That led Adam Langley of the Chromium team to propose adding a bitmask of allowable system
calls for a new seccomp mode. That would have allowed processes to make a
binary choice (allowed or disallowed) for each system call. At the time,
Ingo Molnar suggested using the Ftrace filter
code to make the interface even more flexible by allowing filters to be
applied to the system call arguments. Essentially, that would make for
three choices for each system call: enabled, disabled, or filtered.
Fast-forward to today, and that is what a patchset from Will Drewry implements. It
should come as no surprise that Molnar was pleased to see his idea result in working
code: "Ok, color me thoroughly
impressed - AFAICS you implemented my suggestions [...] and you made it
work in practice!". Eric Paris was likewise impressed, noting that an expanded seccomp
could be used for QEMU. Molnar and Paris did not agree about replacing the
LSM approach using filters, but that was something of an aside. Serge
E. Hallyn also pointed out that the new feature
would be useful for containers: "to try and provide some bit of
mitigation for the fact that they are sharing a kernel with the
host".
The proposed interface, which is likely to change based on comments in the
thread, looks like:
const char *filters[] =
"sys_read: (fd == 1) || (fd == 2)\n"
"sys_write: (fd == 0)\n"
"sys_exit: 1\n"
"sys_exit_group: 1\n"
"on_next_syscall: 1";
prctl(PR_SET_SECCOMP, 2, filters);
That example is taken from Drewry's
documentation file that accompanies the patches.
It would allow reading from two file descriptors (1 and 2) and writing to
one (0), while
allowing any calls to the two other system calls listed. The
on_next_syscall means that the rules would not be enforced until
after one more system call is made. That would allow a parent to
fork(), set up the seccomp sandbox in the child process, then exec
a new
program which would be governed by the new rules.
That on_next_syscall piece drew a few comments. As it turns out,
there are really only two cases that need to be handled, either the rules
should go into effect immediately (for a process that wants to restrict
itself before handling untrusted input for example), or they should go into
effect after an exec (for a parent that is spawning an untrusted child).
Making the "after exec" case the default, while still allowing a
process
to request immediate application, seems to be the way things are headed.
There were also questions about using kernel-internal symbol names like
sys_read. Exporting those as a kernel ABI is not likely to pass
muster, as it might restrict the option of changing those function names
down the road—or require a messy compatibility layer if they did
change. Drewry wanted
to avoid using the system call numbers as Langley's original patch did, but
as Frederic Weisbecker pointed out, those
numbers are already part of the kernel ABI. Drewry is planning to make
that switch and users of the interface will need to use the
unistd.h header file or a library to map system call names to
numbers.
The patches also modify the /proc/PID/status file to output any
existing filters that are applied to the process. Given that most applications
that read that file don't need the extra information, though, Motohiro
Kosaki suggested that seccomp get its own
file. Drewry's plan is to provide that information in the
/proc/PID/seccomp_filter file instead, and remove it from
status.
Since it uses the Ftrace infrastructure and hooks, the new seccomp mode
only works for those system calls that have Ftrace events associated with
them. Using one of those non-instrumented system calls in the filters will
result in an
EINVAL from the prctl() call.
Enabling CONFIG_SECCOMP_FILTER (which depends on
CONFIG_FTRACE_SYSCALLS) will allow the use of the new mode.
Overall, Drewry has been very receptive to suggestions for changes, and
the feedback to the concept has been pretty uniformly positive. Molnar
suggested breaking
out the Ftrace filter engine further—beyond the minimal changes that
Drewry's patches make—so that it would be available for more
widespread use in the kernel. Molnar does wonder whether Linus Torvalds or
Andrew Morton might object to more use of the filter mechanism, however: "are you guys opposed to such flexible, dynamic
filters conceptually? I think we should really think hard about the actual ABI
as this could easily spread to more applications than
Chrome/Chromium." So far, neither has spoken up one way or the other.
Currently it would seem that Drewry is off working on the next revision of
the patchset, and it certainly doesn't seem like anything that would be
merged in the upcoming 2.6.40 cycle. As Molnar notes, the ABI needs to be carefully
thought-out, there are still some RCU issues that are being discussed, and
it probably needs some soaking time in the -next tree, but barring some major
complaint
cropping up, it's a feature that will likely make its way into the
mainline relatively soon. While that won't allow Chromium to immediately ditch
its complicated sandboxing arrangement, it may well be able to
do so a few years down the road. Other applications will benefit
from an expanded seccomp as
well.
Comments (10 posted)
By Jake Edge
May 4, 2011
Back in October, Bryce Lelbach announced that he (and others) had built
a working Linux kernel using (mostly) Clang, the LLVM-based C compiler. At
the Linux Foundation Collaboration Summit (LFCS) back in April, Lelbach
gave a talk about the progress that had been made, and the work still to be
done, for the LLVM Linux (LLL)
project. That talk, along with the rest of the LLVM track, was quite
interesting, and once again showed that having two (or more) "competing"
projects is generally beneficial to both.
Why build Linux with Clang?
Lelbach started off describing the reasons behind the decision to try to
build Linux with Clang, most of which centered around the diagnostics that
the compiler produces. The Clang static analyzer has the ability to show
"what the compiler sees when it's looking at your code", he
said. He thought that a huge codebase like Linux could benefit from that
kind of analysis.
In fact, the Clang diagnostics were quite useful when he was building the
Broadcom wireless driver for his MacBook, he said. Clang doesn't forget
things, so it can show macros before their expansion, typedefs, and so on.
It also shows the line in the source code with a caret pointing to the
offending code, along with "fixit hints". Those hints can be automatically
applied to the source code to fix the problem in question.
The project got a 2.6.36-based kernel running back in October, and now has
working kernels based on .37 and .38. Neither Xen nor KVM worked at the
time of the talk and Xen won't even compile, though KVM
is said to work now. More than 90% of the drivers in the kernel will at
least compile, and many will work. Some out-of-tree binary drivers (Broadcom, NVIDIA) will
work as well. SMP versions of the kernel for both 32 and 64-bit x86
platforms are now working, though some of the code needs to be patched in
order to build correctly.
Things that don't work
The integrated assembler (IA) for Clang does not have support for
generating "real
mode" code using .code16gcc directives, so the Linux boot code cannot
be built using IA. There is a "nasty pile" of real mode code
required to boot on x86, Lelbach said. IA is the default for recent
versions of Clang, but using the
GNU Assembler (gas) was required for the boot code. Adding support for an
LLVM x86-16 backend is the right approach, he said, and LLVM project
members in the audience agreed that it was something that could be added to
IA.
The "vast majority of GCC extensions are supported" by Clang,
even those which are not documented, which makes compiling the kernel much
easier. Things like inline assembly, the __attribute__ and
__builtin__ syntax, and so on, all just work. He expected that
there might be problems with inline assembly, but that has not proven to be
the case. Clang defaults to the C99 standard, though, so the
gnu89 standard needs to be specified to build the kernel.
There are some GCC extensions that aren't implemented, however, including
explicit register variables. That lack blocks Xen and some user-space
libraries (like glibc) from compiling. There
are also some "intentionally unsupported extensions",
including local/nested functions, which is only used in a Thinkpad driver.
A bigger problem is that Clang lacks support for variable-length arrays in
structures (VLAIS). A declaration like:
void f (int i) {
struct foo_t {
char a[i];
} foo;
}
cannot be compiled in Clang, though declarations like:
void f (int i) {
char foo[i];
}
are perfectly acceptable. Code like the former is used in the iptables
code, the
kernel hashing (HMAC) routines, and some drivers. Those parts have to be
patched
in order to be built, he said.
Once again, someone from the audience piped up to say that support for
VLAIS could be added as long as the patches were not "
wildly
invasive". The LLL folks "
prefer adding things to Clang
rather than patching the kernel", Lelbach said.
That led to a question about whether the project was pushing any of its
patches upstream to the kernel. Lelbach said that the PaX team (who is
another LLL developer) had
submitted a few, but that those were rejected; "after three, we
stopped" submitting them. Part of the problem is that the patches
are not ready for inclusion because there is a lack developer time to get
them into shape. As an audience member noted, though, the kernel folks are
quick
to take any patches that fix bugs found by Clang.
Code generation and optimization problems
There are several code generation and optimization options for GCC that
aren't supported by Clang. One of those is -mregparm that governs
the number of registers used to pass integer arguments. That means
calls to functions like memcpy() are generated that ignore the custom
calling conventions.
Also,
-fcall-saved-reg is not supported by Clang and that affects the
uses of the ALTERNATIVE() macro in the kernel, which chooses
between assembly instructions depending on the processor model. For some of the
__arch_hweight*() implementations ALTERNATIVE() buries
the actual function
call inside
assembly code, so Clang doesn't know about it. That means that the
generated code is not saving all of the registers that it needs to, so
uses
of ALTERNATIVE() are commented out and a normal call to the
function is used instead.
Another problem is with -pg, which enables instrumentation code
for function calls in GCC, and is used when building Ftrace. For inline
functions, the calls to mcount() get added multiple times, both
when the code is generated and when it is expanded inline. The
no_instrument_function attribute is not properly propagated to inline
functions, he said.
The final
problem that Lelbach mentioned is the -fno-optimize-sibling-calls
flag that is not supported by Clang. The flag disables tail call
elimination, and the kernel introspection code (like Ftrace) assumes
specific stack
depths in various places. Because Clang doesn't support the flag,
code which walks the call stack can end up dereferencing user-space
pointers, which
leads to runtime crashes. This was worked around by defining
HAVE_ARCH_CALLER_ADDR for x86 and defining
CALLER_ADDR[1-6] as dummy values, effectively disabling the stack
backtracing.
It is not just Lelbach who is working LLL, and he noted that the PaX team,
Alp Toker, and Török Edwin have all contributed, along with various
Clang/LLVM and Linux
kernel hackers. There are plans to create a mailing list for the project
and the beginnings of a wiki are taking
shape. Overall, it's an interesting project that will likely end up
helping to find bugs in the kernel while discovering features that could or
should be supported by LLVM/Clang.
[ Thanks to Bryce Lelbach, PaX team, and Török Edwin for
filling in holes in my notes. ]
Comments (5 posted)
Patches and updates
Kernel trees
Core kernel code
Development tools
Device drivers
Filesystems and block I/O
Memory management
Networking
Architecture-specific
Security-related
Benchmarks and bugs
Miscellaneous
Page editor: Jonathan Corbet
Distributions
May 4, 2011
This article was contributed by Nathan Willis
The Debian project is again seriously discussing proposals to add a user-centric, frequently-updated distribution product to its public offerings. The concept centers around providing a rolling release with a continuous (and therefore current) stream of package updates, as an alternative to the well-honed (but frequently behind-the-times) Debian stable. Most of those involved seem to agree that offering a "Debian rolling" product would benefit the project, but the debate persists over how much it would cost in terms of developer and release-team time — and how that investment would affect the existing release process, which is tailored toward the production of stable.
The idea spun off from discussions around the constantly-usable testing (CUT) Debian concept in a 2007 proposal from Joey Hess, which observed that many users are unsatisfied with the age of packages in stable, and choose to run testing instead. Debian should adjust its release process to better serve those users, Hess said, by making sure that testing was always installable, and by providing more timely bug fixes.
The CUT concept was resurrected at Debconf10 in August of 2010, and since then the debian-devel list has hosted numerous discussions and refinements — including offering rolling releases. The proposals ranged from simply re-naming Debian "testing" as "rolling," to creating regular snapshot releases of testing, to changing the release process in large ways — including the freeze process and even the relationship between unstable, testing, and stable. The latest discussion spans hundreds of individual messages, which prompted Lucas Nussbaum to write a blog post on May 2nd in an attempt to summarize the discussion and the current state of the proposed course of action.
Where rolling stands
Nussbaum starts out by clarifying what supporters of Debian rolling see as the benefits of the sub-project. First, providing a reliable rolling-release will attract new users (and perhaps new contributors) to Debian, in particular those who are uncomfortable using Debian testing because they perceive it as a development branch, but for whom the packages in Debian stable are too old. By doing nothing, he said, Debian currently cedes these users en masse to Debian derivatives like Linux Mint, Aptosid, and Ubuntu.
Second, a successful Debian rolling product would help upstream projects
to gain new users more quickly, providing valuable feedback more rapidly
than is possible in the traditional release cycle. There are, of course,
other rolling-release distributions (including the Debian derivatives
mentioned above), but Nussbaum seems to think that Debian would be offering a unique rolling product by virtue of its size and proven history as a major distribution. Finally, a Debian rolling product would bring more attention to the Debian project, which is often overshadowed by the Debian derivatives in the news, which in turn makes recruiting new contributors difficult.
As for the plan itself, Nussbaum said that it is "mostly about (external) communication" and that it will require "marginal additional work." It would start with a general resolution (GR) from the project members, announcing the testing->rolling name change, Nussbaum said. Raphael Hertzog has already drafted a proposed GR to that effect. Accompanying the name change would be a committed effort on the part of Debian developers to support the packages in rolling with regular updates that prevent breakage of major components, and integrate security-related or other important patches from unstable.
The big unanswered questions at the moment are how to handle the freezes that occur before releasing a new stable. Testing is currently frozen for around six months while stable is prepared, a length of time that could be considered a painfully long freeze for a "rolling release." The options Nussbaum enumerates are to create a "frozen" branch from rolling (and develop stable from frozen on approximately the same six-month time-frame), or to freeze rolling for a shorter amount of time (such as three months), then create the frozen branch and unfreeze rolling.
Both options would create two releases (rolling and frozen/stable) that for at least some portion of each release cycle would each demand divided attention from developers: some working on the frozen branch to prepare it for release as stable, and others continuing to test and update the packages in rolling. This division-of-effort is one of the most common objections to the proposed plan, but Nussbaum argues (as does Hertzog) that this is already the case: as a group, Debian developers must divide their attention between developing for stable-plus-one (in unstable and testing) and back-porting critical updates to stable. The main difference, he insists, is that the proposed plan would do a better job of providing an accessible "Debian product" in between stable releases.
On the other hand, Nussbaum concedes that during freezes, there would be multiple "entry points" for actual packages. Debian unstable would continue to be the entry point for packages promoted to rolling, while frozen-proposed-updates would be the entry point for packages targeting frozen/stable.
Objections and other options
The testing->rolling plan outlined by Nussbaum is not the only proposal under consideration. Other suggestions focus on making smaller changes to the existing development process, so that testing is a better platform for daily usage. For example, reducing the number of architectures required to migrate a package from unstable to testing and encouraging developers to rebuild packages for testing-proposed-updates instead of for unstable would both get packages into testing more quickly, to the satisfaction of users interested in up-to-date packages.
Other proposals include implementing Apt repositories for different packages a la Ubuntu's personal package archives (PPAs) for developers, and attempting to streamline the packaging process as a whole (thus hopefully reducing the length of freezes themselves) so as to speed up development. The PPA idea does not seem to have widespread support, partly because it has high infrastructure requirements, partly because it might require all involved to settle on a single version control system. But the debate over it rages on on debian-devel — while Nussbaum argues that the "streamlining" suggestion consists of good ideas that should be considered regardless of what approach the testing/rolling debate settles on in the end.
The proposed approach has its share of critics, including Josselin Mouette, who argued that the proposal does not include enough specific changes to the development process. Mouette believes asking developers to maintain testing/rolling as a usable distribution would require large-scale regression testing "on several upgrade paths" to cope with packages that break when migrating from unstable to testing — for example, packages that break unexpectedly because their dependencies are not listed correctly.
Presumably, then, requiring tighter monitoring of package dependencies would be one of the specific changes Mouette is asking for, but his preferred solution is different entirely: officially adopting one of the existing derivative distributions as a Debian project. By blessing an existing derivative such as Aptosid, he says, the project gains the benefit of their work without requiring any additional outlay of effort from the ranks of current Debian developers. He has also suggested implementing rolling as an Apt suite that interested parties can add on top of testing, which would leave the existing process unchanged.
But the fundamental concern of most who are wary of the proposal is that
by putting emphasis on rolling as a second "Debian product," Debian stable
will suffer. Part of the concern is that there is just too little
developer-power to spread around, but it is also possible that some fans of
rolling will stop participating the existing release process for stable,
leaving packages untested. Samuel Thibault summed it up nicely, saying he fears "that a lot of developers will see this switching point [unfreezing rolling] as 'ok, now it's debian-release's matter, I can again focus on rolling only'."
Others, like Sean Finney, have argued that the rolling release approach itself is a distraction from the more fundamental problem, which is that during freezes, essentially all work grinds to a complete halt. Rolling releases might help the grinding-to-a-halt problem, Finney said, but they might have no effect at all.
Despite the varying viewpoints, the project as a whole seems to be converging toward a single plan. For his part, Debian Project Leader Stefano Zacchiroli seems generally in support of the rolling idea, but warns that "there might be plenty of lower hanging fruits than a user-targeted testing out there, but for me there is no reason not to discuss other topics as well."
What's next?
All of the interested parties appear to agree on one thing: no matter what proposal reaches consensus through the various mailing list discussion, it can only be tested by being put into practice, and seeing whether or not there is enough developer momentum to see it through.
Case in point: one of the other original Debian CUT approaches focused
on building regular "snapshot" releases from testing, thus providing a
known-to-be-good entry point for new users. The snapshot project is already underway, and
making installable nightly builds as well as regular snapshots available
for download. The snapshots do not directly address the same concern as
the testing->rolling proposal, but they do meet a need that is not served by the existing development process.
It is a rare case when the vast majority of participants in a large project like Debian agree on the need to undertake a new tactic. There is not universal consensus on the testing->rolling approach yet, but over the past year a rough consensus has emerged that at least something ought to be done to provide a product for users interested in Debian, but requiring fresher package updates than stable (and its multi-year development cycle) can provide.
Nussbaum concludes his post by suggesting that the next step for the project ought to be a GR to measure support for the proposed plan. Zacchiroli indicated that he continues to hear strong support for some form of "user-targeted testing" from the public, but that their preference between CUT, rolling, and simple alterations to the release process remains muddled. He is preparing a user survey to clarify what the end users want. There is no time-frame set for either step at the moment, however, as the discussion continues over on debian-devel.
Comments (4 posted)
Brief items
Since this has been a major request from users for a long time, I can only
[be] cool with the idea of seeing the Debian project support a rolling
release. However I'm not pleased with the proposed ideas, since they don't
actually include any serious plan to make this happen. Sorry guys, but a
big GR that says « We want a pony rolling release to
happen » doesn't achieve anything.
--
Josselin Mouette
The feedback that started (or at least helped springboard) this massive
thread was that "when we're releasing, everything else grinds to a halt,
maybe it doesn't need to", followed by pondering whether there was a
better way to manage this. My initial proposal apparently smelled a bit
like rolling, which got out people from both the pro- and con- rolling
camps, and unfortunately it seems that a good deal of the discussion
ended up splintering off in that direction.
--
Sean Finney
Comments (none posted)
The OpenBSD project has
announced
the release of version 4.9. For the gory details see this
list of changes made between
OpenBSD 4.8 and 4.9.
Comments (5 posted)
The
Slackware 13.37
release is now available. "
We are sure you'll enjoy the many improvements. We've done our best
to bring the latest technology to Slackware while still maintaining the
stability and security that you have come to expect. Slackware is well
known for its simplicity and the fact that we try to bring software to
you in the condition that the authors intended." LWN
reviewed this release in March.
Comments (10 posted)
The Ubuntu 11.04 "Natty Narwhal" release is out. "
For PC users, Ubuntu 11.04 supports laptops, desktops and netbooks
with a unified look and feel based on a new desktop shell called 'Unity'.
This version supersedes Ubuntu Netbook Edition for all PC netbooks.
Developer reference images are provided for select Texas Instruments (TI) ARM
platforms, specifically the 'PandaBoard' and 'BeagleBoard'."
More information can be found in
this
press release
Full Story (comments: 32)
Distribution News
Debian GNU/Linux
Stefano Zacchiroli kicks off his second term as Debian Project Leader with
a few bits on his plans for the coming year. "
With a bit more
knowledge than last year, I can now comment about what the DPL---or at
least myself---can and cannot do for you. There is quite a bit of day to
day activity which goes on in the background (media queries, representing
Debian at events, "political" relationships with other Free Software
actors, etc). All that might be relatively uninteresting to you, but it
does happen and strive to keep the DPL pipe full already."
Full Story (comments: none)
The Debian Women project, in collaboration with the OpenHatch project,
will be holding an IRC event on building packages for Debian. "
On
Saturday May 7th, two tutorial sessions will be held on #debian-women on
irc.debian.org to help people rebuild a package for the first time."
Full Story (comments: none)
The sponsorship and paper deadline for DebConf11 is May 8, 2011. DebConf11
will take place July 24-30, 2011 in Banja Luka, Bosnia and Herzegovina.
"
You can still register for DebConf11 after the deadline, but will
need to pay a Professional registration fee if you want accommodation, and
pay for food yourself once you arrive. You can still submit events after
the deadline, but they may not be considered for official proceedings and
scheduling."
Full Story (comments: none)
Fedora
Blacksburg, Virginia has been selected as the location for FUDCon North
America 2012, currently scheduled for January 2012.
Full Story (comments: none)
Tom Callaway has
made
chromium-stable packages for Fedora. "
I've updated my scripts and made packages from the latest "stable" release, as tagged by Google (which, at the time of this writing is chromium-11.0.696.57 and v8-3.1.8). These packages have the same bundled libraries stripped out as my "unstable" packages, and are built from source on Fedora."
Comments (none posted)
Newsletters and articles of interest
Comments (none posted)
Over at Linux.com, Nathan Willis
tries out the developer preview of the MeeGo 1.2 Tablet UX (user experience). "
If you are an application developer, the Tablet UX preview gives you a nice playground in which to imagine your app. The real mettle will be shown when MeeGo 1.2 is officially released and a tablet SDK comes along. For now, all you need to know is that MeeGo tablets are going to provide as smooth a user experience as Android. Under the hood, you have a different set of questions to consider, but the MeeGo APIs are not substantially different between the various UXes."
Comments (9 posted)
Joe 'Zonker' Brockmeier
reviews
Xubuntu 11.04. "
I've used Xfce for many years, off and on, and I have to say that the Xubuntu distribution of Xfce is really nice. The default theme is very attractive, and I like the way it's been configured. Up top you have the Xfce application menu, and on bottom you have a "dock" with some of the more frequently used (or so they assume) applications like Firefox, GIMP, Thunderbird, the Xfce Settings manager, and so on."
Comments (none posted)
Page editor: Rebecca Sobol
Development
May 4, 2011
This article was contributed by Michael Stapelberg
A few weeks ago, the initial
announcement of x11vis, an X11 protocol visualizer, was posted to the xorg
mailing list. Only few people are developing low-level
X programs these days (think of xkill, xwininfo, etc.) and all these tools
actually work. Why would anyone need a tool to visualize X protocol traffic?
In a way, x11vis is similar to tcpdump: both are wire-level analysis tools —
tcpdump shows network traffic while x11vis shows X traffic. Even though most
people are not network engineers, from time to time it comes in handy to check
if a web application is really using SSL.
X basics
When an X client, say Firefox, connects to the X server, it first has to
authenticate itself. As soon as the connection is established, data flows in
two directions: the client can only send requests while the server sends
replies (to those requests), errors, and events.
Let's have a look at the very basic task of creating an empty window on your
screen. The client starts by sending a CreateWindow request which
initializes a specified X ID (an unsigned 32-bit integer) with a
position/size, a parent window, border, etc. Afterwards, properties such as the
window title or icon are set with the ChangeProperty request. To
actually display the window on your screen, a MapWindow request is
sent.
These are the requests which the client sends. None of these actually have
replies, but any request can generate an error — for example if you pass an
invalid parent window in the CreateWindow request. After the window
has been mapped (made visible on your screen), the client will receive a
MapNotify event. Other often used events include things like
KeyPress and ButtonPress, which are generated by keyboard
and mouse input.
Before x11vis: xtrace
Before x11vis, the standard tool for analyzing X applications was xtrace. Like strace, it prints a
textual representation of what happens. While that works
just fine and fits well in the Unix text world, it is not easy to use for
analyzing problems for several reasons:
- The vast amount of plain text output is very hard to understand or even to
navigate in. Each line of xtrace output starts with a number representing the
connection which is used for this particular packet (Firefox could be number 1,
GVim number 2, and so on). The rest of the line contains a full dump of the
packet, including all data. For some requests, this data can be more than I can
fit on my 1280x800 screen.
- In the X protocol, a lot of IDs are used. There are IDs for windows,
atoms, fonts, pixmaps, and so on. xtrace translates atom IDs to human-readable names,
so you will see "UTF8_STRING" instead of 0x113. However, such
a translation is completely absent for window IDs. Analyzing X sessions with
more than one client quickly becomes difficult.
- While it is possible, hiding specific events is tedious in a text
editor. When debugging a real-world problem, you are usually not interested in
packets such as InternAtom or PropertyNotify.
- A user might want to display information related to information in the
packet that is
currently being inspected (for
example all events for a affected window, not only the
CreateWindow packet). This is naturally not currently
possible with xtrace, as it presents only textual, non-interactive output.
Inside x11vis
x11vis strives to be better in those areas outlined above. It consists of two
main parts: the so-called "interceptor" and the GUI. The interceptor is a Perl
daemon that implements a proxy between your client(s) and the X server,
dissecting all packets that are sent through it and dumping them into a JSON
file. The code that dissects the raw bytes into a nice data structure is
auto-generated from the XML protocol description in xcb-proto. The GUI
parses this JSON file and displays all packets in a well-arranged fashion.
The GUI is not a stand-alone application, but is implemented as a web application
using jQuery. This decision was made because building the GUI on top of the
HTML Document Object Model (DOM) with CSS is a lot quicker than writing custom
widgets in Qt or GTK (in terms of development time). Also, it makes x11vis
easily usable on computers on your local network, which is a common setup when
debugging X problems.
Example: Comparing XCB and Xlib
I mentioned XCB as the project which includes the XML protocol
description. XCB stands for X C-language Bindings and is the successor of Xlib.
By automatically generating the bindings from the protocol description, XCB
achieves multiple goals. First of all, every function has a predictable name and
by using xcb_ as a prefix, and it does not clutter the namespace
(unlike
Xlib with types such as Font and Display). More
importantly, XCB does not hide the asynchronous nature of the X protocol from the
programmer. When a typical X application starts, it has to request the Atom
IDs for a number of atoms, say 20. With Xlib, there is the XInternAtom
function that returns the ID for a given name. XCB instead provides two
functions: xcb_intern_atom() and xcb_intern_atom_reply(). The
former returns a cookie which you pass to the latter to get the actual result.
The idea is that you place your requests as early as possible, do
something else, then fetch all the replies.
A good example of both XCB's asynchronous nature and x11vis is analyzing the
xwininfo(1) program. By starting:
xwininfo -id 0xf00 -children
the program will first query the given window (an iceweasel window in this
case) for all of its children and then request some properties for every child.
The screenshot above shows the x11vis output when using xwininfo 1.0.5, which
uses Xlib. On the left, you can see all the requests and replies, organized in
bursts. As Xlib is blocking, each burst contains only one packet.
Compare the Xlib shot
to the one above, where xwininfo 1.1.0 uses XCB to talk to the X server. While
you can still identify three round-trips, you can see that the burst on the
bottom of the screenshot contains requests for different information of more
than one window.
You can see that in the first burst, x11vis displays "Iceweasel" instead of the
window ID 0xf00, even though that information is only available later on. Also,
the description of each packet is a short representation of the most important
facts. The GetGeometry reply is labeled "(3362, 1112) 155 x 21" and
can be expanded by clicking on it. In the xtrace output, the equivalent is the
following line:
000:>:0003:32: Reply to GetGeometry: depth=0x18 root=0x000000be \
x=3362 y=1112 width=155 height=21 border-width=0
Example: Identifying a race condition in i3-wm
x11vis has been used multiple times to solve real-world X problems. For
example, in the i3 window manager, there was a problem with themed mouse
cursors: they would not show up on the very first window decorations that were
created around already existing windows, but only on window decorations for
windows that were created later on.
We know that the problem is related to creating windows and setting the cursor
for these windows. Therefore, I started by scrolling down to the first
CreateWindow request and checked if there were any X errors (pink
background in x11vis). And in fact, as you can see on the screenshot, there is
one corresponding Error packet for every Request trying to use the themed
cursor. You can see the bad_value of the Cursor error being c_0
(unnamed cursor 0) which is precisely the cursor ID we are setting in the
ChangeWindowAttributes request above.
I then used the search function of my browser to see where c_0 was
actually initialized. The location of the CreateGlyphCursor request
for c_0 was actually after the X errors. Now this
explains the symptom, but in the code, the order is correct: First, the
cursor is initialized (line 291), then the
existing windows are handled (line 425). Having a closer look at
the burst reveals that the cursor initialization is actually sent via
the separate Xlib connection instead of the main XCB connection. As
both connections buffer, my next guess was that the code neglects to
flush the Xlib connection. It turns out, the
guess was correct.
This bug was found in a short time due to two factors. On the one hand, X
errors can be spotted very easily in x11vis. On the other hand, distinguishing
the different connections requires only a quick glance to the top of each burst.
Conclusion
In this article, I explained how x11vis tries to help X developers: it
visualizes the X protocol on wire level, providing some helpful features like
markers, as well as folding or mapping human-readable names to connections and X IDs.
x11vis is still a young project and is looking for contributors. If you want
to help making x11vis become a better tool for you, please do not hesitate to
contact me at michael@x11vis.org or
go get
the source and documentation at the project web site.
Comments (5 posted)
Brief items
I am distinctly unimpressed by the fact that in GNOME 3 I have to
click where it says 'OFF' to make my VPN connect, and click where
it says 'ON' to make it disconnect.
Can I set up a translation so that _("ON") == "OFF" and _("OFF") =
"ON"?
--
David Woodhouse
Comments (none posted)
LWN
wrote about Harald Welte's work to
create a free GSM stack for handsets last September. As described there,
that work is aimed at tasks like protocol testing and security; it is not
an attempt to create a user-friendly phone based on free software. It
seems that Harald recently had a discussion with Richard Stallman, who
would like to see such a phone created. In response, the OsmocomBB project
will try to expand its scope (through the recruitment of additional
developers) to the creation of a basic phone stack suitable for real use.
So we agreed to make a public call for volunteers to work on that
aspect of the phone. I understand this will likely cause some
effort on our side (fencing off poeple who don't have the
necessary skills, integrating such code, finally deciding on a
RTOS to use, etc.).
However, I more or less see it as my (and our?) duty to realize the
potential of our protocol stack and baseband firmware. Next to all
our own self-motivated personal interestes, there is a bigger cause
that we can help along: Free Software based telephony. So the
least we can do is to try to find somebody who can work on that
part, and help developer[s] to interact with our code.
Expect to see a "job description" from the Free Software Foundation in the
near future.
Full Story (comments: 1)
Bino is a video player with an
emphasis on stereoscopic 3D and multiple display output; it is licensed
under GPLv3. The 1.0.0 release is out - the first stable release in the
project's history.
Full Story (comments: none)
Longtime GCC release manager Mark Mitchell has announced that he will be
stepping down from that position. "
Fortunately, in 2008, the GCC SC added three co-RMs: Jakub Jelinek,
Joseph Myers, and Richard Guenther. The three of them have shouldered
the majority of the load over the past couple of years. The SC
collectively and I personally have every confidence in their ability to
do continue onwards. In fact, I have no doubt that they will do a
better job together of managing future releases than I did."
Full Story (comments: none)
The GNU MediaGoblin project has announced its existence. MediaGoblin is an
effort to create a free, distributed photo sharing service, with video and
other media planned for later releases. "
GNU MediaGoblin is being built by free software activists. Chris
Webber and Will Kahn-Greene, both longtime Miro contributors, are
leading the Development Team. Matt Lee and Rob Myers from FooCorp, the
makers of GNU FM (the software that powers Libre.fm) and GNU social,
are providing infrastructure. Deb Nicholson, founder of the Women's
Caucus, is helping with community outreach." More information can
be found on the
MediaGoblin web
site.
Full Story (comments: none)
Version 2.2.0 of the PacketFence network access control system is out.
"
This is a major release bringing new hardware support, new features,
enhancements, documentation updates, bug fixes and new
translations." Significant changes include support for a number of
new routers, better user agent blocking, automatic Apache configuration
tuning, and more.
Full Story (comments: none)
The first PostgreSQL 9.1 beta is out. "
PostgreSQL 9.1 contains a
huge volume of new features, possibly more any single release of PostgreSQL
before. These features also include several innovations which PostgreSQL
is the first database system to have." See
this article for more information on the 9.1
release.
Full Story (comments: none)
The
PyPy
1.5 release is out. PyPy is an alternative Python interpreter with a
focus on speed. "
This release updates PyPy with the features of
CPython 2.7.1, including the standard library. Thus all the features of
CPython 2.6 and CPython 2.7 are now supported. It also contains additional
performance improvements."
Comments (8 posted)
Lennart Poettering and Kay Sievers have posted
a
description of how they intend to expand systemd to take over many of
the functions currently handled by ConsoleKit. "
It mostly focuses
on cleaning up seat and login management for users, i.e. the first big
steps in bringing systemd to user sessions. On the way want to fix
multi-seat support properly and running services outside of a session, and
we will get rid of CK." (
Note: a copy of the document has
been posted in the comments for those who are having trouble accessing the
original).
Full Story (comments: 69)
Newsletters and articles
Comments (none posted)
Dave Phillips
takes a look at
several applications that extend the JACK audio server. "
By this point it should be obvious that the programs profiled here are dedicated utilities with clearly defined tasks and goals. Xjadeo is no exception - it simply runs a video in sync with a JACK time master such as Ardour or QTractor, and that's just about all it does. However, it enables the user to match music to video at the frame level quickly and easily. I've tested Xjadeo extensively in Ardour, and I encountered no errors at any of Ardour's supported frame rates. The program also supports MTC (MIDI Time Code) perfectly, as far as my uncomplicated tests could tell."
Comments (none posted)
IT World
covers
a proposal by Oracle to hand control of the Hudson project to the
Eclipse Foundation. "
Under the terms of the proposal, all aspects of
Hudson, including source code, domain name, and copyright, would fall under
the auspices of the Eclipse Foundation (EF), should the proposal be
approved by the Hudson community. This would be a very significant step for
Hudson, since the EF is rather highly regarded as a neutral member of the
open source community and Eclipse governance should go a long way towards
easing any misgivings anyone might have about Oracle maintaining control
over Hudson."
Comments (7 posted)
Lennart Poettering has put up
a lengthy page on why
systemd exists. "
systemd is also a big opportunity for Linux
standardization. Since it standardizes many interfaces of the system that
previously have been differing on every distribution, on every
implementation, adopting it helps to work against the balkanization of the
Linux interfaces. Choosing systemd means redefining more closely what the
Linux platform is about. This improves the lifes of programmers, users and
administrators alike." (Thanks to Paul Wise).
Comments (218 posted)
Page editor: Jonathan Corbet
Announcements
Brief items
The Apache Software Foundation has
announced
that Oracle has served it with a subpoena "
requiring the production
of documents related to the use of Apache Harmony code in the Android
software platform, and the unsuccessful attempt by Apache to secure an
acceptable license to the Java SE Technology Compatibility Kit."
The full document has been provided for those curious to see what Oracle is
asking for.
Comments (3 posted)
The Electronic Frontier Foundation's Peter Eckersley has put out a
call to action for open wireless networks. It goes beyond just advocating for more open access points and sane laws to protect people who open their APs by looking at potential technical fixes for problems like bandwidth hogging. "
The best solution to this problem is to have WiFi routers which make it very easy to share a certain amount of bandwidth via an open network, but simultaneously provide an encrypted WPA2 network that gets priority over the open network. Some modern routers already support multiple networks like this, but we need a very simple, single-click or default setting to get the prioritization right."
Comments (48 posted)
KDE e.V. has
announced
the availability of its
report (PDF)
for the fourth quarter of 2010. The report looks at things that happened
in KDE, the supporting members, recaps from various gatherings, and
includes a look at the project's finances.
Comments (none posted)
Articles of interest
Steven J. Vaughan-Nichols
talks
with Attachmate CEO Jeff Hawn about the future for Novell, SUSE and
openSUSE. "
SUSE sponsorship and participation in key open-source projects is a fundamental element of the business. This commitment is driven by a desire to contribute to and collaborate with the community in a way that fosters the success of open source technologies overall and creates the greatest value for our customers. The openSUSE project is a great example of vibrant and healthy collaboration. SUSE sponsorship and participation in projects like openSUSE creates great value for the community and also for SUSE customers who benefit from the innovations and advancements we create together."
Comments (13 posted)
New Books
No Starch Press has released "The Book of CSS3", by Peter Gasston.
Full Story (comments: none)
O'Reilly Media has released "JavaScript: The Definitive Guide, Sixth
Edition", by David Flanagan.
Full Story (comments: none)
Pragmatic Bookshelf has released "Code in the Cloud", by Mark Chu-Carroll.
Full Story (comments: none)
Resources
The May edition of the Free Software Foundation Europe covers competition
authorities and software patents, the pdfreaders.org campaign, and several
other topics.
Full Story (comments: none)
Contests and Awards
Registration is open for the Pyggy Awards. "
This time, entry is open
to any Python-based game, not just PyWeek games, and not just games
developed during PyWeek or Pyggy. So if you've had a Python game project on
the back burner, you no longer have any excuse for not finishing it off!
Get going!" Judging will be from July 17-31, 2011.
Full Story (comments: none)
Calls for Presentations
PyOhio 2011 will take place July 30-31, 2011 in Columbus, Ohio. The call
for proposals is open until June 3, 2011. "
PyOhio invites all
interested people to submit proposals for scheduled talks, tutorials, and
panels. All topics of interest to Python programmers will be
considered. Standard presentation slots will be 40 minutes plus a 10 minute
question-and-answer period."
Full Story (comments: none)
Upcoming Events
Events: May 12, 2011 to July 11, 2011
The following event listing is taken from the
LWN.net Calendar.
| Date(s) | Event | Location |
May 9 May 13 |
Linaro Development Summit |
Budapest, Hungary |
May 9 May 13 |
Ubuntu Developer Summit |
Budapest, Hungary |
May 10 May 12 |
Solutions Linux Open Source 2011 |
Paris, France |
May 10 May 13 |
Libre Graphics Meeting |
Montreal, Canada |
May 11 May 14 |
LinuxTag - International conference on Free Software and Open Source |
Berlin, Germany |
| May 12 |
NLUUG Spring Conference 2011 |
ReeHorst, Ede, Netherlands |
May 12 May 14 |
Linuxwochen Österreich - Linz |
Linz, Austria |
May 12 May 15 |
Pingwinaria 2011 - Polish Linux User Group Conference |
Spala, Poland |
May 16 May 19 |
PGCon - PostgreSQL Conference for Users and Developers |
Ottawa, Canada |
May 16 May 19 |
RailsConf 2011 |
Baltimore, MD, USA |
May 20 May 21 |
Linuxwochen Österreich - Eisenstadt |
Eisenstadt, Austria |
| May 21 |
UKUUG OpenTech 2011 |
London, United Kingdom |
May 23 May 25 |
MeeGo Conference San Francisco 2011 |
San Francisco, USA |
| June 1 |
Informal meeting at IRILL on weaknesses of scripting languages |
Paris, France |
June 1 June 3 |
Workshop Python for High Performance and Scientific Computing |
Tsukuba, Japan |
June 1 June 3 |
LinuxCon Japan 2011 |
Yokohama, Japan |
June 3 June 5 |
Open Help Conference |
Cincinnati, OH, USA |
June 6 June 10 |
DjangoCon Europe |
Amsterdam, Netherlands |
June 10 June 12 |
Southeast LinuxFest |
Spartanburg, SC, USA |
June 13 June 15 |
Linux Symposium'2011 |
Ottawa, Canada |
June 15 June 17 |
2011 USENIX Annual Technical Conference |
Portland, OR, USA |
June 20 June 26 |
EuroPython 2011 |
Florence, Italy |
June 21 June 24 |
Open Source Bridge |
Portland, OR, USA |
June 27 June 29 |
YAPC::NA |
Asheville, NC, USA |
| June 29 |
Scilab conference 2011 |
Palaiseau, France |
June 29 July 2 |
12º Fórum Internacional Software Livre |
Porto Alegre, Brazil |
July 9 July 14 |
Libre Software Meeting / Rencontres mondiales du logiciel libre |
Strasbourg, France |
If your event does not appear here, please
tell us about it.
Page editor: Rebecca Sobol