User: Password:
|
|
Subscribe / Log in / New account

LWN.net Weekly Edition for May 5, 2011

The Grumpy Editor gets organized with Astrid

By Jonathan Corbet
May 4, 2011
Your editor does not normally need any computerized assistance when it comes to knowing what needs to be done next; it's usually just a matter of dealing with whatever is on fire at the moment. Still, there are times when it would be nice to have a task management tool with a longer view, where a different set of priorities can be expressed. Recently it became time to work on one of those longstanding todo items and examine Astrid as a possibility for that role.

[Astrid task list] A task manager which works with the phone obviously makes sense; one cannot always be at the keyboard when trying to figure out what to do next. Anybody who has an Android-based phone has a long list of task management applications to choose from; simply working through the list is the sort of todo item that can make one decide to just drink beer and mess around on the net instead. So why pick Astrid out of the pile of applications? It comes down to two reasons: it comes well recommended, and it's free software - Astrid is licensed under GPLv2 and can be downloaded from github.

Astrid provides the kinds of features that one might expect from this type of application. One can enter tasks and assign an importance and a deadline to each. This is the 21st century, so, naturally, tasks can be tagged as well. A timer mechanism allows the user to track how much time goes into the accomplishment of each task - a useful feature, perhaps, if that time will subsequently be billed for. And, naturally, one can mark tasks as being done.

There are various options for how obnoxiously Astrid should nag as deadlines approach and just how hard it should be to make the notifications go away. A true procrastinator with a masochistic streak can configure the application to a phone-smashing level of annoyance. For the rest of us, simple notifications and, possibly, a desktop widget will do.

In general, Astrid works as advertised. Entering new tasks is easy, and the notifications work as expected. There is a whole set of options for filtering the display of tasks which must be useful to some people; [Astrid filters] work-oriented tasks can be separated from shopping lists, for example. The desktop widget works but, like a number of other Android applications, it makes poor use of the space which is given to it. About half of the allotted space contains task information - truncated to the point of near uselessness. Some thought put into the organization of that widget might make it worth dedicating scarce desktop space to. As it is, your editor couldn't justify deleting something else to make space for it.

One of the sticking points with phone-based task managers is the simple inconvenience of entering tasks through an on-screen keyboard. The motivation to turn an "oh, yeah, I have to do X" moment into a documented task fades when it requires using a small screen and a tiny keyboard. The obvious answer to this problem is synchronization with some sort of online service, preferably one which enables access over the web from real computers as well. To this end, Astrid will synchronize with the Google mail task list or with an online service called Producteev. Gmail provides simple online storage and synchronization, while Producteev has (for a fee) all kinds of options for shared task lists, report generation, and more. The fact that use of SSL is a paid option on Producteev is not particularly encouraging, but so it goes.

The online documentation talks about synchronization with Remember the milk, but version 3.7.3.1 does not appear to actually support that service.

Your editor only tried out the Google-based synchronization. It works as expected; tasks move nicely between Google and the application. Of course, using this feature requires providing one's Google password to Astrid and providing one's task lists to Google, but, after all, it's not like Google knows anything else about us anyway.

This kind of synchronization is useful, and, honestly, worrying about storing some todo items on Google's server seems pretty silly if one is willing to take the privacy-reducing step of walking around with an Android phone in one's pocket. Still, it would be nice to be able to synchronize applications like this to a more private backing store - preferably one which does not require a great deal of pain to set up. We need a mechanism by which one can install a simple backup appliance onto a server anywhere on the net; otherwise we'll always be dependent on services provided by companies whose interests will not always align with our own.

(Along these lines, it's worth noting that Astrid, in its default configuration, phones home to a service called Flurry with "anonymized" information about how the application is used. This reporting can be turned off in the settings menu.)

While Astrid is free software, it supports add-ons which are not. The Astrid store offers a couple of paid add-ons. One of which offers timers, "bigger better widgets," voice-based features, and "No Ads!" A fair amount of time playing with Astrid has not turned up an advertisement yet, but, one assumes, that could happen at some future time. There is also a "Locale add-on" which allows tasks to be keyed on the phone's location.

Some of these features, seemingly, could be added by anybody willing to dig into the Astrid source. Certainly, any sort of advertising antifeature could easily be subtracted that way. One can only assume that patches to that end would not be accepted upstream. Some quick searching failed to turn up anybody who has rebuilt Astrid with a different feature set; it is nice to know that a fork is a possibility, though, should Astrid's masters take an overly user-hostile direction.

There could be changes in store for Astrid; recently its developers announced that they had "brought in some external funding" and will be focusing more on Astrid in particular. There will be a new synchronization service at Astrid.com, an iPhone version, and more. Whether this money - and the revenue expectations that certainly come with it - will push Astrid in a less friendly direction remains to be seen. Hopefully the open-source nature of the code will help to keep the developers focused on keeping users happy.

For now, there seems to be little to worry about; the only visibly commercial aspect of current Astrid builds is the "add-ons" button in the main menu. Astrid's developers have been successful in creating a task management application which is appealing to a wide range of people. With luck, it will just get better now that more developer time will be made available.

Comments (11 posted)

New features in PostgreSQL 9.1

May 2, 2011

This article was contributed by Josh Berkus

The first beta release of PostgreSQL 9.1, the next annual release of the well-known open source relational database, came out on May 2nd. The new version contains many new features sure to please the hearts of database geeks around the world. There are enough new features, in fact, that we can't cover them all in this article. Here's four of the more interesting ones:

Transaction-controlled synchronous replication

Last year, PostgreSQL 9.0 introduced asynchronous binary replication in PostgreSQL for the first time. Building on that foundation, version 9.1 includes an option for synchronous replication as well. Synchronous replication means that transactions do not return to the application until they have been received both on the master server and on a replica server. This ensures that data from any committed transaction cannot be lost even if the master server goes offline permanently.

This feature culminates a three-year-long development effort by NTT Open Source and 2ndQuadrant, which will allow Japanese telecommunications giant NTT to finally replace most of their Oracle servers with PostgreSQL. The PostgreSQL clustering project PostgresXC is the remaining part of this replacement effort.

There is also a substantial penalty in response time for synchronous replication, since the transaction needs to be written to disk on two servers before returning. To alleviate this, PostgreSQL goes beyond the synchronous replication offered by other database systems, and offers the ability to control whether commits are synchronously or asynchronously replicated on a per-transaction basis. This allows application developers to distinguish between critical data which must not be lost (like financial transactions), and less critical data for which response time is more important, to optimize the performance of the system.

Each replica (or "standby") gives itself a name in its recovery.conf file when it connects to the master:

    primary_conninfo = 'host=master01 user=replicator application_name=replica1'

Then the master sets a priority list of synchronous replicas in its postgresql.conf file:

    synchronous_standby_names = 'replica1'

Then you can commit a transaction as synchronous or asynchronous:

    -- commit synchronously to the standby
    SET synchronous_commit = 'on';
    UPDATE user_balance SET balance = 573.29 WHERE user_id = 40021;
    -- messages are not important, send asynchronously to the standby
    SET synchronous_commit = 'local';
    INSERT INTO user_messages VALUES ( 40021, 57843, 'HI!' );

PostgreSQL also supports having a priority list of synchronous replicas, thus supporting higher-availability configurations. The new pg_stat_replication database view allows database administrators (DBAs) to monitor the status of database replicas, and other administrative settings control what to do if the standby goes offline. In future versions, PostgreSQL plans to support different synchronization modes, as well as the ability to have "quorum" synchronization. The latter is for "synchronize to two out of these five servers" replication, for improved response time without lowering data integrity.

Unlogged tables

The burgeoning popularity of non-durable "NoSQL" databases like Redis and MongoDB, along with the well-established Memcached, have demonstrated that there is a class of data for which data loss after a crash is less important than response time. EnterpriseDB developer Robert Haas has added a feature to PostgreSQL to handle this kind of data: unlogged tables.

The word "unlogged" refers to the transaction log, which guarantees durability of writes. With an unlogged table, if the database server shuts down, on restart the table will be empty. However, by not syncing to disk, you can write to unlogged tables up to 20 times as quickly as to durable tables. They can also be thought of as global temporary tables or "in memory" tables.

Unlogged tables are intended to hold data which is high-volume but not very valuable. For example, many PostgreSQL users currently log website user session activity to Memcached. Unlogged tables are also useful for holding raw bulk load data during batch processing in data warehouses. Both of these uses will simplify life for PostgreSQL users by allowing them to reduce the number of single-purpose databases in their application stack.

SELinux integration

Another multi-year development effort that is being first released in PostgreSQL 9.1 is the unification of data access control in PostgreSQL and SELinux. NEC programmer Kaigai Kohei has spent the last three years developing SE-PostgreSQL, which integrates the concept of label-based data access rules into PostgreSQL. PostgreSQL is the first SQL database system to have this level of integration.

This feature is intended for high-security environments where even the DBA may have limited access to data, such as for national security databases. Security policies established at the system or network level through SELinux may now be applied to table, view, function, and column permissions. This enables the establishment of a single consistent security policy regardless of whether data is in files or inside the database.

Regular readers of LWN will be aware that SE-PostgreSQL had a lot of trouble getting integrated into mainstream PostgreSQL. Indeed, the SE-PostgreSQL which appears in 9.1 beta is almost completely rewritten from the original version. First, with the help of other hackers, Kohei rewrote all of the calls to SELinux permissions lookups as hooks which are enabled at PostgreSQL compile time, or not, in order to remove any impact on non-SE-enabled installations. Second, all of the management and tools for SE-PostgreSQL were moved to an optional loadable module.

Finally, certain actions had to be exempted from Mandatory Access Control because of difficulty of implementation. Chief among these is row-level access control, since there are a number of unresolved theoretical issues about how to implement it. This means that, of course, Kohei still has work to do.

Extensions and PGXN

Speaking of optional modules, one of the primary strengths of PostgreSQL has always been its extensibility. Users and developers have defined data types, operators, functions, and aggregates to support a variety of types of specialty data, including genomic, inventory, mathematical, astronomical, time series, and geological data. Other plug-ins support integration with queues, compatibility with other database systems, and experimental new features. The most famous plug-in to PostgreSQL is probably the PostGIS geographic database.

However, most of these "plug-ins" to PostgreSQL have been difficult to find, install, back-up, and upgrade, limiting their use or even causing users to reinvent them several times. That's changing with version 9.1.

First, Dimitri Fontaine of 2ndQuadrant has created a packaging concept in PostgreSQL 9.1 called Extensions. By packaging a collection of database objects as an Extension, the administration of adding and removing plug-ins, and most importantly upgrading them, becomes easily scriptable. Users can even create their own extensions for in-house database customizations which need to be versioned.

Second, David Wheeler of PostgreSQL Experts created PGXN, "The PostgreSQL EXtension Network". PGXN is patterned on online repositories for programming languages like Perl's CPAN and Ruby GEMs. PGXN will give PostgreSQL users a central place to find and download PostgreSQL plug-ins not distributed with the core database. Currently in development is a unified download and installation tool with dependency tracking, like CPAN or apt-get.

PostgreSQL development

The PostgreSQL project has been experimenting for the last couple of years with changes to its development cycle, patch review, and release processes. The biggest problem the project struggles with is the constant torrent of patches and feature ideas being submitted, which has been growing at a much faster rate than the pool of reviewers and committers has. 9.1, for example, will contain more new major features than 9.0 did. 9.0, in turn, had more features than 8.4.

As a result, contributors to the project frequently discuss tinkering with the development cycle in order to optimize reviewer, contributor, and developer time. A recent discussion grappled with those topics.

Should the project branch off version 9.2 right after beta or wait until the release candidate? Historically, PostgreSQL has refused to begin development on a new version before development on the prior version was completed, and that seems likely to continue to be the policy. PostgreSQL developers have found that working on more than one version at a time causes a great deal of backporting, as well as lack of attention to fixing bugs.

Should PostgreSQL continue to have CommitFests (periods of intensive focus on patch review and merging rather than development) every other month or move to a one-week-a-month cycle? Doing a week-a-month would give committers more of their own time as well as giving developers faster feedback. However, there are some serious doubts as to whether it's even possible to implement given the distributed, part-time nature of most of PostgreSQL's contributor base.

What should the schedule be for 9.2 development? This depends, of course, rather strongly on answers to both of the above. 9.0 and 9.1 both began development in July and released 12 to 14 months later. Some contributors feel that a regular annual schedule is a benefit while others don't find it important. This discussion is far from over, and may even result in changes which are tried and then reverted if they don't work.

In related news, the PostgreSQL code base has passed 1 million lines of code with version 9.1. And long-time project contributor Magnus Hagander has been elected to the PostgreSQL Core Team.

There are, as mentioned, many other features in this beta release, which you can read about in the release notes. Several of these features are innovations which will appear in PostgreSQL before any other relational database. Since this is a beta version, it is considered not yet ready for production use. The PostgreSQL developers expect the final version to be out in two to four months, depending on testing and bug-fixing.

Comments (13 posted)

Who maintains RPM? (2011 edition)

By Jonathan Corbet
May 3, 2011
Back in 2006, LWN looked at the rather complicated story around the maintainership of the RPM package manager. Given the importance of this tool for any RPM-based distribution, the lack of a clear story on how it was being maintained was somewhat discouraging. Later that year, the Fedora project announced the creation of a new, community-oriented project around RPM. Since then, things have been on the quiet side, but recent events show that the RPM story has not yet run its course.

The above-mentioned RPM project lives on at rpm.org; the 4.9.0 release was announced at the beginning of March. The code is actively maintained and sees the addition of some small features, but the project does not show any real signs of having big plans for the future. Only a handful of committers show up in the repository; almost all of them work for Red Hat. By all appearances, RPM is at least halfway into maintenance mode.

But rpm.org is not the only RPM out there; a fork exists at rpm5.org. This version is maintained primarily by Jeff Johnson, a former Red Hat employee who has set out to create a better RPM outside of the company. This version has focused on wider portability, has added features like new compression formats, and a number of other things; the full feature list is not exactly easy to find, and searching for the significant entries in the changelog is not a project for anybody who is short on time. One of the key features, though, seems to be transactional package management (also called "RPM ACID"); the idea here is to ensure that any RPM operation either succeeds or fails fully, regardless of what may happen in the middle. According to Jeff, killing an rpm4 operation in the middle can leave a corrupted system behind; rpm5 is meant to eliminate that possibility.

The rpm5 fork arguably has more development activity and some interesting new features. That said, it has remained a relatively obscure project; it has been picked up by distributions like Alt Linux, ArkLinux, and Unity Linux, but has not been seriously considered by the larger distributors. That has changed, though, in recent months; the Mandriva 2011 plans include a switch to rpm5. That has brought a lot more attention to this fork, with some interesting outcomes.

It has not always been clear to everybody in the Mandriva camp that this switch is a good idea; back in November, Eugeni Dodonov started an extensive discussion by asking about the reasoning behind this change:

As far as I know, rpm4 is supported by Fedora/RedHat/CentOS and SUSE/OpenSUSE among the major vendors. Rpm5, on its hand, is used by AltLinux and Unity Linux. RPM4 is mostly stable, with long-time pending issues and its behavior is well-known. RPM5 is under constant development, with many features and without any large install base.

Many others spoke up on both sides of the issue; the discussion grew rather heated and unpleasant at times. The decision to go with rpm5 seemed to be motivated by a number of considerations. Some of the new features were attractive to Mandriva. The development community for rpm5 was seen as being more active and open than for rpm4, which was seen as dominated by Red Hat and relatively closed. Mandriva was carrying a long list of patches against rpm4 which, evidently, could be upstreamed into rpm5 but not into rpm4. The notion of compatibility with the other large RPM-based distributions was seen as illusory at best; each distribution was seen as heavily patching rpm4 for its own needs and it has always been difficult to do non-native installations of RPM packages. And, importantly, Mandriva's RPM maintainer, Per Øyvind Karlsen, wanted to go that way:

I've made it clear several times that I've planned on going with rpm5, and given that I'm the maintainer, have the best knowledge of and this change really shouldn't affect people in a negative way if you look beyond the reckless updating from testing. As I am the one to maintain it, and it shouldn't have inconveniences pushed on others, I find it my own decision to make whether I want to maintain a totally messed up rpm.org version which is bogged down with patches and pain to regenerate for every release.

So that is how it went. Per Øyvind's assertion that the change "really shouldn't affect people in a negative way" turned out not to be quite the case, though; Mandriva's "cooker" mailing list has been dominated by discussions of rpm5-related issues ever since. Other small transitions - to systemd, for example - have not generated a fraction of the traffic. A transition of this type was never going to be easy; it has been further complicated by the facts that integration with Mandriva's higher-level tools has been a big job and rpm5 has had some growing pains in the process of scaling up to a distribution of the size and complexity of Mandriva.

It must be said that, through this process, Jeff has clearly put in a massive amount of time supporting Mandriva. One could have easily gotten the impression that he had been hired by the company as part of this project. As problems were reported, Jeff worked to solve them. It is a rare development project that will put that much effort into supporting its users - even high-profile users; Jeff has performed a real service for Mandriva.

That support faltered a bit at the end of April, when Jeff abruptly announced that the rpm5 mailing list archive (and much of the rest of the site) had been taken down. His responses to queries about why this happened are best described as abusive; by his own admission, he wanted to stop the discussion as quickly as possible. The site has since returned; there is also a new rpm6.org which, for now, redirects to rpm5.org.

What is going on is far from clear, but there are hints in this message from Jeff where he accuses an ex-Mandriva developer of being "responsible for destroying the rpm5 brand." He is clearly tired of dealing with problem reports and complaints about rpm5, many of which, he says, are not really related rpm5 problems. But much of the trouble seems to come from the fact that Mageia - a fork of Mandriva - has not followed Mandriva into the rpm5 transition. Mageia's position, as described by Anne Nicolas, seems reasonable: the Mageia developers are busy enough just trying to get a releasable distribution together. Given all that needs to be done, throwing in a low-level package management system change just didn't seem like the best idea.

It would appear that there is some bad blood remaining between the Mandriva and Mageia camps. There are hints of non-public discussions that have been rather less friendly than what can be seen on the public lists; Jeff and rpm5 may well have been dragged into some of that. At least, that's what one might conclude from comments like:

You've had *MONTHS* to make peace with rpm5 and mostly haven't. rpm5 is being cited as a "fiasco" and is blamed for most anything you don't like. What you don't like has nothing whatsoever to do with this "tourist with camera" wandering the dungeons hacking on RPM.

I can tell you that a "fork" is a huge waste of time & energy from first hand experience. I can also say that the M&M distros are more similar than different, and strongly suggest that you make peace somehow and move on.

Since then, the tempest seems to have mostly passed, and Jeff is back to talking about making rpm5 work better with Mandriva. But this incident has been a wakeup call for some people in the Mandriva community who now feel they have cause to worry about the foundation on which the 2011 release is being built.

It is going to be interesting to watch what happens from here. Mandriva appears to be well committed to the rpm5 transition at this point, and, seemingly, things are beginning to stabilize. If rpm5 performs well in the final Mandriva 2011 release, it could motivate questions from users of some of the other distributions on why they are stuck with the "older" version. Alternatively, users could see the pain Mandriva has been through and, if the result doesn't appear to be worth it, they may decide that they're happier with their relatively boring rpm4. Either way, it seems that this particular drama has not played itself out yet.

Comments (89 posted)

Page editor: Jonathan Corbet

Inside this week's LWN.net Weekly Edition

  • Security: Seth Schoen stumps for SSL; New vulnerabilities in firefox, kernel, php, qemu-kvm, ...
  • Kernel: Raw events and the perf ABI; Expanding seccomp; Building the kernel with Clang.
  • Distributions: Debian rolling proposal; OpenBSD, Slackware, Ubuntu, ...
  • Development: X11 wire-level analysis with x11vis; Free GSM phones; Bino, GCC, PyPy, ...
  • Announcements: ASF subpoenaed by Oracle; EFF: Open Wireless Movement; Interview with Linus Torvalds
Next page: Security>>

Copyright © 2011, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds