LWN.net Weekly Edition for February 24, 2011

The Freedom Box gets off the ground

The Freedom Box is starting to roll, with a fundraising drive that met its goals in a few short days, along with a newly formed foundation to oversee its development. What started as an idea in a talk given by Eben Moglen just over a year ago has more recently gained a lot of momentum. What can we expect to see from this "personal server running a free software operating system, with free applications designed to create and preserve personal privacy", and when can we expect to see it?

The "when" question may have become somewhat clearer since the "Push the FreedomBox Foundation from 0 to 60 in 30 days" Kickstarter fundraising effort has clearly been a success. The fundraising drive was set up on February 17, with the goal of getting $60,000 in donations in 30 days, but it has exceeded that—and quickly. As of this writing, there are more than 650 supporters who have donated over $64,000 in just five or six days. Based on the Kickstarter appeal, reaching the goal (and quite possibly far surpassing it) should result in a software release in six months. With luck, that means we will see the first Freedom Box release in August or so.

It should be noted that, perhaps a bit oddly, the project is called "Freedom Box", but the foundation is the "FreedomBox Foundation".

Like the Diaspora fundraising drive last May, the FreedomBox effort shows that there is a pool of money available for privacy-respecting tools and applications. So far, Diaspora, which is an attempt to provide a privacy-respecting Facebook alternative, has delivered some code and is running a private alpha. Whether Diaspora gains any sort of traction remains to be seen, but it may fall flat because the vast majority of internet users do not seem to put privacy anywhere near the top of their priority lists.

But, clearly some internet users do have a privacy focus and are willing to fund projects they see as advancing that agenda. There are also a large number of people whose privacy may be more than just a preference and is, instead, a life or death matter. For those folks, what will the Freedom Box offer? The high-level goals are spelled out on the foundation's website; the basic idea is to decentralize web applications and services, so that governments, companies, and other organizations will find it difficult to disrupt or eavesdrop on Freedom Box users' communications. To accomplish that, the project's goals are quite ambitious.

The goals

Unlike some other projects, Freedom Box is not just a software solution. It is targeting various types of low-end hardware servers to run a Debian-derived Linux system that implements its plans. The current targets are so-called "plug computers" (or "plug servers"), which are small, low-cost, low-power computers that often have the form factor of a "wall wart" power supply. These devices would be always-on gateways to the internet, with an interface that allows them to be used by both technically savvy and less sophisticated users.

While providing "safe social networking" is one of the aims of the Freedom Box, it is only part of the picture. The project wants to protect users' data as well as their communications, including internet traffic, email, and voice. Beyond that, Freedom Box is specifically targeted at routing around ISPs' restrictions on the types of traffic they will carry, as well as attempts by governments to do similar traffic restrictions. In short, the goals of the Freedom Box live up to Moglen's original vision, as spelled out in his February 2010 talk at the New York branch of the Internet Society, as well as those outlined in a more recent talk at FOSDEM 2011: it is geared towards restoring users' freedoms.

Those freedoms are best guarded by keeping our data safe within the walls of our homes, because there are typically more legal protections there than there are when storing data on some company's servers. We have already seen that companies will often bow to governmental pressure in ways that would be more difficult to orchestrate when the data is spread out across the net. To that end, Freedom Box also plans to provide ways to securely back up encrypted data on friends' and neighbors' servers. In addition, it will provide ways for those under repressive regimes to anonymously publish information, such that those regimes will find it difficult to stop or track down the publishers. If the FreedomBox is going to handle all of these kinds of things, obviously the security of the device itself is paramount, but it is also targeted at protecting other systems in the home that live "behind" the Freedom Box.

Did we mention that it is an ambitious vision? It is that, without question, and will certainly not be fully delivered in the six-month time frame. One would guess it will be a few years before it fulfills all of its goals, but those goals are important.

Development

Development, or at least planning, has been taking place on the Debian wiki's Freedom Box project page. One would guess that the infusion of some funding will accelerate the process, but there is already a fair amount of information about the parts and pieces that could come together as the Freedom Box. As Moglen has said, almost all of those pieces needed for the project already exist in one form or another. In some sense, the project will be an integration effort for many different free software projects. That part will be tricky for sure, but fairly straightforward; the harder part will be getting the user interface "right".

The Debian Freedom Box "vision statement" describes that part of the problem well:

Making all of the envisioned functionality easy to configure and use will be an enormous challenge. Focusing on just a few—or even one—hardware platform(s) will help with that process, but there are a lot of disparate pieces to be integrated—and to be made to mostly "just work". It would appear that the planning for that part has barely started, but there has been some work done on defining and describing the underlying guts of the system.

The "Design and ToDos" page outlines the base system as well as the extensions—based on existing free software tools—that will replace various "cloud" services (Facebook, Twitter, Flickr, Dropbox, Google Calendar and Reader, and so on) that are in use today. It also has a list of issues that underscores the amount of work to be done.

The base system will be based on Debian (obviously) with encrypted filesystems (which immediately raises a question about key/password management for users), a web server, AppArmor for security, a configuration system possibly based on Config::Model, and Tor for anonymous communications. The server extensions that are listed cover all kinds of different services including web-based email (Roundcube, SquirrelMail, ...), blogging (Wordpress, Drupal), file sharing (Sparkleshare, ownCloud, ...), telephony (Asterisk, Yate), social networking a la Facebook (Appleseed, Jappix, Diaspora), and so on. The extension list seems to cover most or all of the web applications and services that folks are using today, but it's a little hard to say if, for example, SquirrelMail is truly an acceptable Gmail alternative.

The project mailing list starts back in August, but the posting volume trailed off late last year. Since the advent of the FreedomBox Foundation, along with Moglen's FOSDEM talk, things have rapidly picked back up. Discussions there have mostly centered on high-level requirements, thoughts, and plans.

Funding and the role of the foundation

One of the more interesting postings to freedombox-discuss, was a transcription of an IRC question and answer session with Ian Sullivan, who is helping to coordinate the activities of the foundation. The Q&A was held on February 18 on the #freedombox channel on OFTC, and outlined some of the goals of the foundation along with the plans for the funds that are being raised:

Presumably, how the funds will be used will be dependent on how much is raised. The current plan is not to hire full-time developers—$60,000 wouldn't go very far in doing so anyway—but to use the funds as something of a seed to get more people involved. Sullivan mentioned the idea of "buying plug computers and sending them to developers who promise to work on the project" as one possibility for using the funds. But, part of the idea of the funding drive is to increase the visibility of the project and, hopefully, increase the enthusiasm of potential contributors:

Clearly the project and the foundation are in their early stages, with much left to be worked out—not just technically, but organizationally as well. The foundation's web page notes that "in coming weeks we will be announcing here the technical leads for Freedom Box and its component projects". The foundation is incorporated as a Delaware non-profit and will seek non-profit recognition by the US Internal Revenue Service (IRS) "as soon as the paperwork is ready", Sullivan said.

Sense of urgency

Recent unrest in the Middle East, along with Egypt and Libya governments' internet shutdowns, have clearly increased the sense of urgency in the need for a device like the Freedom Box, as the Kickstarter appeal makes clear:

In the end, $60,000 is not a lot of money for a project of this scope. Even if the amount doubles (or more) before the Kickstarter campaign ends, it's really just a drop in the bucket. Moglen was quoted in the New York Times as saying that "slightly north of $500,000" would be enough to develop Freedom Box 1.0 in a year, so one might guess that the foundation has some other fundraising plans—perhaps approaching well-heeled individuals, other foundations, or companies to make up the difference. The interest and enthusiasm shown by the Kickstarter effort may be enough to shake loose some bigger donations.

The problem that the Freedom Box is seeking to solve is real, and recent events have only helped clarify that. We will have to wait and see whether the project and foundation are successful in solving it. Even if they fail, which is an outcome few would hope for, all of the work that is done will be available to others who want to head down that path. That is just another example of the freedom inherent in free software.

Comments (22 posted)

Python 3.2: toward the future of the language

The Python 3.2 release was announced on February 20, exactly 20 years after 0.9.0, which was the first public Python release. Given that Python 2.x remains the version of the language used by most programmers and most existing code, one might be tempted to write off this release as being relatively unimportant. But the 3.2 release has some changes which will be important to Python developers going forward, so, even if one isn't planning on moving to Python 3 right away, this release merits a quick look.

Since Python is under a moratorium on the addition of new language features, one might think that a new release - even a major release - would be relatively boring. But the moratorium only applies to the core language; the libraries - which is where much of the interesting action is to be found - are unaffected. A look at the What's new in Python 3.2 document indicates that the libraries are evolving quickly indeed. Some of the more significant changes include:

• A new "argparse" module for the handling of command-line options. Those of us still using getopt have been left far behind; the current "optparse" module has also been deprecated as of version 2.7. Argparse would appear to go beyond mundane argument parsing into the creation of command-line languages. It can probably handle more details than most people will ever want to use.

• There is an ongoing effort to gather concurrency-related modules under the "concurrent" namespace. The first addition there is concurrent.futures, a mechanism for the submission and management of tasks in multi-threaded and multi-process environments.

• The handling of compiled .pyc files has changed to reflect an environment where multiple Python runtimes coexist. They now have the interpreter name and version built into their names and have been banished into a separate __pycache__ directory. There is a similar mechanism for the handling of shared libraries.

• Many other modules have seen significant improvements; see the "what's new" document for details.

A couple of the most significant improvements may be elsewhere, though. One of those is the definition of a stable ABI for extension modules. Anybody who has been through a Python version update knows that the associated rebuilding of extension modules is not a lot of fun. As of version 3.2, modules which restrict themselves to a subset of the extension module ABI should continue to work indefinitely into the future. It's not yet clear how many real-world modules can live within the restrictions of this ABI; also unclear is how much that ABI could be extended without slowing further development of the language. But it's a step in the right direction toward the solution of a real problem.

Another partial solution to an ongoing problem can be found in the rewrite of the global interpreter lock (GIL). The GIL is Python's equivalent to the kernel's Big Kernel Lock; it ensures that only one thread can be executing in the bytecode interpreter at any given time. Since running bytecode is what Python programs do, the GIL can be seen as a rather significant constraint on how much concurrency is possible in a multi-threaded environment. Some extension modules release the GIL while they are doing extensive computations, and the GIL (like the BKL) is released while waiting for I/O, but that doesn't solve the real problem. The failure to remove (or at least reduce the role of) the GIL during the Python 3 development process is, for many developers, one of the biggest disappointments of Python 3.

The 3.2 GIL rewrite does not change the fundamental nature of the GIL, but it does reduce its impact somewhat. As described by Antoine Pitrou, the principal hacker behind this work, two significant changes have been made:

• Previously, the GIL would be passed from one contending thread to the next after a certain number of opcodes had been executed. But opcodes do not execute in constant time, and some of them (such as calls into an extension module) can execute for a long time indeed. The new GIL is, instead, passed on after a bounded time period (5ms by default).

• The GIL is implemented in an inherently unfair manner; once it has been released, any process which comes along can claim it. Prior to 3.2, that "any process" is often the process which just released the lock. That process is supposed to wait before attempting to reacquire the GIL, but the fact that it is running and cache-hot means it's still likely to get there first. The new GIL is still unfair, but it will at least force the releasing process to wait until a contending process has acquired the lock. That should fix some of the long latencies seen by Python programmers in some situations.

Given the scalability limitations inherent in a single, global lock, one might think that eliminating that lock would be a priority for the Python developers. The Python glossary suggests that this isn't the case:

The addition of fine-grained locking which did not hurt single-threaded code could certainly be a bit of work; it might well involve techniques like run-time patching of the interpreter. For a system which is supposed to run on many operating systems, such a solution could indeed be brittle and hard to maintain. In its absence, though, the scalability of multi-threaded Python programs will continue to be limited.

That said, Python 3 is clearly getting better. Over time, adoption appears to be on the increase; the number of distributions and modules which support the language is growing. Python 3 continues to be a sufficiently hard sell that a group of developers recently contemplated reopening feature-oriented development on version 2.x, but that idea fell by the wayside when it became clear that the developer interest wasn't there. Python 3 thus appears to be the future for those who want a language which continues to evolve. Based on what can be seen in the 3.2 release, that evolution is going full speed, even in the face of a moratorium on new core features.

Comments (31 posted)

Easy, powerful, stable: Pick two with OpenShot 1.3

OpenShot is a video editor for Linux that aspires to be simple, powerful, and "the very best open source video editor." OpenShot 1.3, which was released on February 13, brings it a little closer to that goal. This release brings a theme for the UI, support for adding multiple clips, new 3D animations, and a wizard for uploading video directly to YouTube or Vimeo. It may be the best open source video editor, but only if one is willing to overlook some stability issues.

Video editing is an area where Linux has lagged somewhat behind Windows and Mac OS X. This isn't to say that Linux users have had no options for editing video on Linux, but the selection of tools is not as broad, nor in many cases as full-featured or well-polished. Mac users have tools like Apple's iMovie that are very easy to use — though inflexible and decidedly unfriendly to open formats like Ogg Theora. Professional and advanced amateur users have quite a few options on Mac and Windows, depending on what they'd like to achieve and how much they're willing to spend.

Linux, on the other hand, has just a handful of viable alternatives. There's Cinelerra and its offshoot Cinelerra-CV, which are very capable editors, but also extremely complex and likely to intimidate most hobbyists. Kdenlive is another effort for providing a free software alternative for video editing on Linux (as well as FreeBSD and Mac OS X), that's much easier to use than Cinelerra. It might be a bit more intimidating than, say, iMovie, but it's usable by mere mortals.

Another editor that aims to be intuitive, but full-featured, is PiTiVi (which was reviewed here in June 2009). This is an LGPLed effort sponsored in part by Collabora and developed around the GStreamer framework. It is relatively easy to use, and is currently the default video editor for Ubuntu. The development for PiTiVi seems somewhat slowish, and the developers seem to be struggling to find contributors.

There's also Kino, which does (or did) a fair job of balancing features and functionality — but its development seems to have slowed to a crawl if not entirely stopped. The last release came out in September of 2009.

Those are just a few of the standouts. You'll find quite a few video editors for Linux in various states of completion and competence, but the landscape is littered with half-baked editors that are not entirely suitable for "prime time" when it comes to usability or ability to produce professional-quality videos.

OpenShot is a relative newcomer. Development is led by Jonathan Thomas, a software and Web developer who had his first taste of Ubuntu Linux in 2008 and found no video editors he felt were easy, powerful, and stable. So Thomas started with Python, the Media Lovin' Toolkit, and set off to try to realize a easy, powerful, and stable editor with OpenShot.

Easy, Powerful, but Stable?

The 1.3 release of OpenShot is available for Ubuntu in a Personal Package Archive (PPA), so I installed it and started practicing with a handful of pictures, a few short movies shot with my phone, and an MP3 to provide a background track. The project is also on the AV Linux LiveDVD, but the 1.3 release is not yet included with the live DVD. There's also an installer for Fedora 11 through 13, but it's not clear if the packages will work with Fedora 14. Naturally, source is also available.

For background, I don't claim any great skill in the area of non-linear video editing beyond having pieced together a number of videos from conference interviews using Kino in 2006 and 2007. Many years ago, I spent about a year working for a small television station (KTVO) in Kirksville, Missouri — which included linear video editing for broadcast news using antiquated (even at the time) 3/4" U-Matic tape.

OpenShot has a very simple interface. The left-hand corner holds a set of tabs for files, transitions, effects, and a history tab. The files tab holds all the clips (pictures, video, and audio) used to create videos. Effects are filters to modify audio, still images, or video — this can be used to give a video clip a sepia tone, for example, or apply an echo effect to audio. The transitions are, as you'd expect, a way to provide transitions from one piece of video to another. OpenShot has everything from simple dissolve and clock transitions to more elaborate star wipes or fractals.

On the right-hand side OpenShot has a video preview, and the bottom part of the OpenShot window holds the timeline that shows clips that have been integrated into the working video from the project files organized by track and file, and a small selection of tools for manipulating clips.

OpenShot is particularly easy to get started with. Drop a few video clips, pictures, and/or sound files into the project files tab and start dragging them into the order you'd like in the clips pane. For very simple projects involving just a few clips and music, it's possible to whip something together in ten minutes even if you've never used OpenShot before.

The OpenShot 1.3 release features a new theme for the interface and uses the stock desktop icons. I haven't used OpenShot prior to the 1.3 release, but looking at screenshots of older releases, it does look like the new theme is an improvement.

But there's more than just a facelift for the project. This release is supposed to feature improved stability over previous releases of OpenShot, as well as auto saving. Using OpenShot 1.3 from the PPAs on Ubuntu 10.10 on a system with 4GB of RAM (and a Core 2 Duo processor), OpenShot was still a bit fragile. When I first started experimenting with OpenShot it crashed because I tried to apply the Resize tool (apparently meant for still clips) to a video. It also crashed when applying an effect to a video, but worked just fine on restart, and crashed a couple of times when exporting movies — though the export finished successfully before OpenShot simply stopped responding. OpenShot 1.3 may be more stable than prior releases, but it's certainly not bulletproof.

One usability enhancement in the 1.3 release is a simplified export dialog. This is a really intuitive dialog when using the Simple tab, but its Advanced tab exposes just about any option that one might want when exporting a project. On the Simple tab, you have the option of Blu-Ray, DVD, Device, or Web. Blu-Ray and DVD have several options that are reasonable for those formats, while Device has presets for Xbox 360, Apple TV, and Nokia nHD. The Web preset features options like Wikipedia (Ogg Theora), FlickrHD, and a few options for Vimeo and YouTube. The Advanced tab provides just about any option that most users would want. Certainly more than enough for the bulk of home users looking to edit family vacation videos or funny pet videos. It may not offer every option that professionals may want, but it's certainly a good start.

The 1.3 release also simplifies organizing and finding files. Since I was only juggling about 10 files at a time, I didn't really find it hard to keep track — but this would be a useful feature for more ambitious projects.

OpenShot 1.3 also adds an "Add to Timeline" feature for pulling in multiple files. For example, you can select a handful of still images, and use the Add to Timeline feature to pop it into the timeline at the precise start time you'd like, as well as setting transitions and/or fades between the clips. The tool also gives the option of re-arranging the order of the files, or just shuffling them if you'd prefer a random order. This would be a very nice tool for creating a video out of family photos.

There's very little that I miss about working with linear video editing equipment at KTVO, but I do miss the physical controls for working with video. The shuttle (dial) for moving back and forth through video frame by frame gives a lot more control than trying to use the mouse and a slider. It is a bit surprising, perhaps, but the scroll wheel doesn't work for single-stepping either. Thankfully, OpenShot has keyboard shortcuts for frame stepping, pausing, etc., that allow just as much control while editing (though they lack the feel).

One thing that OpenShot has done very well, that I missed in Kino, is create titles. Whether you need credits at the start and end of a video, or overlays (like captions) over a portion of the video, OpenShot makes it very easy to do. If you have Blender installed, OpenShot will let you create animated 3D titles. Unfortunately, Ubuntu 10.10 ships with Blender 2.49, while OpenShot 1.3 expects 2.56. Herein lies one of the strengths and weaknesses of open source video packages that I've encountered over the years — many video editing packages build on readily available libraries or supporting packages (like Blender), but tend to be fussy about versions. Getting all of the dependencies right is quite a headache for users who want to use the most recent releases. Waiting for downstream projects means being several months behind the curve — and given the amount of catching up that open source video editors have to do, is also undesirable. A feature like uploading to YouTube — which is new in OpenShot 1.3 — is expected in a commercial package.

For anyone who's going to be at the upcoming Southern California Linux Expo, Thomas will be providing an in-depth look at OpenShot covering basic video editing to advanced effects. There's also a comprehensive guide for those new to video editing or just new to OpenShot. Developers interested in becoming involved should see the Launchpad project and mailing list.

OpenShot 1.3 represents significant progress on the open source video editing front. While it has some work to do in terms of stability, its feature set is certainly at the "good enough" point for many users. OpenShot is worth a serious look by anyone who's interested in doing video editing on Linux.

Comments (38 posted)

New LWN feature: HTTPS-only access

We would like to announce a new LWN feature that readers may find useful: forcing HTTPS (i.e. SSL/TLS) connections for every page in the site. Enabling the feature will help prevent man-in-the-middle eavesdropping and session hijacking. If you want to give it a try, go to the "My Account" page, then to "Customize your account", and the "Force SSL" option is the first listed. You will have to log out and log back in for it take effect and, because of Google ads, you may get a browser popup complaining about insecure content the first time you access the site. If you run into any problems with the new feature, please let us know at lwn@lwn.net.

Comments (62 posted)

Page editor: Jonathan Corbet

Inside this week's LWN.net Weekly Edition

• Security: CentOS 5, RHEL 5.6, and security updates; New vulnerabilities in awstats, Java, telepathy-gabble, webkitgtk,...
• Kernel: FIEMAP and delayed allocation; Notes from the block layer; debugfs: rules not welcome; Optimizing Linux with cheap flash drives.
• Distributions: Btrfs by default in Fedora 16?; RHEL 4.9, Ubuntu 10.04.2, SQLninja in Fedora, ...
• Development: Rethinking interactive fiction games with Curveship; GNOME shell, libchamplain, mixxx, ...
• Announcements: How to root a Nook Color, Building the Technology Stack, The Cognitive Style of Unix, ...

Next page: Security>>