User: Password:
|
|
Subscribe / Log in / New account

Leading items

Happy Holidays from LWN

Welcome to the final LWN.net Weekly Edition for 2011. This issue contains some of our traditional backward-looking content, including the final installment on the 2011 timeline and a review of our predictions from January. Readers will also find our usual mix of kernel content, a look at openSUSE's struggles with its systemd transition, the GNOME accessibility challenge, and even a look at an alternative search engine.

As is also traditional, we will be taking the final week of the year off from our usual publication schedule, so the next Weekly Edition will come out on January 5, 2012. There will be occasional daily updates during the break, but the news is usually slow in coming during that time, so updates will not come at a great pace either. We wish all of our readers a happy and restful holiday period, and we look forward to seeing you all again in 2012. Thanks for supporting LWN through another great year!

Comments (6 posted)

A look back at 2011

By Jonathan Corbet
December 21, 2011
It is that time of year again: your editor, having, as usual, delayed engaging with that whole "holiday shopping" thing until the last minute, can be counted on to be rather more grumpy than usual. Clearly what is needed is some comic relief, and there are few things more comic than a critical look back at the predictions made at the beginning of the year. As usual, some of those predictions worked out, while others proved to be badly wrong indeed; still others should have been made but were not.

Things got off to a reasonably good start (prediction wise) with the assertion that the LibreOffice project would take off, while OpenOffice would languish. LibreOffice has, indeed, been successful in attracting developers, building enthusiasm, and getting the releases out; the project's fund-raising drive early in the year was highly successful. Distributors are picking it up almost universally; it is clearly a project that will be around for the long haul.

What your editor didn't foresee was that Oracle would simply give up on OpenOffice.org and cast it off to "the community." The new project has struggled to come to terms with "the Apache way," review the licensing of all the code (eliminating non-Apache-compatible code along the way), figure out its mailing lists and web sites, and set up a working governance model. There have been no OpenOffice.org releases since 3.3.0 came out in January, 2011. This project hopes to start making releases again in early 2012; how many people will care remains to be seen.

The thought that Mageia and IllumOS would do less well than they would have liked seems to have been mostly correct. Mageia did manage to get a release out, and it does have a dedicated core of developers, but things are moving slowly and adoption appears to be small. The Mageia developers continue their work, though, and a second release is in alpha test as of this writing. Meanwhile, traffic on the IllumOS lists has dwindled. IllumOS has developed some commercial life in the form of SmartOS, which includes a port of the KVM virtualization subsystem - your editor did not see that one coming. There is no real way to tell how well SmartOS is doing at this point.

The predictions confidently claimed that MeeGo would be a surprisingly big success in 2011, which would meanwhile be an iffy year for WebOS. The WebOS prediction was just about right, clearly showing that your editor's crystal ball is still in good working order; there's no need to talk about that other prediction at all. It was indeed a "make or break" year for WebOS, with a heavy emphasis on the "break" part, though the decision to open-source it may yet give WebOS another life. So let's just think about WebOS and pay no attention to that MeeGo behind the curtain...

Oh, OK, might as well rub it in. Perhaps it's true that your editor is dense enough to have been the only one to not see the "Elopcalypse" coming; once Nokia decided to go with Microsoft, any possibility of MeeGo continuing as a shared project came to an end. In truth, the seeds of MeeGo's demise may have been sown long before; Intel and Nokia seemed to have widely differing views on where that project should go. It is a shame; your editor still believes that MeeGo was a project with the potential to do great things. But that story appears to be at an end; "Tizen" may yet surprise us, but it would be a big surprise indeed.

Did Google become a "major kernel contributor" as predicted in January? Since the release of 2.6.37 on January 4, Google has contributed 789 changes to the kernel - 1.6% of the total. That makes it the 13th biggest contributor of changes, ahead of companies like AMD, Microsoft and Oracle, but behind Nokia and Samsung. The numbers for 2010 (technically, 2.6.32-2.6.37, so just over one year) were 489 changes, 1.0% of the total. So Google has indeed increased its contributions, but your editor would like to believe that there is a lot more to come.

ChromeOS was predicted to struggle in 2011. Some "Chromebooks" have found their way to the market, but ChromeOS has not, yet, taken the computing world by storm.

Your editor predicted huge legal battles - a fairly easy prediction to make. Even so, your editor cannot claim to have foreseen just how bad the mobile patent wars would get. The thought that we might see a Stuxnet-like attack against Linux systems hasn't become reality - that we know about, anyway - even though the Linux community did endure some severe security-related problems this year. Alas, the hopeful thought that we would see a free driver for an embedded graphics chipset proved to be too hopeful; the slowly-improving gma500 driver in the staging tree doesn't quite count.

What about the prediction that the tension between providing stable code and providing leading-edge code would increase? That one is hard to judge. The big fights within Fedora that inspired that prediction would appear to have simmered down without slowing Fedora's tendency to ship very new stuff. If one reinterprets the prediction as applying to the tension between "the way we've always done it" and new subsystems embodying new ideas, then the prediction certainly held true in 2011. Yes, that must certainly have been what your editor was trying to say.

January's predictions finished out with a couple of ideas, the first being that openSUSE would adopt ultra-stable and leading-edge variants. On the stable side, the "Evergreen" project seems to be getting off to a slow start. The rolling "Tumbleweed" distribution, instead, has been active for some time and seems to have a small core of users. The final prediction was that business models depending on control over the code - things like "open core" and those based on copyright assignments - would fade away. It's not really clear that this has happened, but one can at least say that copyright assignment policies do not have the best reputation at the moment.

So what did the January predictions miss entirely? One obvious candidate is the GNOME 3.0 release and the firestorm of criticism that followed it. At the end of the year, it would appear that the worst of that storm has passed; the 3.2 release has earned a better reception than its predecessor. Hopefully the GNOME project will be able to continue to woo back the users it has lost while gaining the large numbers of new users they hope for.

Predicting continued success for Android would have been an easy home run. Even so, it would have been hard to imagine a world where a half-million 700,000 Android devices are activated every day. Given the sheer size of this success, it is not surprising that the lawyers are circling around Google.

Your editor predicted the demise of the big kernel lock in 2010 - just a bit ahead of his time, as usual. That prediction was not repeated this year, which was a mistake: the actual demise of the BKL came with 2.6.39 in 2011 - not a moment too soon.

All told, it was a year with a lot of big ups and downs. Some things went poorly, to the point that some commentators have written the whole year off as a bad one. But one need not look too hard to realize that the free software community got a lot done in 2011, that it is as strong and vibrant as ever, and that we are poised to push even further in 2012. Legal hassles, failing projects, and clueless companies are nothing new. We have dealt with them before; there will be more of them to deal with in the future. None of these challenges have really slowed us down thus far; there is every reason to believe we will be equally successful in the future.

Comments (1 posted)

DuckDuckGo: A privacy-conscious search engine

December 21, 2011

This article was contributed by Bruce Byfield

The recent release of Linux Mint 12 surprised many by using the little-known DuckDuckGo (DDG) as its default search engine. Through a confidential agreement, Linux Mint will "share the revenue generated by the sponsored links" when users click on them using DDG. That arrangement is a creative way to help fund the distribution. On the other side, though, readers may wonder why, when Google, Yahoo!, and Bing so thoroughly dominate search engines, is DDG developing another one?

DDG founder Gabriel Weinberg has a ready answer:

"[There are] things we think the bigger search engines don't do well for a variety of reasons (generally not technical, but legal, business, and cultural). These areas of focus have been: way more instant answers, way less spam, real privacy, and a less cluttered user interface.

Add an emphasis on using free software and being a good citizen within the community, and perhaps DDG has a chance to prosper, even though its 12 million searches per month are next to nothing compared to the December 2009 figures for some of the larger players: 88 billion for Google and 9.4 billion for Yahoo!.

An MIT graduate with a master's degree from the Technology and Policy Program, Weinberg is a small-time angel investor with a strong interest in companies powered by open source technologies. Four years ago, he founded DDG, which has grown slowly to three employees, a number of part-time-contributors, and what he calls the "growing open source wing" of about twenty collaborators.

Anticipating market directions

According to one of his blog entries in which he discusses his various investment strategies, Weinberg's approach with DDG is "to work within a big market and concentrate on where you think it is headed." Part of what this approach means is that DDG tries to improve the results returned on a search. Lacking the resources to do all its own web crawling, DDG relies on fifty other sites for its results, including Yahoo! and Bing, as well as more specialized sites. To a limited extent, users can control which sites are used through the combo box to the right of the search field, choosing, for example, to use Bing or Google for image searches. Users can also choose whether to order results by date or alphabetical order. Weinberg continued:

We believe that for any given search, there is usually a vertical search engine out there (or API or data set) that does a better job at answering that query than a general search engine. Our long term goal is to get you information from that best source, ideally in instant answer form.

From the initial results, DDG filters ad-heavy portal sites and presents results without bubbling — that is, ordering results in light of your previous searches. In fact, DDG claims that, unlike larger search engines, it doesn't collect information about user's searches at all. Instead, it attempts to order results by crowdsourcing, just as YaCy, another new and small search engine does.

In fact, DDG makes some efforts to protect user privacy and to educate users about why they should care about privacy. Although no details are given, DDG claims to redirect a search request "in such a way so that it does not send your search terms to other sites. The other sites will still know that you visited them, but they will not know what search you entered beforehand." In addition, DDG uses an HTTPS version of a site when one is available.

Weinberg described these features as offering users "real choice," adding:

Previously, there haven't been many real choices when it comes to protecting privacy online. You could either disengage completely (not a great option) or decide to give up significant privacy (another not great option). We've taken it over as part of our mission to both a) help educate people on issues and b) give people real control over their privacy, and thus a real choice when it comes to search privacy.

Searches support the syntax users may know from other search engines, such as the use of quotations mark to search for an exact phrase, or a minus sign to exclude results that contain a specific word or phrase. Users can also filter results by toggling "safe search" or "meanings" (provides choices for ambiguous terms) settings, or using the region setting to filter results for increased relevance.

By default, search results are topped with a red box, the so-called "zero click" or "instant answer" feature which tries to place the most important result first. When searching for a concept, the red box result might be a link to the basic definition; for a person, to their blog or web page. To the right of the red box, a list of suggestions for refining the search appears. Should the query have more than one meaning, suggestions similar to the disambiguation pages on Wikipedia are given.

Another feature of DDG is !bang searches: automatic searches, somewhat similar to Google's "site:" searches, which are available for common sites like Amazon or YouTube. For instance, if you enter "!youtube pogues," DDG shows you the results on YouTube for The Pogues, saving you several additional clicks. Similarly, you could specify !openstreetmap at the start of your search to get results from OpenStreetMap or !monster to search Monster for a job description. These !bang searches include a wide variety of different categories, such as major corporations, domains, programming languages, shopping sites, tech domains, research topics, news, and online services. The only problem is that, if you haven't memorized a supported !bang search, you either have to take a chance that your site of choice is supported, or else look it up on the DDG site.

DDG also includes the ability to do calculations, measurement and currency conversions, and to answer direct questions on weather, food, geography, and time-related topics such as tides or sunrises in a specific location. Also included on DDG is a small number of apps, as well as popup how-tos about adding DDG to common browsers.

Many of these features, such as the ability to do calculations, are paralleled in major search engines. Others are unique to DDG. However, what stands out is not any particular feature so much as the total number of them. For some reason, DDG lacks a summary of total results, which is often used as a rough indication of a topic's importance (and for ego surfing), but, otherwise, the main drawback is that taking advantage of DDG's features requires a willingness to learn — a willingness that might be lacking in many who are only mildly curious about such a niche service.

Interacting with the open source community

DDG has always relied on free software, as it is written in Perl and JavaScript, runs on FreeBSD, and uses nginx for a web server. However, according to Weinberg, DDG "started out as closed source and is slowly becoming more and more open source. We've been making much of our new pieces completely open source."

These new pieces include the upcoming community platform, which Weinberg describes as a collection of "tools for communities to use to help participate." So far, the tools include a translation interface and a server for Extensible Messaging and Presence Protocol (XMPP), with "a data store to store settings and URL shortener" expected to follow. DDG is also encouraging contributions to expand and improve DDG's zero-click info repos, the source of the material in the red boxes at the top of search results. "As these contributions increase, the percentage of DDG that is open source is going up. I'm not sure about becoming completely open source for a variety of reasons (spam paramount among them), but we are certainly thinking about [the possibility]," Weinberg said.

Even if DDG does not become completely free, Weinberg emphasized his determination to use free software "as much as possible." In particular, referring to DDG's multiple sources for results, he suggests that the use of free software is directly related to the availability of expert results: "If you know an esoteric piece of the query space really well, you should be able to develop for it." In other words, using free software not only produces more specialized sources, but also indirectly increases the accuracy and usefulness of DDG's results.

Similarly, asked to comment on Katherine Noyes's suggestion that taking results from Microsoft's Bing might deliver results with an anti-open source bias to Linux Mint users, Weinberg pointed out that Bing is only one of over fifty sources. "I haven't seen any compelling evidence that we're biased against open source," Weinberg said. "And in fact, we're working on ways to essentially do the opposite." For example, DDG already uses Ask Ubuntu as a source for technical results, and is currently working on tighter integration with alternativeTo in order to increase the accuracy of free-software related queries.

In return for bootstrapping off free software, Weinberg said, he would "very much like to help start a movement where companies that use open source give back in systematic ways to those communities." As a preliminary effort, he has established Foss tithe, a site on which corporate owners can pledge to donate a percentage of their net income to the community. The suggested tithe is ten percent.

So far, only one other company (search [co.de]) has pledged to tithe, and Weinberg himself has not done much to develop the idea. However, he has made his own tithe, with half the donation decided upon by him and half by the DDG community. In 2010, for the corporate portion, he chose to give $482 to nginx and $475 to FreeBSD, two projects that he described as "an integral part of our architecture." Choosing security and privacy as a donation theme, the DDG community chose to donate $238 to each of Tor, Clamwin, Tahoe-LAFS, and OpenSSH.

Whether DDG will ever be a major contender among search engines is doubtful. A buyout by a larger competitor is an obvious possibility, though it is unclear whether DDG's privacy policies and options would survive such an event. However, by seeking out closer ties with free and open source software, DuckDuckGo might just find itself the search engine of choice among a small, dedicated group of users with enough knowledge to appreciate its philosophy and features. That could be a path to success and financial sustainability for a smaller search engine like DDG.

Comments (6 posted)

2011 Linux and free software timeline - Q4

Here is LWN's fourteenth annual timeline of significant events in the Linux and free software world for the year.

We broke the timeline up into quarters, and this is our report on the final quarter, October-December 2011, though there may be an addition or two for December. The previous quarters can be found as follows:


This is version 0.8 of the 2011 timeline. There are almost certainly some errors or omissions; if you find any, please send them to timeline@lwn.net.

LWN subscribers have paid for the development of this timeline, along with previous timelines and the weekly editions. If you like what you see here, or elsewhere on the site, please consider subscribing to LWN.

For those with a nostalgic bent, our timeline index page has links to the previous thirteen timelines and some other retrospective articles going all the way back to 1998.

October

Red Hat acquires Gluster, the makers of the open source GlusterFS (press release).

XML - the kudzu of the internet.

-- Valdis Kletnieks

A rootkit that is alleged to be used for surveillance by the German government is analyzed by the Chaos Computer Club (CCC report, LWN blurb).

WineHQ database is compromised leading to the exposure of users' encrypted passwords (announcement, LWN blurb).

openSUSE announces the first release of its openQA distribution testing tool (announcement, LWN article). [ownCloud logo]

ownCloud 2 is released; ownCloud is a free cloud storage and synchronization web application (announcement).

So you need another heuristic to handle that, and of course "heuristic" is an ancient african word meaning "maybe bonghits will make this problem more tractable".

-- Matthew Garrett

Plasma Active One, the KDE-based interface for touchscreen devices, is released (announcement, LWN article).

Samba changes its longstanding policy on corporate-copyrighted code, which relaxes the requirement for personally copyrighted code (announcement, LWN look at the discussion from July). [Subversion logo]

Subversion 1.7.0 is released (announcement, release notes).

The time zone database is briefly shut down due to copyright complaints from an astrology company (LWN blurb and article).

KDE celebrates its 15th anniversary (reflections from Cornelius Schumacher, LWN article).

For a while people were promoting the idea that its good to be lenient in what you accept as input and strict in what you send out. I think people are starting to realize that was a horrid mistake since now they're getting utter crap and people don't even know what right is anymore.

-- Peter Zijlstra

Ubuntu 11.10 ("Oneiric Ocelot") is released (announcement, release notes).

Dennis Ritchie, of Unix and C fame, passes away (LWN blurb, Rob Pike's Google+ "obituary").

Linux 3.1 is released (announcement, KernelNewbies summary, A look at the 3.1 development cycle).

The 13th Realtime Linux Workshop is held in Prague, Czech Republic, October 20-22 (Realtime minisummit coverage, Proceedings). [Kernel summit]

The 2011 Kernel summit is held October 23-25 in Prague (LWN coverage).

Debian is pretty bad at making choices. Almost always, when faced with a need to choose between alternative solutions for the same problem, we choose all of them. For example, we support pretty much every init implementation, various implementations of /bin/sh, and we even have at least three entirely different kernels.

-- Lars Wirzenius

Lisp creator John McCarthy passes away at 84 (TechCrunch obituary).

The second GStreamer conference is held in Prague, October 24-25 (LWN coverage: GStreamer 1.0 and 0.10 and Xiph.org).

LinuxCon Europe is held in Prague, October 26-28 (LWN coverage: Kernel panel, UMMS, an audio/video abstraction layer and A btrfs update).

The Embedded Linux Conference Europe is held in Prague, October 26-28 (LWN coverage: Till Jaeger on AVM vs. Cybits, The embedded long-term support initiative, and Sandboxing for automotive Linux; Conference videos).

November

OpenBSD 5.0 is released (release notes).

It's important not to show a smug expression on your face while printing if users of non-Linux OSs are still dealing with driver CDs or vendor downloads.

-- Don Marti

The Trinity Desktop Environment releases 3.5.13 as a continuation of the KDE 3.5 series (announcement, LWN review).

Samba notes its first contribution from Microsoft employees, which actually happened back in October (announcement). [Fedora logo]

Fedora 16 is released (announcement, release notes).

Google announces the availability of the source code for Android 4.0 ("Ice Cream Sandwich"), after withholding the source to 3.x (announcement, LWN article).

They went out of their way to let researchers in, and now they're kicking me out for doing research. I didn't have to report this bug. Some bad guy could have found it instead and developed real malware.

-- Charlie Miller gets banned from Apple's developer program

openSUSE 12.1 is released (announcement, release notes).

AVM loses its case to restrict anyone from modifying the GPL-covered code in its routers (gpl-violations.org announcement).

Barnes & Noble decries Microsoft's "trivial" patents used to fight Android (LWN blurb, Groklaw article).

[ColorHug logo] Richard Hughes announces the ColorHug open hardware/software colorimeter (announcement, LWN blurb).

A serious denial of service attack against BIND 9 is seen in the wild (ISC advisory).

I admire and respect the fact that you can make free software do exactly what you want - that's precisely what I set out to support in founding Ubuntu. What I did not set out to found was a project which pandered to the needs of a few, at the cost to the many. Especially when the few can perfectly well help themselves, and the many cannot.

-- Mark Shuttleworth

Lennart Poettering and Kay Sievers unveil "the Journal" as an alternative to standard Linux unstructured logging; the announcement is not met with widespread acclaim (announcement, LWN article).

YaCy, a peer-to-peer search engine, makes its 1.0 release (LWN article).

Linux Mint 12 is released (announcement, LWN review).

Cinepaint is resurrected and releases version 1.0 though it's rather unclear where the GIMP fork with support for 16 and 32 bits per channel will go from here (Libre Graphics World report).

December

Download.com is found to be bundling Nmap with adware/spyware for Windows users of the security scanner (announcement, update page).

Disclosing security vulnerabilities is good for security and good for society, but vendors really hate it. It results in bad press, forces them to spend money fixing vulnerabilities, and comes out of nowhere.

-- Bruce Schneier

extensions.gnome.org launches as a site for GNOME Shell extensions (announcement). [LLVM logo]

The LLVM compiler suite releases version 3.0 (announcement).

The QEMU system emulator releases version 1.0 (announcement).

[webOS logo] HP announces that it will contribute the webOS code to the open source community (announcement, LWN article).

Ugh - and people continue to get exploited from a preventable, fixable and already fixed VFS design flaw.

-- Ingo Molnar on the continued existence of symlink races

Facebook releases the HipHop virtual machine for faster PHP execution as open source (announcement). [Plasma Active logo]

KDE announces the release of Plasma Active Two, the second iteration of its interface for touchscreen devices (announcement).

Rockbox 3.10 is released on the tenth anniversary of the music player alternative firmware project (announcement).

Note that only a witless moron could ever actually be confused (rather than simply annoyed) by "1 files". Unfortunately, we actually deal with these witless morons on a daily basis: they're called computers. And as it happens, they're actually much more likely to be confused by the difference between "1 file" and "2 files", especially if we were to switch to using the latter 6 years in.

-- Matt Mackall

BT sues Google for patent infringement in Google Music and the Android Market (LWN blurb).

CentOS 6.2 is released, right on the heels of RHEL 6.2 (announcement, release notes).

The Android mainlining project is announced; progress is being made (announcement, LWN article).

Qt 4.8.0 is released (announcement). [Mozilla logo]

Google and Mozilla agree to financial terms for Google to continue as the default Firefox search engine (announcement).

Comments (none posted)

Page editor: Jonathan Corbet
Next page: Security>>


Copyright © 2011, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds