DuckDuckGo: A privacy-conscious search engine

The recent release of Linux Mint 12 surprised many by using the little-known DuckDuckGo (DDG) as its default search engine. Through a confidential agreement, Linux Mint will "share the revenue generated by the sponsored links" when users click on them using DDG. That arrangement is a creative way to help fund the distribution. On the other side, though, readers may wonder why, when Google, Yahoo!, and Bing so thoroughly dominate search engines, is DDG developing another one?

DDG founder Gabriel Weinberg has a ready answer:

Add an emphasis on using free software and being a good citizen within the community, and perhaps DDG has a chance to prosper, even though its 12 million searches per month are next to nothing compared to the December 2009 figures for some of the larger players: 88 billion for Google and 9.4 billion for Yahoo!.

An MIT graduate with a master's degree from the Technology and Policy Program, Weinberg is a small-time angel investor with a strong interest in companies powered by open source technologies. Four years ago, he founded DDG, which has grown slowly to three employees, a number of part-time-contributors, and what he calls the "growing open source wing" of about twenty collaborators.

Anticipating market directions

According to one of his blog entries in which he discusses his various investment strategies, Weinberg's approach with DDG is "to work within a big market and concentrate on where you think it is headed." Part of what this approach means is that DDG tries to improve the results returned on a search. Lacking the resources to do all its own web crawling, DDG relies on fifty other sites for its results, including Yahoo! and Bing, as well as more specialized sites. To a limited extent, users can control which sites are used through the combo box to the right of the search field, choosing, for example, to use Bing or Google for image searches. Users can also choose whether to order results by date or alphabetical order. Weinberg continued:

From the initial results, DDG filters ad-heavy portal sites and presents results without bubbling — that is, ordering results in light of your previous searches. In fact, DDG claims that, unlike larger search engines, it doesn't collect information about user's searches at all. Instead, it attempts to order results by crowdsourcing, just as YaCy, another new and small search engine does.

In fact, DDG makes some efforts to protect user privacy and to educate users about why they should care about privacy. Although no details are given, DDG claims to redirect a search request "in such a way so that it does not send your search terms to other sites. The other sites will still know that you visited them, but they will not know what search you entered beforehand." In addition, DDG uses an HTTPS version of a site when one is available.

Weinberg described these features as offering users "real choice," adding:

Searches support the syntax users may know from other search engines, such as the use of quotations mark to search for an exact phrase, or a minus sign to exclude results that contain a specific word or phrase. Users can also filter results by toggling "safe search" or "meanings" (provides choices for ambiguous terms) settings, or using the region setting to filter results for increased relevance.

By default, search results are topped with a red box, the so-called "zero click" or "instant answer" feature which tries to place the most important result first. When searching for a concept, the red box result might be a link to the basic definition; for a person, to their blog or web page. To the right of the red box, a list of suggestions for refining the search appears. Should the query have more than one meaning, suggestions similar to the disambiguation pages on Wikipedia are given.

Another feature of DDG is !bang searches: automatic searches, somewhat similar to Google's "site:" searches, which are available for common sites like Amazon or YouTube. For instance, if you enter "!youtube pogues," DDG shows you the results on YouTube for The Pogues, saving you several additional clicks. Similarly, you could specify !openstreetmap at the start of your search to get results from OpenStreetMap or !monster to search Monster for a job description. These !bang searches include a wide variety of different categories, such as major corporations, domains, programming languages, shopping sites, tech domains, research topics, news, and online services. The only problem is that, if you haven't memorized a supported !bang search, you either have to take a chance that your site of choice is supported, or else look it up on the DDG site.

DDG also includes the ability to do calculations, measurement and currency conversions, and to answer direct questions on weather, food, geography, and time-related topics such as tides or sunrises in a specific location. Also included on DDG is a small number of apps, as well as popup how-tos about adding DDG to common browsers.

Many of these features, such as the ability to do calculations, are paralleled in major search engines. Others are unique to DDG. However, what stands out is not any particular feature so much as the total number of them. For some reason, DDG lacks a summary of total results, which is often used as a rough indication of a topic's importance (and for ego surfing), but, otherwise, the main drawback is that taking advantage of DDG's features requires a willingness to learn — a willingness that might be lacking in many who are only mildly curious about such a niche service.

Interacting with the open source community

DDG has always relied on free software, as it is written in Perl and JavaScript, runs on FreeBSD, and uses nginx for a web server. However, according to Weinberg, DDG "started out as closed source and is slowly becoming more and more open source. We've been making much of our new pieces completely open source."

These new pieces include the upcoming community platform, which Weinberg describes as a collection of "tools for communities to use to help participate." So far, the tools include a translation interface and a server for Extensible Messaging and Presence Protocol (XMPP), with "a data store to store settings and URL shortener" expected to follow. DDG is also encouraging contributions to expand and improve DDG's zero-click info repos, the source of the material in the red boxes at the top of search results. "As these contributions increase, the percentage of DDG that is open source is going up. I'm not sure about becoming completely open source for a variety of reasons (spam paramount among them), but we are certainly thinking about [the possibility]," Weinberg said.

Even if DDG does not become completely free, Weinberg emphasized his determination to use free software "as much as possible." In particular, referring to DDG's multiple sources for results, he suggests that the use of free software is directly related to the availability of expert results: "If you know an esoteric piece of the query space really well, you should be able to develop for it." In other words, using free software not only produces more specialized sources, but also indirectly increases the accuracy and usefulness of DDG's results.

Similarly, asked to comment on Katherine Noyes's suggestion that taking results from Microsoft's Bing might deliver results with an anti-open source bias to Linux Mint users, Weinberg pointed out that Bing is only one of over fifty sources. "I haven't seen any compelling evidence that we're biased against open source," Weinberg said. "And in fact, we're working on ways to essentially do the opposite." For example, DDG already uses Ask Ubuntu as a source for technical results, and is currently working on tighter integration with alternativeTo in order to increase the accuracy of free-software related queries.

In return for bootstrapping off free software, Weinberg said, he would "very much like to help start a movement where companies that use open source give back in systematic ways to those communities." As a preliminary effort, he has established Foss tithe, a site on which corporate owners can pledge to donate a percentage of their net income to the community. The suggested tithe is ten percent.

So far, only one other company (search [co.de]) has pledged to tithe, and Weinberg himself has not done much to develop the idea. However, he has made his own tithe, with half the donation decided upon by him and half by the DDG community. In 2010, for the corporate portion, he chose to give $482 to nginx and $475 to FreeBSD, two projects that he described as "an integral part of our architecture." Choosing security and privacy as a donation theme, the DDG community chose to donate $238 to each of Tor, Clamwin, Tahoe-LAFS, and OpenSSH.

Whether DDG will ever be a major contender among search engines is doubtful. A buyout by a larger competitor is an obvious possibility, though it is unclear whether DDG's privacy policies and options would survive such an event. However, by seeking out closer ties with free and open source software, DuckDuckGo might just find itself the search engine of choice among a small, dedicated group of users with enough knowledge to appreciate its philosophy and features. That could be a path to success and financial sustainability for a smaller search engine like DDG.

---

(Log in to post comments)