User: Password:
Subscribe / Log in / New account Weekly Edition for July 17, 2014

On satellite and aerial imagery for free software

By Nathan Willis
July 16, 2014

In our recent look at running a Google-free Android system, it was noted that the free-software mapping and routing application OsmAnd lacks support for using satellite photos as a background layer. OsmAnd is not unique in this limitation, of course; Navit lacks it as well, and it is an oft-cited issue in in-vehicle infotainment (IVI) discussions. Even the OpenStreetMap (OSM) community unsuccessfully attempted to build an open-source companion project using aerial imagery, eventually finding the task impractical. But the free-software community may be able to work around this obstacle by using other flavors of map data.

To recap, members of the OSM community launched an "OpenAerialMap" project in 2007, which ultimately ended up shutting down in 2008. The plan had been to merge several sources of freely licensed orthographic imagery, including satellite images and aerial photographs—the latter of which, in general, are the more useful option, since they are far less expensive (per-pixel) at any meaningfully high resolution. Aerial photographs are frequently available due to government land surveys and other research; airplanes can not only get better resolution than satellites, they are also better at collecting cloud-free images by flying lower.

The root cause of OpenAerialMap's shutdown was that raster images are simply far more difficult to work with on a global-scale map than are the vector-based maps that form the backbone of OSM. Many hundreds of terabytes of storage would be required to hold the images (images that compress far less than vector data even in those unpopulated regions with few roads), every image requires processing and alignment, and tremendous bandwidth would be needed to send it out to users as a map layer. Some day, perhaps, increased computing power and cheap connectivity may change all of the equations, but it was simply not a viable project.

Furthermore, even at the time, there were other freely-available (albeit not freely-licensed) aerial data options available for use with OSM: Yahoo agreed to let the project use its aerial-photo database as a background layer in the OSM map editor. The arrangement between Yahoo and OSM was discontinued in 2011—but, in its absence, OSM has an even better relationship with MapQuest. The company makes a "MapQuest Open Aerial Tiles" layer available for free to any application that credits MapQuest as the source and whose developers contact MapQuest if they think usage will exceed 4,000 tiles per second.

The MapQuest data is culled from US government sources, including NASA and the Department of Agriculture's (USDA) Farm Service Agency, so it would likely be the same as the images available to a project like OpenAerialMap. In addition, the USDA imagery is available directly from the US government, which has its own policy for attribution, thus making its terms of usage quite similar to MapQuest's.

Finally, although it might be wise from a software-freedom (and redundancy) standpoint for there to be more than one source for such data, as a practical matter, each end user tends to use aerial map layers in a transient fashion only. That is, the tiles directly under the current map or route are of interest, but caching them long term or storing them permanently does not add anything. Combine these factors together, then weigh them against the cost of running an aerial-data web service, and MapQuest Open Aerial Tiles have effectively nixed the desire for the community to run a competing service.

Alternative alternatives

On the other hand, it can also be instructive to consider what purpose an aerial photo layer serves in a mapping application. It is purely for the benefit of the user; all computation is performed on the vector layer (including finding routes and placing points of interest). To the user, an aerial data layer can assist in several ways. It can help the user better visualize the surrounding area and spot inconsistencies with the vector map, such as a road that leads directly into a lake. It can also provide a more realistic visualization of what lies ahead—showing buildings, hills and valleys, hazardous-looking sinkholes, and so forth.

The importance of the map-inconsistency case depends on how good the map is. OSM and other free-software mapping projects are very much of the "best effort" variety and, in theory, an inconsistency between OSM data and the real world will get spotted and corrected by OSM contributors. Thus, an aerial photo layer would be good, but it would also be a hedge against problems, rather than directly providing a necessary function.


For the improved-realism case, though, OSM is proceeding on two separate fronts. The first is to create a more realistic rendering of the map terrain by using topographic relief data. The largest project here is TopOSM, which is based on bulk data-set imports from government agencies (many of which are the same agencies that provide aerial photography data).

As of right now, TopOSM is limited in coverage to the United States, but on the plus side, the TopOSM layer can render its elevation data in shaded relief that presents a quasi-3D view of the terrain. For simply getting a feel for what the surrounding countryside should look like, this makes it a fairly good substitute for an overhead photo. There are several other OSM-based efforts that focus on other regions, such as OpenTopoMap in Germany and Alpenkarte for the Alps.

The other front is visualizing buildings and other man-made structures. For this, OSM already supports the addition of 3D models. This, like all OSM sub-projects, is volunteer-driven, so 3D building coverage varies considerably from one place to another. Coverage in Germany is extensive; there is even a separate OSM-3D project (maintained by the University of Heidelberg) that focuses on providing a usable 3D map interface.


How useful a 3D model is for visualization also depends on its level of detail; on that front there is yet another OSM sub-project called—perhaps confusingly—OSM-4D that seeks to apply realistic textures to the 3D building and item objects in the OSM database.

Although it is not quite the same issue as aerial photography, the "Street View" function in Google Maps could be considered a related feature, since it, too, helps the user visualize the real world depicted in the map. Here again, OSM has a crowd-sourced project in the works that aims to fill the same niche. It looks to be in its early days, but there are also other ways to find geotagged photos associated with a location on the OSM map. Naturally, coverage in such a crowd-sourced project will be best near high-traffic spots like tourist sights.

There is always a chance that some as-yet-unforeseen development will suddenly make an OSM-like open database of aerial and satellite imagery a viable project. In that case, history suggests that the OSM community will not be slow to jump on the opportunity. In the meantime, though, it is interesting to see how the community has routed around the unavailability of such a free database.

Based on exploring TopOSM and OSM-3D in the web browser, both projects certainly go a long way toward meeting the needs of the user who is stuck without a satellite imagery option. Perhaps the more pertinent question is why the various mapping application projects have been slower to add support for these alternatives. It may be that the incompleteness of the topographic and 3D data sets makes for a less-than-inspiring map experience for many users. Although, if that is the case, nothing seems to motivate an OSM contributor quite like seeing a blank spot on the map.

Comments (6 posted)

Steam comes to Linux

July 16, 2014

This article was contributed by Adam Saunders

Linux has frequently been viewed as a second-rate desktop gaming platform. While Linux has a number of compelling open-source games, including the 3D kart racer, SuperTuxKart, and the turn-based strategy game, Battle For Wesnoth, the number of quality native Linux games with high production values has been small for a long time. With the relatively low install base of Linux, and its fragmentation into multiple distributions, one could understand why game developers not otherwise familiar with desktop Linux would be hesitant to develop for the platform. But that has changed with the advent of Steam for Linux.

A bit of history

Things started getting better in May 2010, when the first Humble Indie Bundle was unleashed. Gamers could pay what they wanted (including setting aside any portion of that money they'd like for charity) to download a group of DRM-free copies of high-quality video games made by independent developers or development teams. These games, and most games in the many Humble Indie Bundles to come, included support for Linux, as well as Windows and Mac OS X. With Linux purchasers of the first Humble Indie Bundle voluntarily paying more for the bundle than the Windows or Mac OS X gamers paid, there was suddenly hard evidence that Linux might be a profitable platform for game developers.

The Humble Indie Bundles, then, effectively paved the way for increased interest in Linux from the major players in the video game industry—those making the games with the highest production values and largest budgets on the planet, also known as "AAA" games. For personal computer gamers, the biggest gorilla in the jungle is Valve, which not only makes highly acclaimed and widely popular games, but also owns Steam. Steam is a proprietary software distribution platform with over 3,000 games (as well as some non-game software), most developed by third parties, and an estimated 75% of the worldwide market for PC gaming.

[Half-Life 2]

With development of a Steam for Linux client officially announced in 2012, and a stable release in early 2013, the biggest commercial player in desktop gaming effectively announced to game developers worldwide that it thinks Linux is part of the future of desktop gaming. Steam also recently concluded a Summer Sale, which had many Linux games available at a steep discount. All of that adds up to a good time for a detailed look at Steam for Linux.

Steam for Linux

Initially available for Ubuntu only, Steam for Linux is now available for download for almost any recent Linux distribution. System requirements for the client itself are fairly light, with the biggest hurdle being graphics card support; at least Intel HD 3000 integrated graphics is needed, though having an AMD or NVIDIA card will let one play more games—using the binary drivers will give one access to even more. Individual games on Steam have their own system requirements (example), and while many need something more powerful, like a recent NVIDIA card using the binary driver, there are lots of quality games (including fairly recent AAA games) available for someone with integrated graphics and open-source drivers.

Before installing the software, the user must agree to the EULA. The user then creates an account, which is no more difficult than making, say, a new Gmail account. When Steam launches, the user is greeted with the storefront to find games to buy. Linux gamers can choose to browse by platform, where they can find hundreds of games available for Linux.

[Steam client]

Steam comes with a number of conveniences for gamers. Aside from the integrated store, it acts as a library, neatly organizing a player's games. Some games also feature "cloud saving" to allow gameplay to continue on one computer from where a player left off on another. Steam automates and simplifies game updates, providing bug fixes and new features. It also provides social networking features to connect with friends on the service and to find players for multiplayer games.

Linux gamers on a tight budget and with fairly low-end graphics can find a lot of high-quality entertainment at no cost. A great starting point, for those unsure where to begin, is Valve's own Team Fortress 2, a wildly popular and critically acclaimed first-person shooter. It has a comical but high-quality graphical style with nine different playable classes to suit a variety of different playing styles.

[Team Fortress 2]

Those who like stealth games can choose the Spy, who uses a cloaking device, disguises, and a knife to sneak up on and assassinate opposing players. Those with a more strategic bent can play as the Engineer, who can build a Sentry gun for offensive or defensive purposes, or build a number of supporting buildings to assist teammates. Those interested in simply running straight into the fray—guns blazing—can pick the minigun-toting Heavy, with the highest maximum health among the different classes and a devastating default weapon.

Team Fortress 2 works with relatively modest graphics processing, and is free-to-play, with a completely unintrusive and optional microtransaction system; I've logged 16 hours of game time, so far, without spending a penny, and I've loved every minute of it. Gamers can purchase cosmetic items for their characters, like hats, as well as alternate weapons. Many of these items are also given to players automatically for free after completing certain tasks, so budget players don't feel like they're deprived of all of the fun the game has to offer.

Another free-to-play game with optional, non-game-breaking microtransactions is Dota 2. It is currently the most popular multiplayer online battle arena game in the world, with millions of players and millions of dollars in cash prizes available for winners of official tournaments. Those interested in single-player gaming can try Floating Point, a free and relaxing "physics" game, where your character swings from a grappling hook to collect points in a procedurally generated 3D environment accompanied by great ambient music.

While there are some open-source games targeting the platform, such as Narcissu 1st & 2nd, the overwhelming majority of games are closed-source. Interestingly, the open-source Krita painting program has a release on Steam, but only for Windows. Steam users can buy Krita Gemini, which is based on the regular Krita desktop program, but adds a touch-friendly interface for tablets and uses Steam's convenient automatic updating functionality. $49.99 buys upgrades through to version 3.0 (roughly a year of upgrades) or lifetime upgrades are available for $119.99.

The future

With Linux users only making up 1.20% of Steam's user base (click on "OS Version"), compared to Windows gamers making up 95.04% of Valve's customers, some might be wondering what Valve is thinking. Valve is playing a long-term game here, guarding against what it perceives to be Microsoft's alienation of its users due to Windows 8, as well as the global decline in PC sales. By focusing on an open-source platform that they can control, Valve can pivot to protect itself from these market conditions. Following the July 2012 announcement of the development of Steam for Linux, Valve co-founder Gabe Newell made clear that he sees Linux as the company's future: "We want to make it as easy as possible for the 2,500 games on Steam to run on Linux as well. It's a hedging strategy. I think Windows 8 is a catastrophe for everyone in the PC space."

This is where SteamOS, and the Steam Machines it will run on, come into the picture. Based on Debian, SteamOS is a full desktop Linux operating system that launches by default into Steam's Big Picture mode, which is a user interface for Steam optimized for television sets. Currently, SteamOS is in beta; it will launch later this year on various Steam Machines—multiple video game console hardware platforms made by different manufacturers.

SteamOS gives Valve much greater control over the Steam platform's future. Steam Machines would also expand Valve's target market to "living room" gamers. As only native Linux games will work on SteamOS and Steam Machines, Valve needs its games to work on desktop Linux and needs third-party developers to release native Linux games on Steam. Hence the big push for Steam on Linux first, and then the focus on SteamOS and Steam Machines.

DRM and privacy

Perhaps the biggest concerns with Steam on Linux, aside from the flood of proprietary games that have concerned some free-software hardliners, are Digital Rights Management (DRM) and privacy. By default, Steam is online; users must authenticate before activating "Offline Mode" to be able to play indefinitely without an Internet connection. On top of that, Steam offers third-parties the option to use its DRM technology through its Steamworks API. This optional DRM includes "Custom Executable Generation", which ties a game copy to a particular account. Validating these executables does not rely on authenticating the hardware itself, so users would be free to play the game on any supporting computer, as long as they can log in to their account.

Valve also offers optional encryption to allow pre-ordering users to download the game before release date, but to prohibit them from playing until that day. Steamworks also provides hooks for third-party DRM integration, so some games have stricter restrictions on their players.

Steam's own DRM measures remain fairly light for an industry notorious for aggressive DRM. But others have been concerned about privacy for Steam users; the Valve privacy policy contains broad language allowing Valve considerable leeway with its use of personal information. For example: "Personally identifiable information is used internally by Valve to develop, deliver and improve our products, content and services." The most visible user-facing issue here is the Valve Anti-Cheat (VAC) software that runs in the Steam client and aims to detect cheaters in certain multiplayer games and ban them from Steam.

In March 2014, there were allegations on Reddit that VAC was sending its users' entire DNS cache to Valve. After some fury and concern by video game players, Newell responded by explaining clearly what VAC was doing. While that response may have dispelled the rumors, the explanation, combined with the privacy policy, may be enough to deter some potential users. One notable line from his post explains why VAC can be unnerving: "VAC is inherently a scary looking piece of software, because it is trying to be obscure, it is going after code that is trying to attack it, and it is sneaky."

Those concerns aside, in less than two years, Steam has already had a major and, arguably, positive effect on desktop Linux. Valve has put some of its considerable resources into improving the open-source graphics stack, including its financial backing of experimental development work on Mesa. But perhaps the most substantial effect is the positive perception that gamers on other platforms will be getting of Linux. With hundreds of games available, and more in development, ranging from independent, experimental games to some of the best that the AAA world has to offer, desktop Linux has finally cleared one major hurdle to widespread adoption: availability of quality video games.

Comments (10 posted)

Genealogy research with Gramps

July 16, 2014

This article was contributed by Vladimir Perić.

Genealogy is a fairly popular pursuit, and those wishing to use open-source software in their hobby have their choice cut-out for them—Gramps is the only complete, actively-developed free-software solution. The project was started in 2001 and initially known as GRAMPS; the first stable release was in 2004. The latest, version 4.1.0 ("Name go in book") was released on June 18.

Before looking into Gramps's internals, it would be prudent to examine what genealogy entails. Researching family history is not just a matter of raw data (names, birth, death and marriage dates, locations, etc.), but also of understanding the lifestyles and motivations of people. Research can be focused on an individual and their ancestors and descendants, as is common in hobbyist genealogy, but it could also focus on a single surname (a one-name study) or a location (one-place study); the scope might simply be constrained by the free time of the researcher.

The de-facto standard format for genealogy data exchange is GEDCOM (GEnealogical Data COMmunication), though most applications also have their own, more full-featured formats, addressing issues in the slow-to-develop GEDCOM standard (the latest version was released in 1996). Gramps is able to import and export a number of file formats, while internally it uses XML.

[The People view in Gramps]

At its core, the Gramps model uses three basic object types: Person, Family, and Event. A Person has one or more names and a gender, while a Family represents two Persons and their children (i.e., a "nuclear family"). There is no larger structure connecting Family objects; Persons are just members of multiple Families—typically, one with their parents and one with their spouse. Events can either be connected to a Person or to a Family, and contain a date (or date range) and, optionally, a Place.

Other object types are used as well. Notes and Media can be attached to any other object. A Source object contains information about a single data source, while multiple Sources can grouped in a Repository object. Finally, Citation objects are used to connect a Source to other objects. All of these objects together form a Family tree. Although this relational scheme might appear complicated at first glance, it allows great flexibility for Gramps in adapting to a given workflow.

Giving freedom to the user is a defining characteristic of Gramps. There are plenty of Event types already available, encompassing everything from Birth, Baptism, Death, and Cremation to various vocational (e.g., Occupation, Military service), academic (Education, Graduation) and travel (Migration, Naturalization) events. But the user can just as easily create a custom Event by simply typing in the drop-down menu instead of selecting an Event. The same freedom is available everywhere else, though the "official" names should be preferred whenever possible (as they can be localized, used automatically in reports, etc). I have never had to introduce a custom type, as everything required was already provided by Gramps, but the option is there.

[The Relationship view in Gramps]

The Gramps interface is organized around the listed object types, with two additional tabs available, "Relationships", which provides a more intuitive representation of a Family object, and "Charts", where the user can quickly view several visualizations around the currently selected "Active person": included are a pedigree, timeline pedigree, as well as fan charts of descendants and ancestors. Each screen also has a sidebar and a bottom bar, where the user can use "Gramplets" to show whatever is required. Gramplets are a type of plugin; they represent mini-views which can be used to personalize the interface: Age on Date, Citations, and a simple to-do list are some of the available Gramplets. By default, the sidebar contains the Filter Gramplet, which can be used to quickly filter the data shown in the main window.

A workflow example

Whenever an object is opened, a new window pops out containing the data associated with the object; clicking on other objects will pop out yet more windows. This can sometimes result in many windows being open at once, which can be confusing. To make the entire process more understandable, I will provide an example of my typical workflow.

[Object editing in Gramps]

All data should be sourced, so I first check whether or not I have used this particular source before. Let's say I have found a marriage certificate on some microfilm that I had overlooked earlier. As I already have the Source object in Gramps, I would open the Family view and find the family of interest. Next, directly on the Family object, I would add a new Citation, using an existing Source, to confirm that the family in fact is indeed married. The crucial information on every Citation is the volume, page, or other information I would need to find the relevant entry again. I could also specify the confidence of the citation (a feature I do not currently use) in case I have multiple conflicting sources (for example, a birth certificate will likely have more accurate birth date information than a marriage certificate, but both pieces of data should be recorded).

Since my source has additional information, I would then open the useful Clipboard Gramplet and drag-and-drop my Citation there. Now I can easily drag it to any other object I decide to cite, simplifying my workflow and ensuring no duplicates occur. Having all of my data connected in such a way makes it easy to add an alternate source location (for example, if the certificate becomes available online) or I can attach a Media object containing a scan of the page.

Readers should note that this is just my personal workflow, and others might be equally valid; some professionals might have just cringed at my butchering of data and some hobbyists might think I am over-complicating the simple matter of drawing a pedigree. Luckily, as noted above, Gramps makes no presumptions on its users' workflow, so can be used as seen fit. For those unhappy with their current data representation, Gramps offers a number of tools for database management, including merging objects and automatic extractions of data. All of these features combine together to offer very flexible data input.

[Visualization in Gramps]

However, collecting data is only half of genealogy, with the other half being arguably more interesting—visualization of said data. Other than the aforementioned "Charts" views, which can be used for quick inspection of data, Gramps offers "Reports," divided into textual, graphical, and web. By default, 14 textual and 10 types of graphical reports are offered, though there is some overlap. Textual reports, such as an Ahnentafel table or a detailed ancestor report, can be created in various formats, including HTML, PostScript, and ODT. Graphical reports offer various tree views of data and are available in PostScript, SVG, and other formats.

Perhaps the most interesting is the narrative web report, which can represent the entire database as a website, and can be an attractive way to present the data to non-users. The plugin generates a static website, containing a page for each selected Person and all of the appropriate index pages; here's an example. All reports can be customized as appropriate or printed in any of the supported languages. Special care is given to privacy; it is easy to mark select records as private, or to not print records related to living people (and optionally those recently deceased).

Internationalization is another major focus of Gramps, exploiting a niche that proprietary software makers tend to avoid. Not only is Gramps translated into over 20 languages, it is also possible to define custom date handlers, which allow for locale-specific data parsing and display, and relationship calculators, which offer translated kinship terms (e.g., a "second cousin, thrice removed" in one culture is simply called a "brother" elsewhere). When adding dates, it is possible to use several calendars: the standard Gregorian, but also the French Republican, Swedish, and several others. Adapting any of these features to a new language or locale is relatively simple.

[Report generation in Gramps]

Gramps also offers a plugin system, which can be used to update or download new reports, or to write Gramplets, new tools, relationship calculators, and several other possibilities. Plugins are written in Python, the language Gramps itself is programmed in—at over 450,000 lines, it is one of the larger open-source Python applications. Gramps has plenty of dependencies, so installing it can be somewhat bothersome. In particular, osm-gps-map library is required for a Geography view, which I haven't successfully managed to get working. Binary packages of Gramps are available for download and most distributions provide Gramps in their package repositories (although distribution packages other than Debian and Ubuntu tend to lag behind the project). Gramps is also available for Mac OS and Windows.

Gramps switched to GTK+3 in version 4.0 and has full Python 3 support as of version 4.1. The major feature in 4.1 is a new place hierarchy model. There are also improvements to localization capabilities and other assorted updates; a full list is available in the release announcement and on the bug tracker. Unfortunately, it seems that the move to GTK+3 has resulted in performance regressions; these will undoubtedly be worked on and fixed. The program is still fast enough, but it certainly felt "snappier" in earlier versions. As is common with open-source projects, Gramps has too few developers working on it and help is always appreciated. The plugin architecture and a robust API make it easier to jump into development.

In short, Gramps is a powerful, flexible, and mature project, offering many features, with a strong focus on user freedom and internationalization. While it can certainly feel rough around the edges sometimes, it also easily surpasses proprietary solutions on several fronts; perhaps no open-source alternative has developed because Gramps can already accomplish almost everything. Anyone with an interest in genealogy will be well-served by giving Gramps a try.

Comments (24 posted)

Page editor: Jonathan Corbet

Inside this week's Weekly Edition

  • Security: The security of; New vulnerabilities in claws-mail, docker-io, java, kernel, ...
  • Kernel: System calls part 2; Control groups part 3; inotify.
  • Distributions: Debian adds continuous integration; Debian 7.6, FreeBSD 9.3, ...
  • Development: The glibc s390 ABI break; KDE Plasma 5.0; Sandboxed applications for GNOME; Mozilla improving JPEG encoding; ...
  • Announcements: Mapbox runs like an open source project, Tracing Summit, ...
Next page: Security>>

Copyright © 2014, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds