Leading items

With Madison it's not just a bill

US House Representative Darrell Issa has been an active participant in "open government" advocacy in the United States over the past few years; among other things, he co-founded the OpenGov Foundation, which is dedicated to increasing access to government data. Free software advocates will also remember Issa's participation in the opposition to SOPA and PIPA in December 2011. That effort produced an online "legislative markup" application called Project Madison. The Madison source code has now been released on GitHub under GPLv3, for immediate use by DIY-legislators and armchair founding fathers — and potentially by other communities interested in collaborative editing and criticism.

In its original form, Madison allowed critics to log their complaints about the SOPA/PIPA legislation, and to help crowd-source an alternative bill known as the OPEN Act. Issa told the O'Reilly Radar blog in July that he was working to get the OpenGov Foundation (which is not to be confused with the Sunlight Foundation's OpenGovernment.org) registered as a 501(c)(3) nonprofit in the US, and that the Madison code would be released under an open source license as part of that effort. The source code release was announced on September 28 on the OpenGov blog. A live Madison installation is running at the KeepTheWebOpen site, which currently hosts nine bills and related documents for public commentary and improvement (including the commentary recorded for SOPA and PIPA).

Obviously other tools exist for collaboratively editing documents (EtherPad derivatives perhaps being the most well-known). But Madison is designed to preserve the canonical form of the document while still enabling feedback. The stated goal of Madison is to permit such feedback in a way that makes contributing easy, but also enables administrators to sort through potentially thousands of comments in a meaningful fashion. Madison presents a document in structured form, divided into paragraph-based sections (it appears that legislation is often drafted in one-sentence-per-paragraph style, so this is in fact quite granular). Users can attach separate comments to each section, as well as propose re-wording suggestions. Both types of feedback are presented in a sidebar to the document, but suggestions and comments are displayed in separate boxes in the interface.

Users can also register "likes" and "dislikes" on each feedback submission posted, as well as flag inappropriate comments. The interface tracks likes and dislikes, plus users' Facebook "likes" and Twitter tweets spawned by the submission. For each section, the interface sorts user contributions by an aggregate of these community metrics so as to allow popular ideas to bubble up to the top for easy consumption by the administrator. If the administrator chooses to incorporate a user-contributed change into the main document, that change is highlighted in the document interface with a different background color. Anonymous comments are not supported; the application supports both individual and group user accounts (although group accounts must be requested and approved by administrators). Facebook login via OAuth 2 is also supported.

Under the hood, Madison is straightforward PHP and MySQL. However, a major limitation is that each document presented for public commentary needs to be manually added to the database, with one database row per document section. There is also no installer to set up the database tables and create administrator accounts, which makes getting started more complicated. But the Madison team is well aware of the hardship these limitations impose, and has posted a roadmap on the GitHub page that outlines plans for these and other features. Also on the list, for example, is support for larger, multi-part documents, in addition to general improvements like additional third-party-account support (Twitter, Reddit, Google Plus, etc.).

World of commentary

Madison was written during a Congressional hackathon in December 2011. The source code release follows on the heels of the White House's August release of its online petition application We The People. We The People is implemented as a Drupal module, however; hopefully Madison will evolve into a component more easily integrated with existing sites, because there are clear use cases for such an application that have nothing to do with politics.

For example, online document mark-up and commentary have become an integral part of the free software community's license revision process. The FSF commissioned its Stet mark-up tool for use in crafting GPLv3, and the COMT application was used by Mozilla during the comment period for MPL2. Madison offers some features not supported in Stet or COMT, such as the user-voting mechanism, and differentiating between individual and group accounts. However it is still a bit behind on the feature front overall; the other tools support things like version comparison that are valuable whenever wording is expected to change.

Madison also has user interface issues to be resolved, and the administrative interface lacks some tools that would be helpful in a lengthy comment-gathering process (such as the ability to reclassify a comment as a suggestion, or vice-versa, and the ability to indicate when a suggestion has been acted upon). But there is no denying Madison's real-world suitability. According to The Atlantic Wire, the application successfully managed more than 200,000 visitors during a 12-hour SOPA editing marathon. At that scale, Madison's ability to help administrators separate the wheat from the chaff could be the difference between a useful comment period and chaos. One could also argue that the demise of SOPA and PIPA is in part an indicator of Madison's success; although there were clearly more important factors, improving access to the text of the bill and encouraging citizens to delve into the details certainly helped the public's grasp of the issues.

A good commenting and annotation system is critical to the creation of any document that needs public acceptance. In recent years, the free software world has seen several such processes, and frequently those that are seen as doing a poor job listening to or responding to public comments attract criticism (such as Project Harmony). Madison is a noteworthy release because it represents progress in the advancement of open government principles, but it is also valuable for enabling anyone to collect document feedback and contributions from the public, which is a principle that the free software community holds dear.

Comments (2 posted)

XDC2012: Programming languages for X application development

To what extent is the choice of programming language important in terms of making it easier to build desktop applications? This was the question asked by Bart Massey in his talk on day two of the 2012 X.Org Developers' Conference.

Bart observed that we've now reached the point where it's harder to write a desktop application than it is to write a mobile or web application. Furthermore, the quality of desktop applications is often worse than for mobile and web applications. In Bart's opinion, the situation should be the other way round, and he is puzzling over why it is not.

Some factors that make programming desktop applications harder are beyond the X developers' control, he noted. For example, desktop applications may be much bigger than mobile and web applications. They may also have requirements that are not present, or are simply not addressed on mobile and web platforms. For example, desktop applications may need to manage quite sophisticated user interactions, and provide full internationalization support (which is omitted from many mobile and web applications). It's hard for the X developers to control these factors, but Bart noted one area where they can have some influence in making the development of desktop applications easier: the choice of programming language.

When it comes to programming languages, how are mobile and web applications different? Bart pointed out that—unlike desktop applications—it is notable that they are almost exclusively not written in C or C++. Objective C is used in the mobile space, but Bart noted that that language is special (because it provides some features not found in C/C++ that are found in the other languages used for mobile applications). In the mobile and web space, nearly everyone is using languages that have automated memory management; that eliminates problems with memory leaks and mishandled pointers. These programming languages commonly support late binding and dynamic loading. Languages that provide these sorts of features are attractive because they allow developers to be much more productive. That's especially important because the mobile and web spaces are fast moving and programmers need to be able to put applications together quickly. By contrast, developing for the desktop—and Bart made it clear that here he was speaking about desktops in general, not just X—requires a much greater initial effort before a programmer is productive.

On the X desktop, the common programming languages have been C and C++, and Bart noted a number of ways in which these languages have been a source of pain for application developers. There are many reasons to use multiple threads in GUI programs, most notably because a windowed interface in a GUI environment is naturally concurrent. However, threading support in C and C++ (typically via POSIX threads) is not easy to use for the average application developer. The frameworks used in C/C++ often involve callbacks and other complicated flows of control, which are likewise challenging for programmers. Manual memory management is tricky for programmers as well: monitoring memory usage of desktop applications on any platform (not just X), it's easy to find cases of applications that leak memory. And while its true that things can go wrong with automated memory management, in practice those problems are not as bad.

In Bart's view, the choice of programming language (and the accompanying frameworks that that choice implies) is one of the things getting in the way when it comes to developing X applications. The X tool kits have a widget mentality, and while he feels that that is probably a good mentality, the toolkits make widget creation a lot of work: too much boiler-plate code and "weird interaction" is required for building widgets. "The problem is that the language doesn't give you convenient ways to express what you want to express."

We use C and C++ on the X desktop because they've always been there, Bart noted. But, what are the alternatives? Bart estimated that there are some 400 programming languages in wide use, but noted that there are obvious attractions to using a mainstream language, in terms of the pool of developers and available support. Java is an obvious choice. C# is another possibility, but he thought that there were no advantages that made it clearly compelling over Java. If one widens the net, other alternatives might include SML, OCaml, or Haskell, but he felt sure that if he asked the room what language should be the future of desktop development, he'd get fifteen answers.

Bart then considered the toolkits used with X a little more deeply. He noted that low-level libraries such as XCB are written in C/C++. There are various reasons for that choice, among them efficiency and the ease providing foreign-function interfaces (bindings) for other languages. The choice of C/C++ at this level of the stack is okay, he thinks. But, as we go up the stack, the next thing we come to is the toolkits, and the choices there are Qt and GTK. Both are old, entrenched, and written in C/C++. Because of that, the applications created above them tend to be written in the same language. It's time to think hard about whether these are the right tool kits for applications built in 2012. The problems of building desktop applications are amplified by using these large, complex libraries, and they don't provide the automated support that is provided in mobile and web frameworks.

Bart noted some summary points from his work during the last ten years. If someone asks him what language to use to write a graphical desktop application, he is typically going to respond: Java. This is ironic, because he, an X developer, is recommending a language that will not produce a native X application. But, he noted that the overhead of writing a Java GUI application is simply lower (than C/C++), even when developing for the Linux/X desktop. Another reasonable alternative that he sees is Python with a toolkit (for example, wxWidgets). As a teacher (at Portland State University), Bart recommends those alternatives to his students, rather than the C/C++-based frameworks available for X.

Is programming language even the right thing to be thinking about, when considering problem of building desktop applications? Bart said that the audience might be able to convince him otherwise. But, he wanted to know, if programming language is not a part of the problem, then just how weird does a programming language need to be before it is a problem? He then mentioned a few factors that may or may not matter with respect to the choice of programming languages and toolkits. One of the more interesting of these was portability of applications across desktop, mobile, and web. It would be nice to have that, though it is "really hard." But, he asked, what can we do in terms of choice of programming language to facilitate that possibility?

On the other hand perhaps we are stuck with GTK and Qt, Bart said. He noted that it was hard enough to replace Xlib with XCB. The task has taken ten years, and Xlib is likely still to be with us for another decade. Perhaps replacing the toolkit frameworks is too difficult, and when it comes to developing desktop applications, we're consequently stuck with C/C++ forever. He concluded, however, that "if we're going to save the desktop—and don't fool yourself, it is in trouble—then I think programming language is part of the problem."

A lively discussion followed the formal part of Bart's presentation. In response to some comments from Supreet Pal Singh about the satisfactions of doing X application development in C/C++, Bart elaborated further on his point. He noted that that there is no ramp to being an X developer—it's a cliff. Developers have to master many technologies before they can do anything. He acknowledged that he too has a different feeling when developing X applications in C++, "but it's not a happy one: I feel smart for being able to do this in C++, but not smart for having done it".

Another audience member noted that Qt Creator can be used for rapid development of X applications. Bart observed that code-generation tools solve some problems, but replace them with other problems as one reaches the limits of what the code generator can do. At that point, the developer then has the problem of trying to understand what is going on under the hood of the generator and possibly tweaking its output, which can complicate code maintenance. He also noted that a GUI application created with a code generator tends to behave less well for the user. For example, he suggested, try using a GUI tool to build an application designed for a small window and then scale that window up and watch what happens. The result is very different from an application where the layouts and layout policy have been hand-crafted so that it works well for all window sizes.

Peter Hutterer pointed out that it may be because of the relative youth of the web and mobile application spaces that they are not suffering some of the same problems as desktop X applications. X has been around for well over 20 years; in 10 years' time, web and mobile applications may have some of the same issues. He mentioned Android fragmentation as a possible example of the kinds of problems to come. Bart agreed that the passage of time may bring legacy problems to mobile and web application development, but noted that many of his points were independent of legacy problems: the need to manage pointers and memory, wacky untyped (void *) interfaces, and complex linking interactions. He constantly sees his students struggling with these problems.

Matthias Hopf pointed out one language that was missing from Bart's discussion, but noted that he was hesitant to propose it as a candidate. That language is of course JavaScript, which started out in the web space, but is making its way to the desktop in the form of node.js. Bart responded that he wasn't against JavaScript as a development language. Some aspects of the language make him suspicious, "but the very fact that people are successfully developing web and mobile applications in JavaScript and HTML5 says that it's worth considering [for the desktop]. It would be interesting to consider what a set of JavaScript bindings for X would look like."

Other members in the audience noted that the combination of QML, JavaScript, and C++ is an effective framework for rapid development of X applications, though Keith Packard interjected that he didn't want to have to write an application in three languages. Although Bart acknowledged that some of the information about QML was new to him, by the end of the session, he still didn't seem to be convinced away from his thesis that the current X programming languages and the frameworks were part of the barrier to developing X applications.

Decades of experience have shown C and C++ are powerful languages for system programming, but few would deny that they provide application developers with too many opportunities to shoot themselves in the foot. Bart was facing a rather atypical audience: a room full of C/C++ experts. Nevertheless, there was some sympathy for his thesis, and one suspects that if a more typical cross section of application developers were asked, there would be rather more agreement with his position.

The X.Org wiki has a pointer to the video of this session.

Comments (62 posted)

Tent pitches a new social networking protocol

The free software community produces a constant stream of ideas about how to displace the proprietary network services that dominate so much online interaction. In mid-2012, Tent became the latest entrant in the conversation, heralding an "open, decentralized, and built for the future" social networking solution that "changes everything." Beyond the project's manifesto, however, there was scant detail, particularly on how Tent related to other distributed social networking efforts like OStatus, the protocol used by StatusNet and Identi.ca. September 21 brought the first look at something more concrete, courtesy of a reference Tent server and initial documentation of the system's protocols.

Staking out the turf

Tent's general idea is familiar enough: a functional replacement for proprietary social networking services like Twitter, Flickr, and Facebook, but built with free software and designed so that individual users retain control over their data (including not handing over personal information to a third party). A key part of making such functionality possible is devising a mechanism for distinct installations to interoperate, thus allowing users to converse, share content, and subscribe to content posted by others — without demanding any permanent ties to other users' software.

This goal is much the same as that of the OStatus community, which led a number of people to open issues on the Tent bug tracker asking what justifies starting the new project. The Tent FAQ (on the project home page) says that "the architects of Tent investigated existing protocols for the distributed social web. Each of them lacked several critical features necessary for modern users." Elsewhere the FAQ comments that OStatus and Diaspora were "first steps" but does not go into detail about what they lack. On the issue tracker, however, developer Jonathan Rudenberg lists three features not covered by existing federated social networking projects: support for private messages, a server-to-client API, and the lack of a "social graph" specification (e.g., existing "friend" or "following" relationships) enabling users to export their user-to-user connections for portability between services. Developer Daniel Siders reiterated those issues in a Hacker News discussion about Tent.

Several commenters found those features to be weak justification for writing entirely new protocols, however. Dave Wilkinson II argued that OStatus does not address private messaging because it does not attempt to address identity management, but that the related standards Webfinger and PubSubHubbub together can be used to implement private messaging. He also said that migrating social graph information is trivial in OStatus precisely because OStatus does not bind to the user's identity. OStatus co-author Evan Prodromou said private messages were in development for PubSubHubbub 0.4 (and subsequently OStatus), and pointed to ActivityPub as an effort to develop a generic server-to-client API.

The Tent project's documentation also sees its definition of "decentralized" web services as being fundamentally different than OStatus's definition of "federated" services. Prodromou suggested on the issue tracker that this distinction was inaccurate, and that what Tent describes is no different than the federation functionality of Status.Net and Diaspora. Siders replied that:

In other words, he continued, Tent differs from federated social networking systems because it combines the server-to-server and client-to-server communication protocols, akin to unifying SMTP and IMAP. "Tent is not a federation protocol because it provides end to end communication between users, not just servers."

The 0.1 specification

Abstract principles aside, the Tent team released version 0.1 of its protocol documentation on September 21, as well as tentd, a demonstration server written in Ruby that implements a Twitter-like service. The documentation outlines the basics of Tent's messaging and network design, the server-to-server protocol, the server-to-client API, and descriptions of post and profile data fields. In general, Tent uses JSON to format all messages, with OAuth 2 authentication for applications and HTTP MAC access authentication to cryptographically verify individual requests and responses.

Every Tent user (or "entity") is associated with a separate server, which is expected to always be online and accessible over HTTPS. Servers are meant to be found through HTTP Link headers and HTML link tags that point to the user's profile URL. Requesting the profile URL returns the user's profile data as a JSON object. Currently the server-to-server protocol addresses only Twitter-style "follower" relationships; user A can follow user B by POST-ing a request that includes user A's own entity URL, the flavors of post the user wishes to subscribe to, the licenses acceptable to user A, and a URL to which user B should send posts. Assuming user B approves the follow-request (which is not addressed in the documentation), user B's server sends its MAC to user A so that subsequent posts can be authenticated.

The documentation does not address private messages directly, other than their usage of MAC authentication. However, the post documentation mentions a permissions object that, in the examples, can be marked as public or list specific entities or groups that are allowed access. Groups and their representation are not currently defined. The notion of "acceptable licenses" is not explained in detail, either; it seems to place the burden on the publishing server to filter out content that individual subscribers do not find acceptable on licensing grounds (the only license example used is Creative Commons Attribution 3.0). The server-to-server API also defines methods for requesting another user's list of followers, the entities that the user is following, canceling or altering a follow-request, and fetching another user's posts (either in bulk or by querying parameters like publication date).

Each of these methods has a corresponding method in the client API; in practice user A's front-end software would relay these requests to the user's Tent server, which would in turn handle the nitty-gritty of subscribing or querying user B's server. As it stands, the scheme is quite simple; there are six post types defined: status (a short, 256-character message), essay (a longer, unlimited-length text entry), photo, album (for a collection of photos), repost (a pointer to another user's post), profile (a notification of changes to a user profile), and delete (a notification that another post has been expunged).

Currently unaddressed are activity-style posts (e.g., Facebook-style "likes," geographic check-ins, or any number of other actions). There was some discussion on the Tent issue tracker about adopting the ActivityStreams format for these post types. User profiles are defined; there are three required fields (entity URL, the licenses under which the entity publishes content, and the canonical API root URL needed to interact with the entity's server). A handful of other properties are defined as well (avatar image, bio, location, etc.).

Interested users can sign up for a free Tent account at tent.is. The site runs the TentStatus application, although free accounts are only permitted to send status-type posts; access to essays and photos costs $12 per month.

Tent invented here

Tent 0.1 is bare-bones, to say the least. Several of the key features that are supposed to differentiate it from OStatus, Diaspora, and other systems are simply not present, which makes it difficult to assess fairly. For example, there is no way to export one's social graph to import it into a separate tent.is account. There are several existing standards for social graph-like information, such as the RDF-based FOAF and XML-based XFN. The Tent team has been critical of using anything other than JSON in its debates on the issue tracker; it would be interesting to see how they implement the social graph functionality.

But there are also aspects of Tent 0.1 that simply need stress-testing. The fact that subscribers tell publishers which licenses they find acceptable for future posts is puzzling, and it will be interesting to see whether that scales well in practice. The diagrams on the Tent site appear to indicate that each publisher sends a separate copy of each status update to each subscriber. When multimedia content is allowed, that could be problematic (and it is one of the problems PubSubHubbub was created to address, regardless of whether or not one finds it an acceptable solution).

At a more fundamental level, though, several commenters in the issue tracker and other discussions are unconvinced that Tent's decision to associate a user identity with a URL is wise. The prevailing wisdom is that users (particularly non-developers) associate URLs with content, not with individuals. Many consider OpenID's requirement that every user have an OpenID URL as an identifier to be one of its greatest flaws. As long as tent.is remains the only Tent site (and tentd the only Tent implementation), the URL identity question will remain unexplored because all users exist in the same namespace.

Of course, until there are other Tent servers and applications, none of the federation/decentralization features can really be put to the test, either —not to mention shaking out Siders's assertion that Tent is not "federated" because it connects users rather than servers ... even though every user is required to have a separate server.

In short, the interesting bits are still theoretical. One only hopes that we will get to examine these other bits before too long. It is not immediately convincing that Tent's approach of bundling identity, server-to-server, and client-to-server into a single API is a strength. But it is clear that by starting over from scratch on all of these topics the Tent team has carved out a much larger task for itself than it would have if it had attempted to implement private messaging in OStatus.

Toward the end of discussions like the OStatus issue tracker thread, a lot of the reasons for design decisions seem to boil down to personal preference: JSON versus XML, HTTP versus Webfinger, and so on. There is certainly nothing wrong with building an application to suit one's own preferences, but in the long run it is a difficult way to establish a standard. The Tent FAQ ends with a statement affirming the project's commitment to "open governance models leading to a ratified standard". But, as Steve Klanik observed, "Working with existing standards is way less fun than just building your own." That said, the OStatus suite of protocols is indeed slow-moving and feature-incomplete; perhaps Tent can spur that community on — it has certainly reinvigorated the discussion already.

Comments (9 posted)