LWN.net Weekly Edition for January 15, 2009

The exceedingly grumpy editor's accounting system update

When your editor posted the Grumpy Editor's next project, he certainly did not anticipate that it would take more than a year and a half for the next installment to be written. Or that, even after all that time, the project of moving LWN's accounting from proprietary software to free software would be incomplete. But the world is full of surprises, even in places where surprises are most unwelcome - like accounting. Happily, your editor's surprises do not involve counterparty risk, credit-default swaps, or anything else of that sort.

So why has this project taken so long? What it came down to is that your editor concluded that he was not sufficiently qualified to rip out a functioning accounting system and replace it with something out of a CVS server somewhere. There is simply too much to know about how the accounting system ties into the company's operations, how our accountant uses it, and how it helps keep the tax agencies happy and the company's officers out of jail. That latter point became especially relevant as LWN's longtime bookkeeper and occasional contributor Dennis Tenney headed off to pursue other opportunities, leaving your editor to take up the legally-liable Treasurer position.

In other words, swapping out the accounting system isn't something to be done on a whim, like, say, putting an -rc1 development kernel onto the production server. Whatever is there has to work. So your editor concluded that the first step in this process was to take over the existing system and come to understand it well enough to be able to properly think about a replacement. After closing out the 2008 books, your editor is able to come to a preliminary conclusion: it is almost possible for a small business to dump a system like QuickBooks and use a free alternative. Almost.

There is an interesting gap in the free software community's offerings in this area. A very small business - one involving a sole proprietor, for example - can use a tool like GnuCash to great effect. Almost everything which is needed is there, and most functions work quite well. On the other hand, a very large operation wanting to install a full-scale ERP system has a wealth of options: Compiere, Adempiere, various packages based on OFBiz, and more. If your operation is willing and able to dedicate full-time staff to developing a customized ERP system and keeping it going, there are plenty of frameworks to start with. These systems are not drop-in tools usable by a small business, though.

Small businesses occupy a niche between the sole proprietor and the massive enterprise. At this level, there's not a whole lot available from the free software community. The most active projects appear to be SQL-Ledger, its fork LedgerSMB, and PostBooks. All three work on top of a PostgreSQL database; SQL-Ledger and LedgerSMB are web-based, while PostBooks is a Qt application. Your editor's sense, at this point, is that PostBooks looks like the most advanced, most ambitious, and most actively-developed project among these three. It is, however, tricky to get running, its development model appears to be strongly cathedral-style (there isn't even a project mailing list), and it is distributed under the questionably-free (though OSI-certified) CPAL license. PostBooks has the look of a classic piece of corporate-controlled open source.

Any of the packages listed above (and GnuCash too) will do basic double-entry accounting. They can produce pie charts, reconcile accounts, and so on. Since they use PostgreSQL with an open (if sometimes poorly documented) schema, integrating them with the rest of the business should be relatively straightforward. They are, in essence, almost everything which is needed to enable a business to move away from a package like QuickBooks.

[PULL QUOTE: The key word is "almost." As far as your editor can tell, there are two crucial bits missing. END QUOTE] The key word is "almost." As far as your editor can tell, there are two crucial bits missing: tax form printing and accountant interfaces.

On the first point: this is the time of year when LWN produces 1099 forms for each of its guest authors - at least, for those who pay their tribute to the U.S. A related form (1096) then goes off to the Internal Revenue Service so they can ensure that none of our authors tries to hide the vast amounts of money we pay them. A tool like QuickBooks tracks payments to outside contractors, and will happily print the requisite forms onto special stock which can be purchased, at exorbitant prices, directly from the application itself. There is no equivalent functionality in the free packages, currently.

In truth, this is an area where free software tends to struggle. The printing of tax forms is just not a task which inspires hackers; it is tedious, subject to highly finicky requirements, and is, for all that it may be considered necessary, somewhat distasteful. This work must be revisited every year as the requirements are subject to the whims of legislators and regulatory agencies. And, lest that challenge seem insufficient, one should also bear in mind that every country's requirements are different, so all this work must be repeated many times over.

Creating this kind of code (and keeping it current) is not fun, it requires some specialized domain knowledge, and it can carry certain kinds of legal liabilities. So it's no wonder that a hacker with some free time will, upon considering this kind of task, usually decide to work on enhancing that Klingon translation of OpenOffice.org instead. The Klingons tend to be more forgiving of bugs.

On the accounting side, the problem gets even worse. A typical business accountant uses proprietary software which, in turn, contains a great deal of knowledge of the tax code. Not all countries have a tax system as twisted and complex as the U.S., but the problem is never simple. Your editor believes that it would be entirely reasonable to require governments to provide free software which interprets its tax codes for ordinary citizens, along with a guarantee that, as long as said citizens fed honest numbers to the system, they would not be subject to penalties if the resulting tax calculation were incorrect. In the real world, though, the job of providing such software falls to companies with a squad of on-staff tax lawyers and a firmly proprietary approach to software distribution. That situation does not appear to be likely to change anytime soon.

For the accountant to use his proprietary tools to come to conclusions about a company's tax situation, he must be able to enter quite a bit of data about where the company's money came from and how it was used. There are a couple of ways in which this can be done: (1) it can be manually entered, at great expense to the company involved, or (2) it can be directly imported from the company's accounting system for free. Small companies tend to be quite sensitive to things involving increased expenses, especially when the expense is for an already unwelcome task like tax compliance. So there is great value in having an accounting system which can export data directly to an accountant's tax preparation tools.

In an ideal world, there would be a nice, XML-based format involving large numbers of acronyms which would make this interoperability possible. In the real world, these formats are proprietary and undocumented. So exporting data to the accountant is not really possible with free tools. And that is the single biggest roadblock to the use of free accounting software in any company whose accounts are even remotely complex. Until free programs can export something which looks like the QuickBooks "accountant's copy," they will not be usable in this context.

What makes this situation even more sad is that, as your editor can now attest through painful experience, QuickBooks really does not have much else to offer. Its interfaces are tedious and error-prone; in many ways, free software has done a much better job. As an example, GnuCash will happily import account data from a bank or credit card company, apply default accounts (categories) learned from experience, filter out any duplicate entries, and allow the user to verify and adjust the whole operation before applying it. QuickBooks is, shall we say, nowhere near as accommodating. Your editor ran into a bug (in QuickBooks 2009) which causes the import operation to fail halfway through; a quick search turned up reports of that bug from 2003. There are reasons why any discussion of QuickBooks harps on the need to perform backups frequently - every fifteen minutes or so is good. Your editor has had to restore those backups many times; meanwhile, use of GnuCash over several years has never, ever resulted in a corrupted database.

What it comes down to is that we have solved at least 95% of this problem, and we have done a better job than the proprietary software companies have. But the remaining gaps are crippling, and they are hard to fill. Accounting file formats are more obscure than, say, document formats; there is no effort to create an OpenLedger specification. Any attempt to create files in those formats from free software is likely to involve reverse engineering efforts, and that will be an error-prone process in an area where errors are most unwelcome. So we may well be stuck with proprietary accounting software for some time yet.

That said, your editor does not intend to give up. There will be ongoing discussions with the accountant and continued tracking of free accounting system projects. The free software community has solved no end of difficult problems over the years; we should be able to find a way to take care of this one too. Stay tuned; hopefully the next update will not be so long in coming.

Comments (52 posted)

Python slithers into Wesnoth

Proposing to change the implementation language for a large project is hardly uncontroversial, but when that proposal calls for moving from C++ to Python, one might expect an enormous flame fest. Surprisingly, a proposal to do just that with the code for the "Battle for Wesnoth" strategy game has resulted in a fairly flame-free discussion. Whether or not the project actually makes the switch—it looks unlikely that any wholesale switch is imminent—there is a great deal of value in the discussion, particularly in its tone.

Eric Raymond is the Wesnoth developer proposing this shift, but it is not his "personal fondness" for Python that is behind it. Instead, he sees it as a way to reduce bugs. Raymond has been handling bug triage for the project for the last year or so, which gives him a good grasp of where the Wesnoth bugs tend to be:

Raymond is cognizant of the downsides of moving to Python as well. First, Wesnoth developers will have to be familiar with both languages, which Raymond, at least, does not view as a problem: "Python is much easier to pick up than C++". Performance is another concern, one that he glosses over with a breezy "machines are still getting faster"—others seem less sanguine about the issue—but he does see two major benefits:

Attracting more developers to the project is another reason to move to Python, one that lead developer David White—often referred to by his IRC nickname "Sirp"—is motivated by. Raymond has been thinking about how to move Wesnoth to Python for a while, without making any progress, but recently a new developer, Ivan Illarionov, has appeared on the scene having translated some portions of Wesnoth into Python. Just how much has been done is still something of an open question, but his approach is an evolutionary one. That is important to White:

Overall, the reaction has been fairly positive, Wesnoth developers seem to be open to the possibility that C++ is not the be-all and end-all of languages for game development. That said, they aren't necessarily willing to hear new developers obnoxiously proclaim that Wesnoth should be redone because Python is "better" than C++, without much in the way of details. Unfortunately, that is the tack that Illarionov has taken, which led White to patiently explain:

Illarionov also ran afoul of Raymond, who publicly castigated him: "You [have] done such an inept and -- at times -- arrogant-seeming job of presenting yourself that you have already alienated some senior developers on this project in just the few days since you've shown up here." Illarionov replied contritely, but almost immediately stirred things up again by a posting with the subject "Wesnoth refactoring and future direction plan". In that thread, he also points to Linus Torvalds's rant about C++, which just gets further under the developers' skin.

But, the Wesnoth developers have shown a great deal more patience than many development groups would. As White puts it: "Trust me, if you sent an email to the Linux Kernel Developers Mailing List entitled 'Linux refactoring and future direction', you would receive MUCH more hostile responses then you have received here." It is likely that much of the tone of discussion on the wesnoth-dev mailing list derives from White's leadership; the contrast between his and, for example, Torvalds's more combative tone is quite apparent.

There are hints that some of the conversation has been less civil, especially on IRC but, on wesnoth-dev, even the rebukes have been relatively polite—certainly by the standards of most development mailing lists. This is a project struggling with a difficult decision without lashing out at those questioning it or outright opposing it—or for that matter, those ineptly championing it. The wesnoth-dev conversation went quiet around January 6th, but the project is known to be very active on IRC, so perhaps it moved there. Even if that turned ugly, the email conversation sets quite an example.

Should it come to pass that Wesnoth starts including more Python—by Raymond and Illarionov or others—we will get an opportunity to see if the hoped-for improvements come about. Projects often consider which language to choose, either initially or for a reimplementation, but there are few examples or case studies of comparative language benefits. A year or two down the road, Wesnoth might provide just that kind of comparison.

Comments (62 posted)

GNOME considers DVCS choices

THE GNOME project has completed a survey of active contributors about which distributed version control system (DVCS) they would prefer to switch to in 2009. The project is now in the process of interpreting the results and deciding on the next steps. As the process unfolds, it provides a vivid snapshot of how free and open source software (FOSS) contributors regard DVCS applications.

DVCSes are an idea that have taken hold strongly in the FOSS community in the last few years, and that is starting to edge out older version control systems such as CVS and Subversion. Unlike such older, centralized version control systems, DVCSes do not require a single repository. Instead, on a DVCS, all contributors have their own repositories, and decision-makers decide which ones to merge for a release. In general, DVCSes are seen as more flexible, although critics argue that they can be too decentralized, and that they cause more conflicts than traditional version control systems during merges. All the same, they continue to be popular, partly because of the high-profile of Git, the DVCS originally written by Linus Torvalds for Linux kernel development after the switch from BitKeeper.

The GNOME survey was privately distributed by Behdad Esfahbod, one of the directors of the GNOME Foundation, in December 2008 to gather background information for a possible move away from Subversion, the version control system used by most of GNOME. The survey was distributed to GNOME contributors with Subversion accounts and SSH keys — a total of 1083 people in all, of which 579 replied.

The survey asked those who replied about their current use of Subversion, as well as their role within GNOME, what DVCSes they used or with which they were familiar, and how they felt about switching from Subversion. They were then asked to rank their preferences for Git, Mercurial (often abbreviated to Hg, after the abbreviation for the element Mercury), Bazaar (Bzr), and Subversion.

Analyzing the results

On 3 January, 2009, Esfahbod published the raw results of the survey. Over the next few days, the results were analyzed by a number of people, including Shaun McCance, Andy Wingo and Elijah Newren, all of whom charted the results. Newren's analysis in particular has become a center for discussion about the survey, undoubtedly because it was the most exhaustive analysis.

Looking at Newren's charts, viewers can quickly see approximate but definite results (those who want more exactness can refer to the raw data). The result is a thorough picture of GNOME contributor's views on DVCS.

Newren cross-correlates survey results in every possible way, but here are some highlights:

• About 60% of respondents claimed familiarity with Git, and 25% with Bzr and 20% with Hg.
• 37% of respondents preferred to switch from Subversion, 34% were indifferent either way, and the rest either support Subversion or did not want to switch.
• 48% chose Git as their first choice, and 25% preferred Subversion. Bzr was favored by about 12%, and Hg by 7%, while 5% expressed no preference. Among second choices, Subversion, GIT and Bzr were all between 20-25%, and possibly within any margin of error. However, in the third through fifth choices, Bzr and Hg were favored as much or more than Subversion or Git.
• Preferences for different roles in the project were clearly defined: Package maintainers and coders steadily preferred Git and Subversion. Translators and testers held the same preferences, but less strongly. Surprisingly, documenters preferred Bzr, but, since only four documenters replied, the validity of that result is questionable.
• Those who wanted to switch, or were indifferent both strongly favored Git.

Newren's conclusion was that "there's a strong preference in the community toward switching, and that git has a strong lead in preference among the community."

As might be expected in an online discussion of FOSS, replies to both Esfahbod's publication of results and Newren's results were not long in coming. Still other comments were posted below LWN's brief mention of Newren's analysis.

Replies to the survey and analyses

Probably the most common criticism was that the survey was as much a popularity contest as anything else. Several commenters also wondered why other DVCS software such as Monotone and Darcs were not included in the survey (a question that, so far, no one has answered). In addition, some commenters were quick to talk about Git's shortcomings, while others — obviously unfamiliar with Git — asked questions about its features.

On the GNOME desktop-devel list, Esfahbod's announcement produced dozens of replies. One of the most articulate was by Andrew Cowie, who maintained that "The way the whole survey exercise was conducted it was impossible for Git to lose", and that the desired results were plain beforehand from discussion on the #gnome-hackers channel. Cowie also complained about the fact that GNOME contributors to projects that do not use Subversion were not invited to participate, and that the survey required listing preferences for all choices when he would have preferred not to vote for Git, Mercurial, or Subversion at all.

Much of the discussion below Esfahbod's announcement of the results, did quickly come to center on the assumption that a move from Subversion was now inevitable. Some questioned whether GNOME had the resources to devote to such a move, while others volunteered to be part of a task force to plan and implement the move. Another possibility raised was hiring someone to coordinate the change. Still others attempted to rough out the steps needed to move away from Subversion.

However, while most assumed — however reluctantly — that a change to Git would happen, others tried to raise alternatives. Some suggested that each separate GNOME module should be allowed to use its own DVCS, which others argued would discourage new contributors.

Others championed a suggestion raised on John Carr's blog that GNOME use its bzr-playground server and create plug-ins so that developers could use the DVCS software they personally preferred. However, this idea was dismissed by Esfahbod, who in a reply to the discussion about his announcement condemned such "hacks [developed] in house" because of the potentially high-maintenance they might require in the future.

In the end, however, the usefulness of these criticisms and alternate suggestions is limited. As Esfahbod states: "[...] This thread is not about making decisions. This thread is about giving those making the decision input they need to consider. Who makes the decision? Those who actually have to implement, oversee, and maintain any change" — in other words, the GNOME Foundation directors, and the project's release team and system administrators, the ones who commissioned the survey in the first place. The unspoken assumption in the survey appears to be that the move from Subversion is inevitable, and only the details are up for discussion. So far, though, those details have not been finalized, either by consensus or an announcement.

Meanwhile, for those unaffected by the decision, the survey and the resulting commentary provides a seldom-seen insight into the mindset of active members of the GNOME project — and, very likely, of active FOSS contributors in general.

Comments (25 posted)

Page editor: Jonathan Corbet

Inside this week's LWN.net Weekly Edition

• Security: SSL certificates and MD5 collisions; New vulnerabilities in bind, git, java, python,...
• Kernel: 2.6.29 merge window, part 2; An asynchronous function call infrastructure; Who is the best inliner of all?
• Distributions: OpenSolaris 2008.11; NST 1.8.1 pure:dyne GNU/Linux leek&potato; XO-LiveCD Version 090110; Ubuntu Developer Week
• Development: The Valgrind Project releases version 3.4.0, Firebird Roadmap, CWiid update, WX in 2008, new versions of SQLite, Mandriva Directory Server, web2py, lsscsi, Amarok, qBittorrent, GNOME, KDE, GnuPG, SQL-Ledger, Elisa, DSSI, openSUSE csync, Java INI, tig, MyHDL.
• Press: Mark Shuttleworth profiled, The Perl Future, Linux Day Italy, SCO reorganizes and vows to continue fight, Marvell's PXA168 SoC, Chrome for Linux planned, Drupal for medical content entry, December's audio software, Linux support sites, GIMP 2.6.4 review, seven HTML editors.
• Announcements: Fedora fills Board, Friends of GNOME, Qt relicensed to LGPL, PgUS 2008 summary, CadSoft Eagle 5.4, open-source trading platform, Tim Buckley joins rPath, I'm Linux video contest, LayerOne cfp, OSCON cfp, ShakaCon cfp, SyScan cfp, YAPC::EU cfp, Northwest Python Day.

Next page: Security>>