Development [LWN.net]

Translating software with Pootle

August 12, 2009

This article was contributed by Sayamindu Dasgupta

Pootle is Free/Open Source, web-based software, aimed at making the process of translation of other software simpler. As a typical translation team's workload increases with each new software release, it becomes more and more important to have a smooth workflow, not only for managing the translations, but also for managing the translators as well. Simultaneously, to ensure that translators from all kinds of backgrounds are able to participate, the entire process needs to be kept as accessible and easy as possible. Pootle's goal is to make this a reality.

Pootle is written in Python and is used by FOSS projects such as OpenOffice.org, One Laptop per Child/Sugar Labs, LXDE, Sahana, GNU Mailman, etc. A list of live Pootle installations is maintained on the Pootle wiki. In its backend, Pootle utilizes the Translate Toolkit, which provides an API for a comprehensive set of localization related functions, along with support for multiple translation file formats such as PO, XLIFF, QT .ts, CSV, and Mozilla .properties. Using this backend, Pootle offers a web-based interface for managing translators, adding, modifying and suggesting translations, and support for connecting to a number of version control systems (CVS, SVN, Git, Mercurial, etc).

In a typical Pootle installation the server configuration points to the po/ subdirectory of a project (or an appropriate symlink). The project may be a checkout/clone of a remote repository or standalone. In the former case the language team lead (or other authorized team members) can commit (and push) translations to the remote, upstream repository via the web interface. If the PO files are not part of a repository, the integration step is usually carried out manually. All the PO files (or equivalent translation files) in the po/ directory are exposed via the web UI, and the messages in the files are presented as distinct units to the translators, who can translate them (or edit the translations) using the browser.

Pootle's web-based translation interface lowers the barrier to entry considerably, this is crucial for ensuring success for translation teams (especially those with limited resources and volunteers). Translators often come from non-technical backgrounds, so the web-interface, by abstracting away the underlying complexity of multiple version control systems and the gettext toolchain, makes their work considerably easier.

The web-based interface, despite simplifying things with its ease of use can sometimes exclude people from participating. One barrier is that it relies on continuous Internet access, which may be non-existent or prohibitively expensive in many regions of the world. Fortunately, Pootle provides a feature which allows translators to download PO files, translate offline, and re-upload the files.

This "offline-translation" feature can be highly useful for translation "sprints", where a group of volunteers gather together, translate a set of downloaded PO files using offline tools such as Poedit or Virtaal, then upload back the files to the Pootle server.

However, the overall ease of use brings up yet another potential problem. With a large number of translators from various backgrounds participating in the translation process, quality issues invariably crop up. Such issues can be divided into two broad categories:

Mistranslations: These can be either plain "wrong" translations, or errors like incorrectly typed format placeholders or an incorrect number of escape sequences (e.g. '\n'). This can be partially addressed by having access control rules in place. Pootle has the provision for different levels of access, where a user can be assigned various rights on the basis of the user's experience level. Using this feature, a user can be allowed to perform any combination of tasks such as translating, suggesting possible translations, reviewing suggested translations, committing translations, etc.

To ensure that the quality level of a new contributor is up to the mark, many language team coordinators choose to set the default access level to "suggest". When the initial set of suggestions are found to be satisfactory, the access level can be increased to "translate", "review", or even higher.

Another way of ensuring high quality translation is to use the built-in "checks" feature offered by the software. Currently there is a set of 47 checks in Pootle which test for common errors in translations. Some of these tests, like the capitalization check are language and writing system specific. But others, like the check for translated file paths and function names (normally file paths or function names are kept untranslated) are applicable universally and can be quite useful.

For translators not sufficiently familiar with English (messages to be translated are almost invariably in English), the ability to view reference translations in a third language often helps. Support for this viewing of an alternate language was added in the last stable release of Pootle (version 1.2). For example, this feature was found to be very useful in OLPC's South American deployments, where translators for languages like Aymara and Quechua wanted to see Spanish translations while translating, to better understand the meaning of the original English strings.

A combination of the above three features, if used properly, can reduce mistranslations to a large extent.
Inconsistency: In this commonly occurring problem, two or more translators may translate the same term differently. For example, for a certain term, one translator might want to keep an English transliteration, while another may want to use the literal meaning of the term. This almost certainly confuses the end user, and ensuring consistency is important. To address this problem, Pootle provides a glossary, or terminology feature, where a single set of translations for commonly used terms can be stored. During translation, the appropriate entry from this set is provided as a suggestion whenever the translator comes across a message containing a term from the glossary. This particular feature can also be useful if there is government or language-body sanctioned terminology for computer-related vocabulary.

Though deciding on and creating a glossary of terms manually can be difficult for projects with a large string count, the Translate Toolkit provides a command line tool called poterminology to automatically produce a list of commonly occurring terms from a set of PO or POT (PO Template) files.

With the features outlined above, Pootle can make the life of translators in any FOSS project considerably easier. However, from the author's experience at OLPC/Sugar Labs, there can been a few problems along the road as well. Any project thinking of adopting Pootle should ideally factor in these issues in before taking the plunge:

Speed issues: Pootle can be quite slow during certain operations involving large PO files (as an example, the largest of the PO files served by the OLPC/Sugar Labs Pootle installation has around 4400 strings). These operations that can make Pootle utilize 100% CPU on the hosting server for a significant amount of time. Operations such as merging two large PO files (which usually happens during uploads), updating a set of PO files from the corresponding set of updated reference POT files, or searching through large files for untranslated strings, are examples of potentially lengthy tasks.

Newer versions of Pootle (and the Translate Toolkit) have seen the introduction of sqlite-based databases for storing translation statistics and the use of Lucene/Xapian based indexing of PO files. These changes have significantly improved the search performance.
Communication: Pootle itself does not provide for any mode of communication within teams or between site-administrators and teams. This can be addressed partially by using a mailing list for each translation team, as well as a special "all-hands" mailing list for general announcements and reminders. However, even if mailing lists are used, it becomes increasingly difficult to get in touch and coordinate with individual translators and team leads as the translation community scales up. This can become an especially pressing issue toward the end of a release cycle; often translation leads need to be reminded individually about committing the work they have completed so far.

On the other hand, the Pootle developers seem to be aware of the urgency of the issue, and there is currently a page in the wiki outlining the various use-cases for this feature. Some of the required features have already been implemented in Pootle's SVN trunk.
Error checking: The error checking feature in Pootle needs to invoked manually (via the "checks" system mentioned before). This means that there is always a chance that wrongly formatted PO files may get introduced into the software's source tree, causing the build to break in some cases. Of course, this is not an issue if the language team lead always ensures that at least the the critical "checks" pass for the PO file before a commit is made.

Despite the above weaknesses, from an overall perspective Pootle can be a useful component of any FOSS project's localization arm. In fact, it is not only the translators who benefit from the features that Pootle offers, developers only have to grant repository access to one user account (the one used by Pootle) to take care of all translations. The process of providing the requisite repository access to each translator or language team has been a major bottleneck for quite a few FOSS projects in the past.

As of August 2009, Pootle has been undergoing significant changes in both code and visual design. Along with a switch to the Django web framework, the user interface has changed radically. Now, overall statistics for each project and language are viewable from the first page and UI changes are visible in the individual project/language pages. Among the other user-visible changes that are supposed to ship with Pootle 1.3, notable features include support for multiple alternate languages and initial implementation of intra-team communication methods. Also, with the shift to Django, which is much more popular compared to the older jToolkit framework, it should become easier for external developers to extend and customize Pootle.

The code for Pootle 1.3 (pre-release) can be downloaded from the SVN repository of the project. It is pretty much usable in its current form, though probably not recommended for production use. However, the project does need more help with testing, and reports about any bugs in the latest version from SVN are welcome.

With the impending release of Pootle 1.3, along with the fast pace of growth of other similar tools (Transifex, as well as the recently open-sourced Rosetta), things certainly do look bright for FOSS translation tools. In the next few months, we should be able to see significant improvements in the way Open Source translation is done, and through all these changes, the general translation workflow for FOSS will be greatly improved.

Comments (10 posted)

System Applications

Database Software

Firebird 2.5 Beta 2 kits released

Version 2.5 Beta 2 of the Firebird DBMS has been announced. "The Firebird Team is pleased to release the second Beta of the new Firebird 2.5 version for field-testing. 32-bit and 64-bit builds for Linux, Windows and MacOSX Intel are available."

Comments (none posted)

PostgreSQL Weekly News

The August 9, 2009 edition of the PostgreSQL Weekly News is online with the latest PostgreSQL DBMS articles and resources.