mattdorn.com » technology

Plone i18n: a brief tutorial

mdorn — Fri, 20 Apr 2007 12:47:54 +0000

The following tutorial details the steps I took to internationalize (starting with a Spanish translation) the interface of Plone Translation Hub, a Plone Archetypes-based product.

Contents

Basic Plone i18n with i18ndude
Advanced Plone i18n
- Translating workflow states and transitions
- Accessing the Plone translation machinery directly
Resources

Basic Plone i18n with i18ndude

First, processing your UML model via ArchGenXML (assuming you have i18ndude installed) will produce a file called generated.pot and place it in your product’s i18n directory, which is where all translations must go. Plone makes somewhat customized use of a the GNU gettext architecture for i18n of applications. So generated.pot is a template that contains the labels and text for your Archetypes objects.

Before you can begin translating, though, you’ll need to ensure that the elements in your your page templates that expose text to the user are tagged for translation by the Plone translation architecture. First you want to make sure that you specify the domain element in the template’s root (html) element to correspond to your product:

Next, for field labels, help indicators, etc.–any text that’s visible to your user–you’ll need to apply the i18n:translate attribute:

Enter your name

Once you’ve finished applying this attribute in your page templates, if your product is fairly simple, you’re nearly finished. All you need to do is merge the original PO file from your Archetypes product with your new translatable items from your page templates. Assuming i18ndude is in your path, from your product’s root directory, execute it like so:

i18ndude rebuild-pot --pot i18n/PRODUCT_NAME.pot --create PRODUCT_NAME --merge i18n/generated.pot skins/PRODUCT_NAME/*.pt

Then create an empty PO file for the language(s) you want to create. E.g., for Spanish, you’d want:

touch i18n/PRODUCT_NAME-es.po

Finally, to populate the PO file with the text from the original template so that you can translate it:

i18ndude sync --pot i18n/PRODUCT_NAME.pot i18n/PRODUCT_NAME-es.po

You can now perform your translations with a tool like POEdit.

Advanced Plone i18n

Translating workflow states and transitions

Now, if your product has its own workflow states and transitions, you’ll want to translate those as well. Enabling those translations is not included in the above process. You’ll need to create a separate file with a name reflecting the "plone" domain:

touch i18n/plone-PRODUCT_NAME-es.po

(Note also that the PO file header includes a "Domain" key to which you’ll want to assign the value "plone".)

Essentially you’ll need to edit this file by hand, assigning the msgid key with the original text and the msgstr with your translation:

msgid "Open"
msgstr "Abierto"

msgid "Closed"
msgstr "Cerrado"

Accessing the Plone translation machinery directly

You may have some text in your application that appears neither in your page templates nor in your workflow (obviously a situation that you want to avoid if possible). If so, you may need to access the Plone translation machinery directly, which is best illustrated with an example taken from an External Method used by my product:

from Products.CMFCore.utils import getToolByName
...
translation_service = getToolByName(self, 'translation_service')
msg = translation_service.utranslate(domain='PloneXL8',
                                     msgid='You have taken ownership of this translation.',
                                     default='You have taken ownership of this translation.',
                                     context=self)
msg = msg.encode('utf-8')

Note that you’ll need to add the msgid and msgstr for such into your PRODUCT_NAME.pot and/or PRODUCT_NAME-xx.po file by hand.

Resources

Some helpful links:

Open Source i18n

mdorn — Tue, 17 Apr 2007 11:05:04 +0000

Not so long ago I found myself in the rather embarrassing position of having authored a translation collaboration tool for use by an international user base without having implemented the platform’s (Plone) internationalization (i18n) facilities. That is, I gave the application an English interface and left it at that. Furthermore I began to consider how I might extend this application to enable the more professional users to leverage their existing translation tools, like the industry-standard Trados, rather than being confined to using my application’s Web interface when actually performing a translation (even if a cut-and-paste workaround is available). Correcting the first problem and laying the groundwork for solving the second entailed giving myself a pretty extensive lesson in both open standards and Open Source software for Computer Aided Translation (CAT), which I summarize below.

Contents

Translating software interfaces
Translating documents
Translation Memory (TM)

Translating software interfaces

First, I discovered that there are two aspects of what I’m currently trying to achieve with this tool, both of which have different, but related, technological solutions. First of all, I need translations for the interface of the tool itself. To accomplish this, Plone makes somewhat customized use of a the GNU gettext architecture for i18n of applications. In that model, translations are stored in message catalogs which match original translations with their target language translations contained and which are contained in files with the .po format, which essentially maintains translations through original/translation key-value pairs. In the case of Plone, a utility called i18ndude handles much of the heavy lifing involved in preparing these files for use with your Plone Product, while a PO file editor like POEdit helps you actually perform the translations.

The gettext/PO file approach appears to be the most common solution for translating the interfaces for Linux-based open source projects as well as Web applications (though there are some sui-generis exceptions, such as the Globalize plugin for Ruby on Rails).

So I went ahead and created a Spanish translation for the application, but its current users include translators of French and Portuguese as well, and potentially Italian, Swahili, Greek, and Arabic. I can’t translate the interface into those languages (though I might be able to create a passable Portuguese interface), and I’d like to make it as easy as possible for those who can to help me out. A solution to this problem might be to set up a server tool like Pootle, which lets you visualize progress made on translation of PO files, to help manage the project. Ubuntu/Launchpad’s Rosetta, which is not (yet) open source, appears to be a more sophisticated implementation of the same idea. In conjunction with the server app, I could provide a brief overview of client-side tools that can be used to edit the PO files. I have already mentioned POEdit, but OmegaT, a more ambitious suite of Open Source CAT tools, a also has support for PO files.

Translating documents

So, software interface translation is the first order of translation business for my project. The second issue is the translation of documents, which is the purpose of the application itself. Take the documentation of a software program, for example, with an original document in, say, HTML. A translator could simply translate it free-form, using copy-and-paste and whatever tools she happens to have available. But translation-specific editors like that provided by OmegaT–which supports HTML, OpenOffice, and ODF among several other formats–can help present the text in a way that’s more comfortable for translation, in addition to providing access to Translation Memory tools. As long as the program is capable of exporting the translation to the format of the original (which is presumably what would be reqeusted in most translation jobs), this would appear to be the preferable route.

The problem with this process is that translators are likely to receive documents in any number of formats–the tools they use may be more capable of dealing with some formats than with others, and a separate workflow or collaboration process may need to be defined depending on the type of file. This is the problem that XLIFF, an XML format for translations, attempts to address, by providing a set of elements for tracking information common to translation activity. I don’t go into detail on XLIFF here (and in fact have quite a bit more to learn about it), but Trados and other major translation software vendors already include support for XLIFF, and given its emergence as a standard, any software meant to support translation collaboration should also support XLIFF.

Translation Memory (TM)

Finally, both activities–the translation of software interfaces and the translation of documents–benefit from the use of a Translation Memory (TM) system. POEdit can generate a TM for you by letting you import previously existing PO files based on the language(s) you want to work with. OmegaT generates one for you based on the translations you provide while working on your project, but also allows you to import existing TMs based on the TMX standard.

Pending further investigation, I’m considering complementing Plone Translation Hub with a repository of TMX files for enhanced collaboration. Because it’s Zope/Plone, Tumatxa might provide a good starting point.

Note that the effectiveness of TM relies on a program’s ability to "segment" a source document properly, SRX is the emerging standard for defining segmentation rules.

Updating Plone user roles programatically

mdorn — Mon, 12 Mar 2007 13:35:46 +0000

Today for a site using Plone Translation Hub I had to programatically update the roles for all users who had logged in since March 9th, given that this group was known to be responsible enough to be worthy of the "Reviewer" role. One of the site managers asked that I simply run a "SQL query" to do this–if only it were that easy! Actually it’s not that much harder with the ZODB, but it took me a while to find the "userFolderEditUser" method that I needed to do this, which was frustrating. I ended up simply using a temporary script object placed in the root of the portal:

recent_date=DateTime('2007/03/09')
total=0
recent=0

mship = context.portal_membership
acl_users = context.acl_users
for user in mship.listMembers():
    login_time=DateTime(user.last_login_time)
    roles = list(user.getRoles())
    if login_time.greaterThanEqualTo(recent_date) and 'Reviewer' not in roles:
        roles.append('Reviewer')
        acl_users.userFolderEditUser(user.id, None, roles, user.getDomains())
        recent += 1
    total += 1
print "Recent users: " + str(recent)
print "Total users: " + str(total)
return printed

reStructuredText tools for gedit

mdorn — Thu, 15 Feb 2007 09:20:07 +0000

Over the last couple of days, I put together some reStructuredText tools for gedit, the lightweight text editor for the Gnome desktop on Linux. They include syntax highlighting, some keyboard shortcuts, and an HTML preview feature that I derived from another developer’s earlier work on Markdown support.

This was made possible by gedit’s excellent plugin system, which allows you extend the application via Python.

Gnapsack 0.2 released

mdorn — Wed, 07 Feb 2007 16:03:33 +0000

Version 0.2 of my client for the Backpack Web Services API, Gnapsack, was released today, and now has a new home. The two main enhancements are Windows compatibility and ability to handle multiple lists.

Agile tools in Ruby and Python

mdorn — Thu, 01 Feb 2007 12:51:31 +0000

I spent a few months at the end of last year working on a Ruby on Rails project. While I constantly found myself longing for Python, I did appreciate the extent to which RoR carried certain best practices–among them an MVC architecture and Test-Driven Development–to an extreme, baking them into the framework.

The following table identifies a series of functions relevant to "agile" development and indicates which tools are typically used with which language. I may update this table as more functions or more appropriate tools occur to me.

Function	Tool
()()	Ruby	Python
Continuous integration	Cerberus	buildbot
Test coverage	Rcov	coverage.py
Documentation	RDoc	EpyDoc
Code versioning	Subversion	Subversion
Project management	Trac	Trac
App Deployment	Capistrano *	PythonPaste ?

* Capistrano is language-agnostic. Other tools may be as well.

Trac + Darcs + reStructuredText

mdorn — Thu, 01 Feb 2007 10:50:08 +0000

Edgewall Software’s Trac seems to have become something of a standard for agile management of software projects both within the Open Source community and within closed organizations, and after having the opportunity to use it on a recent project, I can appreciate why. It’s simple to setup and manage, and its self-described "minimalistic approach to web-based software project management" is exactly along the lines of my own philosophy.

For that project I used a default setup with a Subversion repository, and composed Wiki pages using the default Wiki markup. While Subversion is obviously a standard, for personal projects I prefer to use the distributed versioning system Darcs, and I usually insist on doing the majority of my text composition in reStructuredText. So when I found out that there was a Darcs plugin for Trac and that it also supports reStructuredText , I got motivated to get a couple of my own projects up and running. What follows is a brief description of the steps I took to do so.

Contents

1 Get the software
2 Set up your Trac instance
3 Install and configure the Trac plugins
4 Configure Apache and mod_python
5 Configure user authentication and permissions
6 Final notes
- 6.1 Darcs+Tracs bug
- 6.2 reStructuredText in Trac

1 Get the software

I’m assuming your system has fundamentals like Python and Apache installed. In addition:

Trac (I’m using v0.10.x)
Darcs (I’m using v. 1.0.8)
sqlite2 (I personally had problems with version 3 and the Darcs plugin on a Fedora Core 4 system, which have been reported to the project maintainers)
Trac plugins
Python docutils (for reStructuredText support)
SilverCity (for syntax coloring of your code)
Python setuptools (for working with Trac plugins)
Apache mod_python

Note that the items that are not linked above should be available via the package manager of most modern Linux systems. You’ll want to install everything except that Trac plugins before moving on to the next step.

2 Set up your Trac instance

Create a directory where you want to maintain your Trac projects and use the trac-admin command to create the instance and associate it with your Darcs repository:

trac-admin /path/to/MyProject initenv "My Project" sqlite:db/trac.db darcs /path/to/repo /usr/share/trac/templates/

The instance is successfully created, but you’ll get a warning that Trac is not supported. That’s because you haven’t setup the plugins yet.

3 Install and configure the Trac plugins

Download the source for each plugin into a temporary directory. Trac handles plugins in the Python "egg" format. So in the source directory of each, run the following command:

python setup.py bdist_egg

That will create a file with a .egg extension in the dist directory of the plugin you’ve downloaded, which you can now copy to the plugins directory of your new Trac instance. Be sure to copy all three of them.

Now you need to update your Trac instance to understand the Darcs plugin:

trac-admin /path/to/MyProject upgrade
trac-admin /path/to/MyProject resync

Finally, add the following to the conf/trac.ini file of your instance, under the [components] section:

[components]
trac.web.auth.LoginModule = disabled
acct_mgr.web_ui.LoginModule = enabled
trac.ticket.report.* = disabled

You may wish to test your instance now, using Trac’s built-in Web server, and browsing to localhost:8000:

tracd --port 8000 /path/to/MyProject

or:

/usr/sbin/tracd --port 8000 /path/to/MyProject

Make sure you can browse your Darcs repo to confirm that the Darcs plugin is working. If you’re not seeing syntax highlighting, check to make sure SilverCity has been installed correctly.

In this setup, however, we’ll ultimately be using Apache and mod_python to server the instance, and a few extra steps are needed.

4 Configure Apache and mod_python

Trac’s documentation on mod_python is mostly adequate, with one exception. You need to identify a cache directory for your instance’s egg plugins. In your Location directive, composed in accordance with that documentation, including the following line should be sufficient:

SetEnv PYTHON_EGG_CACHE /tmp/trac-eggs

For the sake of completeness, the Location directive inside the VirtualHost setup for your site should look something like this, if you have multiple Trac instances:


  SetHandler mod_python
  PythonHandler trac.web.modpython_frontend
  PythonOption TracEnvParentDir /path/to/my/trac/projects
  SetEnv PYTHON_EGG_CACHE /tmp/trac-eggs
  PythonOption TracUriRoot /projects

Finally, the user under which Apache is run will need read and write access to the following directories in your instance:

db
attachments
log

5 Configure user authentication and permissions

By default, Trac allows the anonymous user access to nearly everything, so you’ll want to make some changes. Fortunately the Web Admin and Account Manager plugins make this fairly easy to do. Before you create any user accounts, I found it was easiest simply to start by adding the TRAC_ADMIN permission to the anonymous group via the command line admin tool:

trac-admin /path/to/projenv permission add anonymous TRAC_ADMIN

Once I have access to the "Admin" button in the Web interface as anonymous I performed the following:

In Accounts > Configuration, identify the path to the HtPasswdStore. Ideally this will probably be a location within your instance. The Web server user will need to be able to write to the location.
In Accounts > Users, add a user. Note that you could also add a user by registering yourself via the "Register" link that should appear after having installed the Account Manager plugin.
In General > Permissions, create an "admin" group in the "Add Subject to Group" box.
In the same box, add the newly created user to the admin group.
Now, remove any "CREATE" or "DELETE" or write-related permissions, especially TRAC_ADMIN, from the anonymous subject.
Using the "Grant Permission" box, add anything appropriate for your setup to the "authenticated" subject. Make sure the TRAC_ADMIN permission is assigned to the newly created admin group.
Add any other groups appropriate to your project (e.g., developers, who would probably have access to everything except TRAC_ADMIN)

6 Final notes

It’s up to you to configure your ticket system with the appropriate values for components, milestones, versions, etc.

6.1 Darcs+Tracs bug

Note that the only major bug with the Darcs plugin that I’ve found is the broken "view changes" button. That will need to be fixed before Trac+Darcs is a completely legitimate product.

6.2 reStructuredText in Trac

Composing your Wiki pages in RST simply requires you to enclose your text with the following syntax:

{{{
#!rst

Your text goes here.

}}}

That’s an inconvenience, however minor, and I think a great Trac enhancement (which too my knowledge has not yet been suggested) would be the ability to set a default markup syntax in the configuration file.

Getting AWStats to show Plone-authenticated users

mdorn — Sun, 15 Oct 2006 19:12:36 +0000

I’m using AWStats to track usage on a Plone site that’s essentially a portal for collaborative document translation. That means that most users of the site need to login to do anything useful, and I want to see who’s logging in along with the rest of the stats on my site.

For hosting the site, I’m using the recommended setup which uses Zope’s Virtual Host Monser to route requests and responses between Apache and the Zope application server appropriately. I.e., I include the following two key lines in my Apache directive:

RewriteEngine On
RewriteRule ^/(.*) http://localhost:8080/VirtualHostBase/http/%{SERVER_NAME}:80/MyApp/VirtualHostRoot/$1 [NC,P,L]

Since I’m going thru Apache, I can record requests in log files as with any other site. Unfortunately, Zope/Plone does not pass information about the users who authenticate via its own system to the HTTP headers. Taking a cue from this thread, I came up with the following solution. It’s tested on Plone 2.1.3, but probably won’t work on Plone 2.5, given the latter’s use of Pluggable Authentication Service.

First, I found a place in the Plone page rendering mechanism that gets executed on each page view. I suppose the place that I chose was somewhat arbitrary, but in the "authenticate" method of the file GroupUserFolder/GroupUserFolder.py, I set a header called X-PloneUser, as shown in the following patch, available for your use:

1020,1022d1019
<         # PATCH FOR TRACKING AUTH USER IN APACHE LOGS --mdorn:
<         if name is not None:
<             request.RESPONSE.setHeader('X-PloneUser', name)

In my Apache configuration, I had to change LogFormat from "combined" to something more specific, and reference the label in the CustomLog:

LogFormat "%h %l %{X-Ploneuser}o %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" plone
CustomLog "|/usr/local/sbin/cronolog /home/httpd/MYDOMAIN/logs/%m-%Y/access_log" plone

To see authenticated users in your AWStats configuration file, you’ll need to change the default value for that setting:

ShowAuthenticatedUsers=1

That’s it. Next time AWStats processes your logs, assuming you had visits from authenticated users, you’ll now see them in the appropriate location on your AWStats report.

Automating AWStats configuration for multiple domains

mdorn — Sun, 15 Oct 2006 18:34:52 +0000

Like most Web sites, this site shares server space with a number of other domains. When I recently undertook to set up AWStats after years of not knowing anything about what kind of traffic my personal sites were getting, I figured it would probably be relatively easy to make it so that anybody else with an account on the box who also wanted stats could avoid the hassle.

With a bit of help from another tutorial, I came up with the following solution.

First of all, I installed the AWStats package on my server’s Fedora system via yum. With that installation, I ended up with AWStats’ default config file in /etc/awstats/awstats.model.conf.

This setup needs a separate AWStats configuration file for each domain, so this default file is the model from which all others will be generated. For that reason, I needed to tweak a few of the default values to work the script described later:

LogFile="/home/httpd/$USERNAME/logs/%MM-0-%YYYY-0/access_log"
SiteDomain="$DOMAIN"
HostAliases="$ALIASES"
DirData="/home/httpd/$USERNAME/awstats"
AllowAccessFromWebToFollowingAuthenticatedUsers="$USERNAME"

(I should also note that I enabled the GeoIP plugin in this file, but that’s a subject for a different post.)

Now, on the Apache side of things, you need to decide on a standard location for the Apache log files for all the accounts. In my case, that’s /home/httpd/ACCOUNT_NAME/logs. (In that directory, the setup described below will generate directories named by month and year (like so: MM-YYYY), and put the log file there.) The Apache virtual host directives of course will need to specify that location. Since I don’t actually know or have any communication with my neighbors who share the box, I’ll leave that to them, but my new directives look like this:

CustomLog "|/usr/local/sbin/cronolog /home/httpd/DOMAIN/logs/%m-%Y/access_log" combined

Where DOMAIN is the name of the directory that holds the Web files (htdocs, logs, etc.) for a given domain on the server.

I’ve also installed cronolog as an easy way to rotate logs. As you can see the Apache config pipes to that program. That’s not strictly necessary for this setup, but if you want to do it differently (e.g., with logrotate), you’re on your own.

Making use of this information, anytime you want AWStats set up for a new domain, you can use the following script, which is named /usr/local/sbin/awstats_script.sh in my setup, and which was adapted from another tutorial , mentioned previously:

#!/bin/bash
echo "Enter the username:"
read WUSERNAME

ACCESS_FILE="/etc/awstats/awstats.pwd"
STAT_DIR="/home/httpd/$WUSERNAME/awstats"

echo "Enter the password:"
read PASSWORD

echo "Enter the main domain:"
read DOMAIN

echo "Enter aliases separated by space:"
read ALIASES

# Create the statistics directory
if [ -d $STAT_DIR ]; then
    echo "Statistics dir already exist"
else
    mkdir $STAT_DIR
fi

# Create the virtual host awstats.conf
cat /etc/awstats/awstats.model.conf | \
sed -e "s/\\\$DOMAIN/$DOMAIN/g" | \
sed -e "s/\\\$USERNAME/$WUSERNAME/g" | \
sed -e "s/\\\$ALIASES/$ALIASES/g" > \
"/etc/awstats/awstats.$DOMAIN.conf"

# Add user/password to password file
if [ -e $ACCESS_FILE ]; then
    /usr/bin/htpasswd -bm $ACCESS_FILE $WUSERNAME $PASSWORD
else
    /usr/bin/htpasswd -bm -c $ACCESS_FILE $WUSERNAME $PASSWORD
fi

The "username" requested is the name found in the /home/httpd directory for which you want to create the site. This will also be the username needed to login (via Apache authentication) to view the stats. The "password" then requested is for that same Apache authentication (will be stored in /etc/awstats/awstat.pwd). The "main domain" is what people use to access the site, (e.g., www.DOMAIN.com), but you’ll also then be asked to include any subdomains that log to the same Apache log file (at least if you want AWStats to process them).

After the setup is complete, you can check out your stats by going to: http://www.DOMAIN.com/awstats/awstats.pl

AWStats knows which config file (automatically generated) it needs by reading the domain from the URL, but you can also view other domains by appending "?config=www.whatever.com" to the URL. That config file is left in /etc/awstats, and the individual AWStats DB file that results from processing the logs is left in each user’s directory in /home/httpd/DOMAIN/awstats

While each user will be able to HTTP-authenticate against any other user’s domain, the values in the AllowAccessToWebToFollowingUsers variable will prevent users from being able to view one another’s stats.

Finally, note that your AWStats installation may have included a cron task to update the AWStats data in /etc/cron/hourly or other. In any case, you’ll want AWStats to automatically process your logs on an at least daily basis via the included script /usr/share/awstats/tools/awstats_updateall.pl.

Gnapsack 0.1.0 released

mdorn — Fri, 28 Apr 2006 18:51:29 +0000

I have just released Gnapsack, a desktop client for Linux that uses the Backpack API. Check out the site for info, screenshots, etc. This is an open source (GPL) project.

While it only runs on Linux currently, I’m hoping to get someone to help me out with a Windows port. It uses the Python bindings to the GTK widget set. A good example of a PyGTK app that has been ported to Windows is the Griffith film collection manager.

I may unwittingly be using some non-standard Python libraries–if any Linux users can give it a try and let me know if I should add items to the dependencies list, I’d appreciate it.

Anyone interesting in contributing to this project in general, please let me know, and I’ll get you oriented.