mattdorn.com » plone

Plone i18n: a brief tutorial

mdorn — Fri, 20 Apr 2007 12:47:54 +0000

The following tutorial details the steps I took to internationalize (starting with a Spanish translation) the interface of Plone Translation Hub, a Plone Archetypes-based product.

Contents

Basic Plone i18n with i18ndude
Advanced Plone i18n
- Translating workflow states and transitions
- Accessing the Plone translation machinery directly
Resources

Basic Plone i18n with i18ndude

First, processing your UML model via ArchGenXML (assuming you have i18ndude installed) will produce a file called generated.pot and place it in your product’s i18n directory, which is where all translations must go. Plone makes somewhat customized use of a the GNU gettext architecture for i18n of applications. So generated.pot is a template that contains the labels and text for your Archetypes objects.

Before you can begin translating, though, you’ll need to ensure that the elements in your your page templates that expose text to the user are tagged for translation by the Plone translation architecture. First you want to make sure that you specify the domain element in the template’s root (html) element to correspond to your product:

Next, for field labels, help indicators, etc.–any text that’s visible to your user–you’ll need to apply the i18n:translate attribute:

Enter your name

Once you’ve finished applying this attribute in your page templates, if your product is fairly simple, you’re nearly finished. All you need to do is merge the original PO file from your Archetypes product with your new translatable items from your page templates. Assuming i18ndude is in your path, from your product’s root directory, execute it like so:

i18ndude rebuild-pot --pot i18n/PRODUCT_NAME.pot --create PRODUCT_NAME --merge i18n/generated.pot skins/PRODUCT_NAME/*.pt

Then create an empty PO file for the language(s) you want to create. E.g., for Spanish, you’d want:

touch i18n/PRODUCT_NAME-es.po

Finally, to populate the PO file with the text from the original template so that you can translate it:

i18ndude sync --pot i18n/PRODUCT_NAME.pot i18n/PRODUCT_NAME-es.po

You can now perform your translations with a tool like POEdit.

Advanced Plone i18n

Translating workflow states and transitions

Now, if your product has its own workflow states and transitions, you’ll want to translate those as well. Enabling those translations is not included in the above process. You’ll need to create a separate file with a name reflecting the "plone" domain:

touch i18n/plone-PRODUCT_NAME-es.po

(Note also that the PO file header includes a "Domain" key to which you’ll want to assign the value "plone".)

Essentially you’ll need to edit this file by hand, assigning the msgid key with the original text and the msgstr with your translation:

msgid "Open"
msgstr "Abierto"

msgid "Closed"
msgstr "Cerrado"

Accessing the Plone translation machinery directly

You may have some text in your application that appears neither in your page templates nor in your workflow (obviously a situation that you want to avoid if possible). If so, you may need to access the Plone translation machinery directly, which is best illustrated with an example taken from an External Method used by my product:

from Products.CMFCore.utils import getToolByName
...
translation_service = getToolByName(self, 'translation_service')
msg = translation_service.utranslate(domain='PloneXL8',
                                     msgid='You have taken ownership of this translation.',
                                     default='You have taken ownership of this translation.',
                                     context=self)
msg = msg.encode('utf-8')

Note that you’ll need to add the msgid and msgstr for such into your PRODUCT_NAME.pot and/or PRODUCT_NAME-xx.po file by hand.

Resources

Some helpful links:

Updating Plone user roles programatically

mdorn — Mon, 12 Mar 2007 13:35:46 +0000

Today for a site using Plone Translation Hub I had to programatically update the roles for all users who had logged in since March 9th, given that this group was known to be responsible enough to be worthy of the "Reviewer" role. One of the site managers asked that I simply run a "SQL query" to do this–if only it were that easy! Actually it’s not that much harder with the ZODB, but it took me a while to find the "userFolderEditUser" method that I needed to do this, which was frustrating. I ended up simply using a temporary script object placed in the root of the portal:

recent_date=DateTime('2007/03/09')
total=0
recent=0

mship = context.portal_membership
acl_users = context.acl_users
for user in mship.listMembers():
    login_time=DateTime(user.last_login_time)
    roles = list(user.getRoles())
    if login_time.greaterThanEqualTo(recent_date) and 'Reviewer' not in roles:
        roles.append('Reviewer')
        acl_users.userFolderEditUser(user.id, None, roles, user.getDomains())
        recent += 1
    total += 1
print "Recent users: " + str(recent)
print "Total users: " + str(total)
return printed

Getting AWStats to show Plone-authenticated users

mdorn — Sun, 15 Oct 2006 19:12:36 +0000

I’m using AWStats to track usage on a Plone site that’s essentially a portal for collaborative document translation. That means that most users of the site need to login to do anything useful, and I want to see who’s logging in along with the rest of the stats on my site.

For hosting the site, I’m using the recommended setup which uses Zope’s Virtual Host Monser to route requests and responses between Apache and the Zope application server appropriately. I.e., I include the following two key lines in my Apache directive:

RewriteEngine On
RewriteRule ^/(.*) http://localhost:8080/VirtualHostBase/http/%{SERVER_NAME}:80/MyApp/VirtualHostRoot/$1 [NC,P,L]

Since I’m going thru Apache, I can record requests in log files as with any other site. Unfortunately, Zope/Plone does not pass information about the users who authenticate via its own system to the HTTP headers. Taking a cue from this thread, I came up with the following solution. It’s tested on Plone 2.1.3, but probably won’t work on Plone 2.5, given the latter’s use of Pluggable Authentication Service.

First, I found a place in the Plone page rendering mechanism that gets executed on each page view. I suppose the place that I chose was somewhat arbitrary, but in the "authenticate" method of the file GroupUserFolder/GroupUserFolder.py, I set a header called X-PloneUser, as shown in the following patch, available for your use:

1020,1022d1019
<         # PATCH FOR TRACKING AUTH USER IN APACHE LOGS --mdorn:
<         if name is not None:
<             request.RESPONSE.setHeader('X-PloneUser', name)

In my Apache configuration, I had to change LogFormat from "combined" to something more specific, and reference the label in the CustomLog:

LogFormat "%h %l %{X-Ploneuser}o %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" plone
CustomLog "|/usr/local/sbin/cronolog /home/httpd/MYDOMAIN/logs/%m-%Y/access_log" plone

To see authenticated users in your AWStats configuration file, you’ll need to change the default value for that setting:

ShowAuthenticatedUsers=1

That’s it. Next time AWStats processes your logs, assuming you had visits from authenticated users, you’ll now see them in the appropriate location on your AWStats report.

Optimizing Plone performance with caching

mdorn — Fri, 09 Sep 2005 15:07:01 +0000

NOTE: This document is something of a brain-dump. I’m still experimenting with various caching techniques, and would like to organize it somewhat better, so it’s currently a work in progress.

I’ve long been a big fan of the Plone CMS, which powers this very Web site. Any ambivalent feelings I may have had about it have generally derived from its sometimes brutal tendency to hog resources. Throw in some Zope- and ZWiki-related memory leaks while you’re trying to run this thing on a machine with only 400 MB of RAM and you’re going to run into problems (this very scenario was the source of major headaches for me over the last year).

But Plone is a powerful, elegantly-designed, object-oriented system. These attributes don’t come for free–they can require heavy-duty hardware resources. When you don’t necessarily have them, the trick is to find a workaround.

In this case, that workaround is server-side caching. That’s something I’ve known for a long time, but for whatever reason, its implementation remained more or less a mystery to me despite having read up on it in a few different sources, The Definitive Guide to Plone among them. It wasn’t until I created this site, refamiliarizing myself with Plone, and then began to reinvestigate the subject that it finally clicked in a way that made me feel silly for not being able to solve this problem in short order on the sites on that poor, memory-challenged server. Revisiting the Definitive Guide and reading through a tutorial on plone.org got me to where I wanted to be. One of the fundamental realizations I made was that the only thing the Plone HTTP Cache Manager really does is apply a cache-related headers to server reponses so that Apache, if in fact it’s being used, can do something with them. It doesn’t do any caching itself at all, unlike the RAM Cache manager (more on that below).

So here’s what I did:

I’m running Plone 2.1, fronted by Apache 2.0 on the server. I knew that what I needed to do to cut back on unnecessary performance hits was to keep requests from ever getting to the Zope server to begin with–i.e., those requests would need to grab a copy of the resource from a cache. For most sites, this one included, anonymous users of the site don’t really need to see brand spanking-new content more than once per hour, so the idea would be that each resource on a site is cached for up to an hour: the first user to grab a page at, say, 5:45 PM will get it fresh from the Zope server, while any subsequent user will get a static copy of that content from the caching server, rather than from the Zope DB, until 6:45 PM, at which point it may have changed, and a new copy will be retrieved from Zope upon the next request for it.

(Incidentally, here’s another consideration to keep in mind: while a listing of recent content might change a few times per day on this site, an individual document will almost never change. If you have the ability to distinguish between these types of pages for caching purposes, it may be worthwhile to do so–i.e., you could expire the content listing every hour, but expire an individual content page once per day.)

Anyway, I think part of my previous problem involved the notion that I had that any kind of caching would necessarily have to involve learning yet another major server-side application, the Squid proxy server. But as it turns out both Apache 1.3 and Apache 2.0 include proxy server caching capabilities. With mod_cache, and the new modules mod_disk_cache and mod_memory_cache, Apache 2.0 can either manage cached items on disk or in memory. Because memory seems to be the resource that’s in short supply in most server environments I find myself in, I stick with mod_disk_cache.

To enable caching on the Apache side, what I ended up with was an Apache entry that looks something like this:


ServerName www.mysite.com
RewriteEngine On
RewriteRule ^/(.*) http://localhost:8080/VirtualHostBase
   /http/mysite.com:80/mdorn/VirtualHostRoot/$1 [NC,P,L]
CacheRoot "/var/cache/apache2/proxy"
CacheEnable disk /
CacheSize 10000
CacheDefaultExpire 1

Essentially I’m identifying and enabling a cache directory that will accommodate 10 MB of items. I believe that CacheDefaultExpire 1 applies a one hour expiration policy to any item that has been signaled for caching, but without specifying a time period. That’s probably irrelevant for Plone caching. There are several other related directives that I don’t include here because after reading up a bit in the Apache docs, the default values seem fine for my purposes.

Once you’ve got Apache running with these new directives, go into your default HTTP Cache Manager in your Plone site. Now, for anything that you want to cache that’s in the ZODB (e.g., in the "custom" directory of portal_skins) you’ll need to undertake the tedious process of associating those items with the cache object in the "Associate" tab. But if the items that you want to cache are filesystem objects (and ideally they will be), you’ll need to create a ".metadata" file associating them with your HTTPCache object. For example. if you have a page template called "my_template.zpt", you’d create a file in the same directory called "my_template.zpt.metadata" with the following contents:

[default]
title = My Page Template
cache = HTTPCache

You can "associate" them in the ZMI, but your settings are lost when Zope is restarted.

It’s also worth noting here that there are probably some objects that you want to cache always. These would probably include images, JavaScript and CSS. By default, many stock Plone objects of these types have metadata files to do just that, but it may make sense simply to add the following directives to your Apache entry to avoid having to manage this in Plone at all:

ExpiresActive On
ExpiresByType image/gif A3600
ExpiresByType image/png A3600
ExpiresByType image/jpeg A3600
ExpiresByType text/css A3600
ExpiresByType text/javascript A3600
ExpiresByType application/x-javascript A3600

By default, HTTPCache is set to expire all your cacheable items 3600 seconds (i.e., one hour) after first access. As I mention above, it would be nice to be able to distinguish between content types and vary that number accordingly. By creating a caching policy (as per page 452 of the Definitive Guide), you can apparently do exactly that, as well as make other kinds of distinctions using any Python expression, but I think I will start with a global expiration time, of any content that I happen to be caching, of one hour. (Note: I couldn’t get this to work properly by following The Definitive Guide Example anyway.)

In order to test that things were working, before implementing caching, I ran Apache benchmark (ab) like so:

/usr/sbin/ab -n 100 http://mattdorn/

which sent 100 requests to the server–each one went to get the content from Zope, and it took nearly a minute. Afterward, I associated the main template that’s used by my home page, which happens to be atct_topic_view, with the HTTP cache. Once I thus had caching implemented, the same benchmark was nearly instantaneous, after the first page access, as you’d probably expect, given that it’s just Apache grabbing some static files out of a cache directory at this point.

I also changed the interval to 30 seconds, and issued requests within and after that time period while tailing $INSTANCE/log/Z2.log to see whether requests were being received by Zope appropriately (i.e., only after the interval had expired). It, too, worked as expected.

So what if your site is a little more dynamic than this one, and you want to be able to, say, keep one of the portlets updated in real time, while you’re fine with the content in the main column only being updated every hour? Well, that requires a somewhat different strategy that involves using the RAM cache (which has nothing to do with HTTP headers at all), and probably some reprogramming of your site. Additionally, you won’t get nearly the performance advantages you realize by the global HTTP caching strategy. The Definitive Guide provides a very good example of RAM Cache use by pointing to the author’s site, ZopeZen.org. On the home page of that site, a Python script generates the Web log entries, with a tally of comments for each entry. That’s a quite expensive call, as it turns out. The site’s owner decides he can live with this listing only being updated every 30 minutes, and so he assigns the script that produces this information to the RAM cache, achieving a five-fold performance advantage. I considered trying this method by assigning my "folder_contents_custom" template, which controls the portion of the page that displays my Web log entries (i.e., it’s called by atct_topic_view), but as far as I can tell, only Python scripts and External Methods can be used with a RAM cache. I suppose I could reprogram my site to do all of the heavy lifting within a Python script rather than within the ZPT if I really needed this.

One final thing to note is that by default, Plone’s HTTP Cache object doesn’t send cache headers when you’re authenticated. That’s a good thing, because when you’re editing content, you implicitly need server responses to be fully dynamic. So content editing sessions will be the only time that your server is put under the standard stress that a Plone site is capable of inflicting.

In retrospect, my discoveries and conclusions here are consistent with a report that I myself commissioned, but in order for me to really understand what was going on, I had to have a real-life scenario to play with. Guess that’s not too surprising.

Unfortunately, the caching procedures described here have not been implemented on this site as of this writing, because I do not adminster the box that it runs on. I’m sure I’ll get it done in short order.

Launching the site

mdorn — Tue, 06 Sep 2005 04:17:36 +0000

Welcome to my third or fourth attempt at a site resembling a Web log! This one runs on a state-of-the-art Python-based, open source Content Management System called Plone. Certainly far more more firepower than any Web log needs, but I’m thinking that using this as a platform will have spillover effects that create learning opportunities for messing around with other technologies. Also, I can no longer imagine composing documents in anything else other than Restructured Text, which is native to Plone (though I’m sure I could have modified any simpler Web log software to let me use that). Probably the biggest drawback I can see is the fact that the main CSS file is 111 KB. Even with a fairly fast connection, first accesses to this site will be unacceptably slow. Once the stylesheet is cached on the client side, it’s fine, but I’m worried about situations where caching may not be taking place.

Anyway, probably the biggest deal (and a big motivator for me to get a site going again), is being able to use this great skin, a Product called "Plix"–look for it on the Plone site.

The first release of this site consists of one folder to contain all the content, and three "smart folders" to display the content on the basis of its subject matter. "Smart folders" used to be called "topics" in Plone, and essentially they’re canned queries that enable you to set a series of criteria for filtering content by content type, location, subject, etc. In Plone 2.1, you can control the order, forward or reverse, in which the results are shown, based on various criteria including creation date, which is what I use here. Another nice aspect of Smart Folders is the built-in batch control, where you can select how many records to show on a page. In my case, I’ve got it set up for 5.

I wanted this site to basically consist of two Web logs, separated because the different content matter (information technology and international relations) is unlikely to be of interest to the same audience. In fact what I’ve decided to do is let the home page mix the two types by simply being a running, chronological log, while maintaining them separate via the "tech" and "ideas" part of the site’s URL. This reduces the utility of the home page, I suppose, and is a policy that will probably will be revisited.

Rather than using one of the Blog products available for Plone (e.g., Quills), it struck me as best to use a tried-and-true standard Plone content type, the "News" item. All I needed to do was modify the template that displays the items ("folder_listing" from "plone_templates") to be more Blog like, which for me meant to display the entire body of the News item on the pages that serve as the Web log. Note that I also sometimes post "Page" content, which tends to be longer, more polished content, like an essay. I conditionalize on the content type to only show the item description if the item is something other than News. Because my Plone stills had become rusty from lack of use, these issues were a little more complicated than I’d thought, but in retrospect involved some pretty simple changes. I also changed the ATContentTypes template "atct_topic_view" to use my own "folder_listing_custom" customization of "folder_listing", so that the normal Plone "folder_listing" template is used for any non-bloggish folders.

As soon as I get a chance to turn my modifications into a filesystem product, I’ll post the results of that effort in the "Open" section of this site.

Some other changes that were made:

Language

For now, there will be no multilingual content on the site, so I wanted to override the Plone default behavior of translating navigation elements into the browser’s language. For this I installed Plone Language Tool (which now comes with Plone), and disabled "Use browser language request negotiation."

Tabs

In "Navigation Settings", I clicked off "Automatically generate tabs". I set these manually in portal_actions in the ZMI. "Tech" for example appears in the tab navigation, because I added an action of category "portal_tabs", set the permissions so that anyone with "view" priveleges (which includes anonymous users) can see it, and gave it a URL that would point to my "tech" Smart Folder, which displays all content relating to information technology. A "Contents" tab appears only to the portal manager, when he logs in. It’s the folder that contains all the content.

Properties

In the properties of the Plone object itself, I made the default_page my "home_listing" Smart Folder which gets all News Items and Documents and sorts and batches them appropriately.
In portal_registration, I made the "join" action invisible.
In portal_properties -> site_properties, I changed localTimeFormat and localLongTimeFormat for maximally human-readable dates.

RSS Syndication

In the ZMI, click the portal_syndication tool.
Activate syndication in the "properties" tab.
Now go to your Plone site and notice that there’s a new "syndication" tab on any folderish objects, which include "smart folders" or queries. When you enable syndication for your selected folders a new "RSS" icon will appear with a link that can be used as the RSS feed.

Language

Random Weirdness

At some point the portal catalog got completely hosed and would not return search results correctly–each item was assigned a 1% relevant rating upon any advanced search, and a simple search would return no results at all. I ended up having to transplant the content to a new Plone site and reconfigure the stuff described above. Hope it doesn’t happen again.

Granted, I did a lot of the development on a release candidate version. I noticed that other random weirdness, not noted here, was resolved by upgrading the final release.

TO DO

Turn everything in the portal_skins "custom" directory into a filesystem product.
Fix CSS of quick search form so that it shows up in I.E. (currently only appears in Firefox).
del.icio.us links portlet
Make calendar dates link to content. Otherwise do away with the calendar entirely.
Enable comments, ensuring comments spam can’t happen.
Make RSS syndication put the first paragraph (or 250 words or whatever) of the entry into the "description" field–I don’t usually supply a separate "description" for my entries.
Figure out how to make Restructured Text the default type.
Implement ATAmazon portlet, since books are a big focus of the "ideas" section. Also book cover graphics give the site a lot more character.