I’m using AWStats to track usage on a Plone site that’s essentially a portal for collaborative document translation. That means that most users of the site need to login to do anything useful, and I want to see who’s logging in along with the rest of the stats on my site.

For hosting the site, I’m using the recommended setup which uses Zope’s Virtual Host Monser to route requests and responses between Apache and the Zope application server appropriately. I.e., I include the following two key lines in my Apache directive:

RewriteEngine On
RewriteRule ^/(.*) http://localhost:8080/VirtualHostBase/http/%{SERVER_NAME}:80/MyApp/VirtualHostRoot/$1 [NC,P,L]

Since I’m going thru Apache, I can record requests in log files as with any other site. Unfortunately, Zope/Plone does not pass information about the users who authenticate via its own system to the HTTP headers. Taking a cue from this thread, I came up with the following solution. It’s tested on Plone 2.1.3, but probably won’t work on Plone 2.5, given the latter’s use of Pluggable Authentication Service.

First, I found a place in the Plone page rendering mechanism that gets executed on each page view. I suppose the place that I chose was somewhat arbitrary, but in the "authenticate" method of the file GroupUserFolder/GroupUserFolder.py, I set a header called X-PloneUser, as shown in the following patch, available for your use:

1020,1022d1019
<         # PATCH FOR TRACKING AUTH USER IN APACHE LOGS --mdorn:
<         if name is not None:
<             request.RESPONSE.setHeader('X-PloneUser', name)

In my Apache configuration, I had to change LogFormat from "combined" to something more specific, and reference the label in the CustomLog:

LogFormat "%h %l %{X-Ploneuser}o %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" plone
CustomLog "|/usr/local/sbin/cronolog /home/httpd/MYDOMAIN/logs/%m-%Y/access_log" plone

To see authenticated users in your AWStats configuration file, you’ll need to change the default value for that setting:

ShowAuthenticatedUsers=1

That’s it. Next time AWStats processes your logs, assuming you had visits from authenticated users, you’ll now see them in the appropriate location on your AWStats report.

Like most Web sites, this site shares server space with a number of other domains. When I recently undertook to set up AWStats after years of not knowing anything about what kind of traffic my personal sites were getting, I figured it would probably be relatively easy to make it so that anybody else with an account on the box who also wanted stats could avoid the hassle.

With a bit of help from another tutorial, I came up with the following solution.

First of all, I installed the AWStats package on my server’s Fedora system via yum. With that installation, I ended up with AWStats’ default config file in /etc/awstats/awstats.model.conf.

This setup needs a separate AWStats configuration file for each domain, so this default file is the model from which all others will be generated. For that reason, I needed to tweak a few of the default values to work the script described later:

LogFile="/home/httpd/$USERNAME/logs/%MM-0-%YYYY-0/access_log"
SiteDomain="$DOMAIN"
HostAliases="$ALIASES"
DirData="/home/httpd/$USERNAME/awstats"
AllowAccessFromWebToFollowingAuthenticatedUsers="$USERNAME"

(I should also note that I enabled the GeoIP plugin in this file, but that’s a subject for a different post.)

Now, on the Apache side of things, you need to decide on a standard location for the Apache log files for all the accounts. In my case, that’s /home/httpd/ACCOUNT_NAME/logs. (In that directory, the setup described below will generate directories named by month and year (like so: MM-YYYY), and put the log file there.) The Apache virtual host directives of course will need to specify that location. Since I don’t actually know or have any communication with my neighbors who share the box, I’ll leave that to them, but my new directives look like this:

CustomLog "|/usr/local/sbin/cronolog /home/httpd/DOMAIN/logs/%m-%Y/access_log" combined

Where DOMAIN is the name of the directory that holds the Web files (htdocs, logs, etc.) for a given domain on the server.

I’ve also installed cronolog as an easy way to rotate logs. As you can see the Apache config pipes to that program. That’s not strictly necessary for this setup, but if you want to do it differently (e.g., with logrotate), you’re on your own.

Making use of this information, anytime you want AWStats set up for a new domain, you can use the following script, which is named /usr/local/sbin/awstats_script.sh in my setup, and which was adapted from another tutorial , mentioned previously:

#!/bin/bash
echo "Enter the username:"
read WUSERNAME

ACCESS_FILE="/etc/awstats/awstats.pwd"
STAT_DIR="/home/httpd/$WUSERNAME/awstats"

echo "Enter the password:"
read PASSWORD

echo "Enter the main domain:"
read DOMAIN

echo "Enter aliases separated by space:"
read ALIASES

# Create the statistics directory
if [ -d $STAT_DIR ]; then
    echo "Statistics dir already exist"
else
    mkdir $STAT_DIR
fi

# Create the virtual host awstats.conf
cat /etc/awstats/awstats.model.conf | \
sed -e "s/\\\$DOMAIN/$DOMAIN/g" | \
sed -e "s/\\\$USERNAME/$WUSERNAME/g" | \
sed -e "s/\\\$ALIASES/$ALIASES/g" > \
"/etc/awstats/awstats.$DOMAIN.conf"

# Add user/password to password file
if [ -e $ACCESS_FILE ]; then
    /usr/bin/htpasswd -bm $ACCESS_FILE $WUSERNAME $PASSWORD
else
    /usr/bin/htpasswd -bm -c $ACCESS_FILE $WUSERNAME $PASSWORD
fi

The "username" requested is the name found in the /home/httpd directory for which you want to create the site. This will also be the username needed to login (via Apache authentication) to view the stats. The "password" then requested is for that same Apache authentication (will be stored in /etc/awstats/awstat.pwd). The "main domain" is what people use to access the site, (e.g., www.DOMAIN.com), but you’ll also then be asked to include any subdomains that log to the same Apache log file (at least if you want AWStats to process them).

After the setup is complete, you can check out your stats by going to: http://www.DOMAIN.com/awstats/awstats.pl

AWStats knows which config file (automatically generated) it needs by reading the domain from the URL, but you can also view other domains by appending "?config=www.whatever.com" to the URL. That config file is left in /etc/awstats, and the individual AWStats DB file that results from processing the logs is left in each user’s directory in /home/httpd/DOMAIN/awstats

While each user will be able to HTTP-authenticate against any other user’s domain, the values in the AllowAccessToWebToFollowingUsers variable will prevent users from being able to view one another’s stats.

Finally, note that your AWStats installation may have included a cron task to update the AWStats data in /etc/cron/hourly or other. In any case, you’ll want AWStats to automatically process your logs on an at least daily basis via the included script /usr/share/awstats/tools/awstats_updateall.pl.

Once you have a Web site that gets a moderate amount of traffic, it’s likely to become the object of the attention of "robots" that crawl your site for the purpose of archiving or indexing its content or otherwise making the site available offline. We have one site whose number of "not viewed" pages exceeds the number of (presumably) human-viewed pages by more than three times, according to our AWStats numbers. While our hosting provider’s bandwidth allocation is generous, given that we’re using a particularly resource-demanding application server for the site I’m much more concerned about the strain on other, physical resources, especially memory and CPU, brought about by this activity.

The real answer in our particular case is to tame the site with a caching strategy and other methods to be able to better tolerate high traffic, but that involves an effort that we haven’t had the time or resources to undertake. So Plan B is to take a closer look at exactly who these robots are, and whether they really need to be looking at your site, because you have a couple of different ways of blocking their access.

Some of them, like Googlebot, you obviously want to give unrestricted access to your site, because they’re the mechanism by which search engines index your site and allow their users to find your content in as a result of their keyword searches.

These legitimate bots, though, are not so well-behaved as you might expect. Yahoo!’s "Slurp" indexing bot, for example, single handedly accounted for well over a gigabyte of bandwidth (and well over 10% of the traffic) in a month (it’s an extensive site but not that extensive). Is this really necessary? I don’t know enough about how these bots work to know for sure, but it strikes me as excessive.

These bots typically identify themselves in your Web server access logs along with other information such as the IP address:

MISSING TEXT

Legitimate search engines can be instructed not to crawl your site, or to ignore sections of it using the robots exclusion standard, but most non-mainstream robots ignore this file anyway, so a more reliable method is using Apache’s Rewrite module.

So say you wanted to block this bot, you might use Apache’s Rewrite module to send the bot a "forbidden" response like so:

RewriteCond %{HTTP_USER_AGENT} ^(.*)Yahoo(.*) [OR] RewriteRule .* – [F,L]

(You can probably use regular expressions with a bit more precision than I do here.)

There’s the additional matter of whether these bots are really who they say they are. The information illustrated in the above access log entry is easy to spoof, including the IP and the user agent information at the end. Bad bots can even impersonate a browser like Microsoft Internet Explorer. If IPs aren’t being spoofed, you can also use Apache Rewrites to block by IP, but if they are, what can be done?

What I haven’t had much time to investigate is what purpose these rogue bots serve and to whom. Going down that path just a little, you find yourself in a sordid world of things like referrer log spamming–which I now realize that we’re also being victimized by–and in the company of lots of other folks who are trying to navigate the treacherous waters of a sea that used to be pretty calm as far as these things go.

For now, we’re sticking with the blunt instrument of blocking virtually all non-browser agents except Googlebot, and hoping that user agent spoofing is relatively rare.