August 2011 – Frank Leonhardt's Blog

Spamassassin, spamd, FreeBSD and “autolearn: unavailable”

I recently built a mail server using FreeBSD 8.2 and compiled spamassassin from the current ports collection, to run globally. spamd looked okay and it was adding headers, but after a while I noticed the Baysian filtering didn’t seem to be working in spite of it having had enough samples through.

A closer look at the added headers showed “autolearn: no”, or “autolearn: unavailable” but never “autolearn: ham/spam”.

What was going on? RTFM and you’ll see spamassassin 3.0 and onwards has added three new autolearn return codes: disabled, failed and unavailable. The first two are pretty self-explanatory: either you’d set bayes_auto_learn 0 in the config file or there was some kind of error thrown up by the script. But I was getting the last one:

unavailable: autolearning not completed for any reason not covered above. It could be the message was already learned.

I knew perfectly well that the messages hadn’t already been learned, so was left with “any reason not covered by the above”. Unfortunately “the above” seemed to cover all bases already. There wasn’t any clue in /var/maillog or anywhere else likely.

I don’t much care for perl scripts, especially those that don’t work, so after an unpleasant rummage I discovered the problem. Simply put, it couldn’t access its database due to file permissions.

The files you need to sort are at /root/.spamassassin/bayes_* – only root will be able to write to them, not spamd – so a chmod is in order.

A better solution is to move the Bayesian database out of /root – /var would probably be more appropriate. You can achieve this by adding something like this to /etc/spamd.cf (which should link to /usr/local/etc/mail/spamassassin/local.cf):

bayes_path /var/spamassassin/bayes/bayes bayes_file_mode 0666

I suspect that the lower-security Linux implementation avoids these problems by setting group-write-access as default, but FreeBSD, being a server OS, doesn’t. It’s also a bug in the error handling for the milter – it should clearly report as a “failed” and write something to the log file to tell you why.

You should be able to restart spamd after the edit with /usr/local/sbin/spamdreload, but to be on the safe side I use the following after shutting down Sendmail first.

/usr/local/etc/rc.d/spamass-milter restart /usr/local/etc/rc.d/sa-spamd/restart

I don’t know if Sendmail can cope well with having spamass-milter unavailable, but why take the risk?

15-August-1115-August-11

OpenLDAP, Thunderbird and roving address books

IMAP is great. It lets you keep your mail synchronised between any number of machines, including webmail, and everything works just fine. The only snag is that your address book isn’t included. I’d always assumed this was what LDAP was for: a centralised directory of names, and other things, with the useful bit being the address book. Thunderbird, is my current favourite mail client on the basis that actaully works better than Outlook. It supports LDAP address books, and has offered to configure one for me many times. All I needed to do was configure slapd (the OpenLDAP server deamon) and point Thunderbird at it.

This blog entry isn’t a tutorial in configuring FreeBSD, OpenLDAP and Thunderbird to work together. I’m saving you from wasting a lot of your time trying. It does “work”, once you’ve sorted out schemas and got to grips with the arcane syntax of the configuration files and the hierarchical nature of the thing. It’s just that it’s useless even when it’s working because it’s READ-ONLY. Being able to add and amend entries in my address book is so fundamental to the nature of an address book that I didn’t bother to check that Thunderbird could do it. What’s the use of a read-only address book? Well there might be some point in a large organisation where a company-wide address book is needed, administered by a tame geek in the basement. For the rest of us it’s as fileofax with no pen.

So what are the good people at Mozilla playing at? The omission of read/write has been listed in their bug database for over ten years, and no one has tackled it. I thought about it for a while, but given the that Lightweight-DAP is a misnomer on a spectacular scale I thought again. Clearly no one who knows about LDAP actually likes it enough to want to help; either that or none actually understands it apart from the aforementioned geek in the basement, and he’s sitting tight because allowing users to edit address books might be detrimental to his pizza supply.

The time is right for a genuinely lightweight protocol for sharing address books in a sane and sensible manner; something like IMAP for addresses. I’m therefore writing one. Unfortunately I’m not so clued up on Thunderbird’s internal workings; if you are and wish to implement the front end please drop me a line and I’ll write a protocol and server that works.

Unfortunately this one issue is a killer app for Microsoft’s lightweight over-priced Mail system called Exchange. It’s a big of a dog (inflexible) but at least Microsoft seems to have got this fundamental functionality for sharing personal address books between mail clients sorted out. I believe it uses something similar LDAP underneath (along with IMAP for the mail itself); so it’s not impossible.

I’m very surprised to find myself having anything good to say about Outlook/Exchange Server. It might still be traumatised from the discovery that my assumption that the obvious LDAP solution was nothing of the sort. It’s just it’s so damn complex for no apparent reason that it gives the impression it must be great if you could only understand it.

15-August-1129-September-11

WordPress ends up with wrong upload directory after moving servers

If you’ve moved WordPress from one server (or home directory) to another, everything looks fine until you come to upload a file. It then complains that it can’t write to the upload directory. Something like:

Unable to create directory /usr/home/username/wp-content/uploads/xxx/yy. Is its parent directory writable by the server?

A search through the PHP config files doesn’t reveal a path to the upload directory, and if it’s possible to change it from the Dashboard I’ve yet to find where.

The only remaining option was the mySQL database, and fishing around in the likely sounding wp_option table I found a option_name of “upload_path”. A quick query showed this was the home directory.

To save you fishing, here’s the SQL needed to set things right. Obviously select the appropriate database with “use” first!

update wp_options set option_value='<path-to-your-wordpress-installation>/wp-content/uploads' where wp_options.option_name='upload_path';

This is how you do it using straight SQL from the command line. If you’re using some sort of restricted web hostinig account “See your system administrator” and point them here.

9-August-1129-August-11

Apps to force Web into decline?

Who’s going to win the format war – iOS (Apple iPad) or Android? “What format war?” you may ask. Come on, it’s obvious. Some are saying that the web is either dying (dramatic) or at the least being impacted by the modern fashion of Apps, and these run on iOS or Android (mostly). Actually, by sales Apple is winning hands-down.

This IS a format war, because developers need to support one or other platform – or both – and users need to choose the platform that has the content they need, and there is some sense in it when databases contents are queried and displayed in Apps rather than on web pages.
Apple has the early advantage, and the cool factor. But it’s the most expensive and the most hassle to develop for, as Apps can only be sold through Apple. Android is a free-for-all. Apps can be sold through Google, or anyone else making them available for download in the future. It’s an open standard. The security implications of this are profoundly worrying, but this is another story.

So, running iOS is expensive, Android is insecure and neither are very compatible. That’s before you consider Blackberry and any requirement to run an App on your Windows or Linux PC.

But, I don’t think this is a conventional format war. It’s mostly software based, and open standards software might just win out here (and I don’t mean Android). People like paying for and downloading Apps. Web browsers can (technically) support Apps, using Java and the upcoming HTML5 in particular. Why target a specific operating environment when you can target a standard web browser and run on anything?

As an aside, HTML5 is sometimes hailed as something new and different when in fact it’s just evolution and tidying up. The fact is that HTML is cross-platform and will deliver the same functionallity as Apps. HTML5 simply standardises and simplifies things, making cross-platform more open-standard, so every browser will be able to view page content without proprietary plug-ins, including better support for mobile devices which lost out in the late 1990’s onwards when graphic designers decided HTML was a WYSIWYG language.

Some modern-day pundits will proclaim that data will be accessed more through Apps in the future, and the web has had its decade. Apparently a third of the UK is now using smart-phones. Whether this statistic is correct or not, they’re certainly popular and I’ll concede that Apps are here to stay. But in my vision of the future they won’t be running on iOS, Android or Blackberry – they’ll be written using HTML5 and run on anything. It’s platform independence that launched HTML and the web twenty years ago, and it’s what will see off the competition for the next twenty years.