Spamassassin, spamd, FreeBSD and “autolearn: unavailable”

I recently built a mail server using FreeBSD 8.2 and compiled spamassassin from the current ports collection, to run globally. spamd looked okay and it was adding headers, but after a while I noticed the Baysian filtering didn’t seem to be working in spite of it having had enough samples through.

A closer look at the added headers showed “autolearn: no”, or “autolearn: unavailable” but never “autolearn: ham/spam”.

What was going on? RTFM and you’ll see spamassassin 3.0 and onwards has added three new autolearn return codes: disabled, failed and unavailable. The first two are pretty self-explanatory: either you’d set bayes_auto_learn 0 in the config file or there was some kind of error thrown up by the script. But I was getting the last one:

unavailable: autolearning not completed for any reason not covered above. It could be the message was already learned.

I knew perfectly well that the messages hadn’t already been learned, so was left with “any reason not covered by the above”. Unfortunately “the above” seemed to cover all bases already. There wasn’t any clue in /var/maillog or anywhere else likely.

I don’t much care for perl scripts, especially those that don’t work, so after an unpleasant rummage I discovered the problem. Simply put, it couldn’t access its database due to file permissions.

The files you need to sort are at /root/.spamassassin/bayes_* – only root will be able to write to them, not spamd – so a chmod is in order.

Please generate and paste your ad code here. If left empty, the ad location will be highlighted on your blog pages with a reminder to enter your code. Mid-Post

A better solution is to move the Bayesian database out of /root – /var would probably be more appropriate. You can achieve this by adding something like this to /etc/spamd.cf (which should link to /usr/local/etc/mail/spamassassin/local.cf):

bayes_path /var/spamassassin/bayes/bayes
bayes_file_mode 0666

I suspect that the lower-security Linux implementation avoids these problems by setting group-write-access as default, but FreeBSD, being a server OS, doesn’t. It’s also a bug in the error handling for the milter – it should clearly report as a “failed” and write something to the log file to tell you why.

You should be able to restart spamd after the edit with /usr/local/sbin/spamdreload, but to be on the safe side I use the following after shutting down Sendmail first.

/usr/local/etc/rc.d/spamass-milter restart
/usr/local/etc/rc.d/sa-spamd/restart

I don’t know if Sendmail can cope well with having spamass-milter unavailable, but why take the risk?

 

OpenLDAP, Thunderbird and roving address books

IMAP is great. It lets you keep your mail synchronised between any number of machines, including webmail, and everything works just fine. The only snag is that your address book isn’t included. I’d always assumed this was what LDAP was for: a centralised directory of names, and other things, with the useful bit being the address book. Thunderbird, is my current favourite mail client on the basis that actaully works better than Outlook. It supports LDAP address books, and has offered to configure one for me many times. All I needed to do was configure slapd (the OpenLDAP server deamon) and point Thunderbird at it.

This blog entry isn’t a tutorial in configuring FreeBSD, OpenLDAP and Thunderbird to work together. I’m saving you from wasting a lot of your time trying. It does “work”, once you’ve sorted out schemas and got to grips with the arcane syntax of the configuration files and the hierarchical nature of the thing. It’s just that it’s useless even when it’s working because it’s READ-ONLY. Being able to add and amend entries in my address book is so fundamental to the nature of an address book that I didn’t bother to check that Thunderbird could do it. What’s the use of a read-only address book? Well there might be some point in a large organisation where a company-wide address book is needed, administered by a tame geek in the basement. For the rest of us it’s as fileofax with no pen.

So what are the good people at Mozilla playing at? The omission of read/write has been listed in their bug database for over ten years, and no one has tackled it. I thought about it for a while, but given the that Lightweight-DAP is a misnomer on a spectacular scale I thought again. Clearly no one who knows about LDAP actually likes it enough to want to help; either that or none actually understands it apart from the aforementioned geek in the basement, and he’s sitting tight because allowing users to edit address books might be detrimental to his pizza supply.

The time is right for a genuinely lightweight protocol for sharing address books in a sane and sensible manner; something like IMAP for addresses. I’m therefore writing one. Unfortunately I’m not so clued up on Thunderbird’s internal workings; if you are and wish to implement the front end please drop me a line and I’ll write a protocol and server that works.

Unfortunately this one issue is a killer app for Microsoft’s lightweight over-priced Mail system called Exchange. It’s a big of a dog (inflexible) but at least Microsoft seems to have got this fundamental functionality for sharing personal address books between mail clients sorted out. I believe it uses something similar LDAP underneath (along with IMAP for the mail itself); so it’s not impossible.

I’m very surprised to find myself having anything good to say about Outlook/Exchange Server. It might still be traumatised from the discovery that my assumption that the obvious LDAP solution was nothing of the sort. It’s just it’s so damn complex for no apparent reason that it gives the impression it must be great if you could only understand it.

Fetchmail, Sendmail and oversized emails

There’s a tendency for lusers to try to email anything these days. If you though a few Gig of outgoing mail queue was enough you haven’t come across the luser who decided to email the contents of a CD (uncompressed) to all her friends. Quite what they’d have made of their iPhone trying to download it I’ll never know.

Sendmail has a method for limiting emails to a sensible size. As a reminder, inside host.example.com.mc you need to add:

# The following sets the maximum message to 5Mb - otherwise it's infinite
define(`confMAX_MESSAGE_SIZE', `5242880')

Then run “make” and “make install” and “make restart”. This will generate the sendmail.cf (and any hashmaps) before restarting. The bit you always forget when changing .mc files is the “make install”. This is all for FreeBSD – Linux types, please do it your own way.

So this is great – anyone sending an over-sized email is bounced from their server, and local users submitting email will be similarly clipped into the world of sane and sensible (if you regard something as large as 5Mb as sensible for an email).

But I came across one interesting issue recently and it could happen to you, too, if you’re using fetchmail.

For those who haven’t come across it before, fetchmail pulls emails from a POP3 box and delivers them to local users – dropping them into your local MTA by default. This is reasonable, as everything then goes through the spam filtering, procmail and anything else you have defined. It’s really useful for legacy situations where someone’s ended up with a POP3 box somewhere and you need to integrate it with the rest of their mail.

Fetchmail does plenty more besides, and has a config file to match the functionality. Presumably as a reaction against the complexity of the sendmail.cf syntax, this one tries to operate in plain English. I’ve never quite figured out the full syntax, but it’s designed to be “flexible” and figure out what you’re trying to say. Personally I don’t think it succeeds in being any more friendly then sendmail.cf in spite of being on the other end of the spectrum.

Anyway, the fun comes when fetchmail downloads an over-sized email from the POP3 box and delivers it locally via Sendmail. Sendmail will reject it, and send a bounce back to the original sender. So far, so good but f Sendmail is running as a cron job every five minutes, the luser gets a bounce back every five minutes because the outsized mail is stuck in the POP3 box. Opps! It may serve them right, but they shouldn’t be allowed to suffer for too long.

Fortunately one of fetchmail’s many options allows you to control the maximum download size, if you could figure out the syntax. It’s available as a command-line option –l , but if you prefer to keep things in the .fetchmailrc file (the best plan) you’ll need to proceed as per the following example. They keywords are “limit” and “limitflush”.

  • local-postmster-account is the login for your local postmaster – undelivered emails go there.
  • pop3.isp.co.uk – mail server with the POP3 box
  • users-domain.co.uk – Domain name who’s email ends up in POP3 box above
  • pop3-username, pop3-password – what you use to log into the POP3 box
  • Tom, Dick and Harry are local mailboxes, with tom being the default.
    set postmaster local-postmster-account

    poll pop3.isp.co.uk proto pop3 aka users-domain.co.uk no envelope no dns:
    user "pop3-username", with password "pop3-password",
    limit 5242368 limitflush to

    dick
    "dick@users-domain.co.uk " = dick
    "richard@users-domain.co.uk " = dick

    harry
    "harry@users-domain.co.uk " = harry

    tom
    "tom@users-domain.co.uk" = tom
    "*@ users-domain.co.uk " = tom

    here

    This isn’t intended as a tutorial in writing .fetchmailrc files – only an example of the use of limit and limitflush.

    So what’s going on? The limit keyword must be part of the poll statement, and is followed by the size (in bytes) of the maximum email to be retrieved. In the example it’s 512 bytes less than the 5Mb used in Sendmail (I feel I need a bit of slack on a boundary condition; it may be okay if they’re identical but I why push your luck?)

    Please read the fetchmail documentation for full details (although it’s light on examples). With just the “limit” keyword in use, over-sized mails will be left I the POP3 box. The following “limitflush” keyword will silently delete over-sized emails so they don’t bother you again. You may not want to do this! If you don’t, someone will have to retrieve or delete the emails form the POP3 box manually.

    Note that putting a limit on the download will prevent the bounce messages going to the original sender as it won’t get as far as sendmail.

chkrootkit finds bindshell infected on port 465

The current version of chkrootkit will throw up a warning that bindshell is INFECTED on port 465 in some circumstances when this is nothing to worry about. What it’s actually doing (in case you can’t read shell scripts, and why should you when there’s a perfectly good ‘C’ compiler available) is running netstat and filtering the output looking for ports that shouldn’t be being used. Port 465 is SMTP over SLL, and in my opinion should very definitely be used, but it is normally disabled by default.

As to whether this should worry you depends on whether you’re using secure SMTP, probably with sendmail. If you set up the server you should know this. If someone else set it up and you’re not too familiar with sendmail, the tell-tail line in the .mc file is DAEMON_OPTIONS(`Port=smtps, Name=TLSMTA, M=s')dnl. Note the ‘s’ on the end of smtp.

Assuming you are using SMTPS, you can easily stop chkrootkit from printing an error (or returning an error code) simply by modifying the bindshell() subroutine to remove 465 from the list of ports to check. It’s on line 269 on the current, 0.49, version of the script.

I’m not so convinced that chkrootkit is any substitute for an experienced operator, but it’s out there, people use it and its better than nothing.