Posted on

Using ISO CD Images with Windows – Burn.Now problems

When CD-R drives first turned up you needed special software to write anything – originally produced by Adaptec but they were soon overtaken by Nero, with NTI and Ulead having lower cost options. Now, when you get a PC it will usually come with one of the above bundled, and Microsoft has added the functionally to Windows since XP (for CD, if not DVD). This is not good news for the independent producers, but Microsoft’s offering doesn’t quite cut the mustard, so most people will want something better.

My new Lenovo PC came bundled with Corel Burn.Now. Corel recently bought the struggling Ulead, and this is fundamentally the same product as Ulead burn.now. Unfortunately Burn.Now is also pretty feeble – it just can’t do the basics.

To duplicate a CD you need to copy all the data on it. Pretty obvious really. If you’re not copying drive-to-drive it makes sense to copy the data to a .ISO image on your hard disk. You can then transfer it to another machine, back it up or whatever; and write it to a new blank disk later. Burn.Now will create a CD from an ISO image, but if you ask it to copy a disk it uses its own weird and whacky .ixb format. Some versions of Burn.Now gave you the choice, but not the new Corel. It’s .ixb or nothing. This matters, because whilst everyone can write .ISO files, only Burn.Now can write from  .IXB format.

Burn.Now is crippled. What about Microsoft’s current built-in options? You can actually write an ISO image using Windows 7 – just right-click on the file and select “Burn disc image”. Unfortunately there is no way to create such a file with Windows. To do this you need add Alex Feinman’s excellent ISO Recorder, which basically does the opposite: Right-click on the CD drive and select Create Image from CD/DVD.

Unfortunately ISO Recorder doesn’t read all disks – it won’t handle Red Book for a start. This is a bit of a limitation – was its author, Mr Feinman concerned about music piracy? Given Windows Media Player can clone everything on an Audio CD without difficulty, his conciousness efforts won’t make a lot of difference.

So – Windows is its usual painful self. If you just want to simply create an image of a CD or DVD with no bells and whistles, go to UNIX where it’s been “built in” since the 1980’s (when CD-ROMs first appeared). Just use the original “dd” command:

Please generate and paste your ad code here. If left empty, the ad location will be highlighted on your blog pages with a reminder to enter your code. Mid-Post

# dd if=/dev/acd0 of=my-file-name.iso bs=2048

An ISO file is simply a straight copy of the data on the disk, so this will create one for you. You can write it back using:

# burncd -f /dev/acd0 data my-file-name.iso fixate
Or
# cdrecord dev=1,2,3 my-file-name.iso

Burncd is built in to FreeBSD (and Linux, IIRC), but only works with atapi drives. In the example it assumes the CD recorder is on /dev/acd0 (actually the default).

Cdrecord works with non atapi drives to, but has to be built from ports on FreeBSD and for other platforms it’s available here – along with lots of other good stuff. The example assumes the device is 1,2,3 – which is unlikely! Run cdrecord -scanbus to locate the parameters for your drive.

Once you have your ISO file, of course, you could use Windows to write it. The choice depends on whether you have strongly held views on whether Windows is a worthy desktop operating system. Corel Burn.Now is, however, a long way from being a worth CD/DVD writing utility.

Posted on

Can’t get PuTTY and FreeBSD with OpenSSH to do a Certificate Login – Myths

Following yesterday’s post about issues getting “Server Refused Our Key” errors when trying to use PuTTY to log in to FreeBSD with a certificate, I thought I’d just lay to rest a few myths I’ve seen on various web sites where people have tried to explain how to do this. It’s easy to see how these myths develop – I’ve laboured for years under the misapprehension that I needed to do something or other when it was just a coincidence it had started working the first time the idea came to me. So here goes with a few of the myths. If you’re not getting this to work, it’s not for one of these reasons:

Myth: You need to specify 0600 permissions for the authorized_keys file (or the .ssh directory)

Simply not true. It may be a good idea to stop others from reading your keys, although they are “public” keys and won’t let anyone else in anyway (unless a they have a suitable cracking tool and a lot of processing power – and I mean a lot). Only your private key needs to be a secret. The only stipulation is that they must only writeable by the user – 0644 is okay, 0664 or 0666 isn’t.

But as I mentioned yesterday, you MUST ensure that your home directory is also not world-writable! You mustn’t have 0777 permissions! 0755 is okay, as is 0711. I’ve not seen this documented anyway, but it’s true for FreeBSD 7.0 to 9.0.

Myth: OpenSSH requires the authorized_keys file to be owned by the user trying to log in

Again no – it simply doesn’t. It has to be readable to that user (not just root) – this may be because it’s world readable or group readable for the user in question. It might as well be owned by root:wheel as long as it’s Other read bit is set.

Myth: If you’re using SSH2, you need a file called authorized_keys2

This might be true on some installations, but not current ones! I’ve no reason to believe that this file would even be considered, never mind required. The file used is defined in the /etc/ssh/sshd_config, and on current versions of FreeBSD (7.0-9.0) it’s definitely authorized_keys

Myth: You must generate the keys using the OpenSSH keygen utility on FreeBSD – puttygen doesn’t work

Well, there’s a bit of truth in this, but not much. Put simply, the format is different, but this only extends as far as the header and comment.

OpenSSH keys look like this:

ssh-rsa AAAAB3NzaC1y… very long line … sXi+fF noone@example.com

PuTTYGen Keys look like this:

---- BEGIN SSH2 PUBLIC KEY ----
Comment: "no one@example.com"
AAAAB3NzaC1y … long line, possibly with breaks … sXi+fF
---- END SSH2 PUBLIC KEY ----

You can convert one to the other using any text editor of your choice, as long as it handles long lines properly (like vi).

I can see there could be all sorts of fun and games if you simply cut/pasted these end ended up with extra line breaks, spaces or truncation – but the key data and its encoding is exactly the same, and that’s the bit that makes it work or not.

If you generate your key using OpenSSH tools you will need to load it into PuTTY Gen and write a Private .ppk key on your Windoze box. Or not. It’s just a text file and you could put the appropriate wrapper on it, but you might as well just use PuTTY Gen.

Myth: You need to edit /etc/ssh/sshd_conf to enable certificate login

No you don’t. The default values as shipped work just fine. Because the file consists of commented out lines of parameters with their default values, I suspect people though that some have been confused about whether the ‘#’ needed to be removed before the parameter came in to effect. They don’t – you only need to remove the comment if you want to change the default value. If you do remove the comment, but don’t edit the value, it’ll make no difference to anything.

What’s Real

In my experience, problems are almost always down to either directory permissions (see above) or errors transcribing public keys from one machine to another – and chaos and confusion caused by the abovementioned myths!

 

Posted on

PuTTY, FreeBSD and SSH certificate logins

I’ve just gone crazy trying to figure out why PuTTY kept getting a “Server Refused Our Key” error when I tried to log in to a host using a certificate for the first time. Looking around the web, there are a lot of interesting theories about how to generate the certificates, and out of desperation I tried them all – nothing worked. So, for what it’s worth, here’s what does.

Generate your certificate on FreeBSD using the OpenSSH utility:

ssh-keygen -t rsa

With the default options this will create a couple of files in the .ssh directory within your home directory, and by default they’ll be called “id_rsa” and “id_rsa.pub”. In other words, if you’re user ID is fred the files will be in /usr/home/fred/.ssh/ with the above names. One’s private, the other is public.

You need to add the public key to the list of authorised keys in the .ssh directory:

cat id_rsa.pub >> ~/.ssh/authorized_keys

(The name authorized_keys with the American spelling is set in /etc/ssh/sshd_config)

Next you need to get the private key back to the machine running PuTTY. It’s just text – you can cut/paste it into a text editor and save it. For PuTTY to use it, however, it needs to be converted in to PuTTY’s own format, which you do using the PuTTY Key Generator, puttygen.exe. Run this, click on the Load button and read in your text file, then use the Save Private button to put write the .ppk file somewhere safe. You may wish to set a passphrase on it if there’s any chance someone else can get hold of it!

You may now get rid of the id_rsa.* files on the FreeBSD host, although you might want to add the public key to more than one user on more than one host – it’s a “public” key so there’s no harm in using it all over the place.

It is possible to use PuttyGen to make the keys and copy them to the FreeBSD host instead. A lot of people seem to have had trouble with this in the past (myself included), and it’s probably easier not to, especially if you’re going to use the keys in OpenSSH format for other purposes on the FreeBSD host anyway.

You’ll see a lot about setting the files in .ssh in some very restricted ways – basically all you need to do is ensure that they’re only writable by you. You can make your .ssh directory only readable by you if you wish but it won’t stop it from working. Also, the default /etc/ssh/sshd_config files is fine, and you don’t need to uncomment anything (in spite of what you might read). The default settings are all good, and all commented out, as it says on the top of the file. (Not quite true now – see 2024 update below)

Now, here’s the trick! What will cause a problem, as I eventually figured out, is if your home directory is writable by others. Don’t ask me how or why this should be true, but I tried this after I’d tried eliminating everything else on comparing working and non-working boxes. I know this for sure with FreeBSD 8.1 – ensure your home directory is drwxr-xr-x (or possibly less).

The final stage is to set up a session profile in PuTTY. This isn’t a tutorial for PuTTY, so I’ll be brief. In the options category open to Connection/Data and set the auto-login username you wish to use (if you haven’t already). Then under Connection/SSH/Auth select the private (.ppk) file you want to use. Remember, you can use this file with as many hosts and user accounts as you’ve added the public key to the .ssh/authorized_keys file. Save the session, and that’s it done. If it doesn’t do it for you, take a look in /var/log/auth.log.

Update 2024:
And finally, twelve years later, there’s a problem. newer versions of SSH will barf at RSA keys. You’ll get a “The server refused our key” message and something like this in auth.log…

 sshd[1539]: userauth_pubkey: signature algorithm ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]

Don’t worry – there’s a quick fix. In /etc/ssh/sshd_config add the following line somewhere that makes sense.

PubkeyAcceptedAlgorithms +ssh-rsa

You might want to use soemethign other than RSA keys going forward, but this is an update to a 2012 article – watch out for a new one.

Posted on

HP Microserver and WOL

Update: See article here

 

They just don’t seem to work. I’ve spent an annoying hour or so trying to get WOL to work with an HP Microserver – no joy whatsoever. I assumed it must be my code until I tried it on a few other machines but they worked just fine.

Now most of my machines are Realtek whereas HP are using Broadcom (as do the Dells). I’m not saying there’s anything wrong with Broadcom, but whenever I have a weird network problem they have a habit of being at the heart of it. Is it my magic packet? As far as I know it’s supposed to be 48-bits of ‘1’ followed by sixteen copies of the MAC address. Does it need a secure-on password? If so, how come you can’t set one in the BIOS.

I’ve asked an HP server expert: “Update the BIOS”. Perhaps, but these are brand new machines of an established design. They either turn on when they receive the packet, or they don’t work, and I can’t believe HP didn’t test them. Then again…

I’m told that these do support WOL on Windows, but not if you’re running anything else. On the face of it this is bonkers. Why should the OS the powered-off drive affect anything. The machine is off; the OS isn’t running. Well here’s a theory – before Windows shuts off it puts something in a register on the Broadcom chip to leave it in a WOL state. With the wrong drivers this doesn’t happen. Setting it in the BIOS doesn’t help, because it’s erased by the OS driver. The BIOS doesn’t restore it as the power is killed, but Windows hits the registers differently.

Unfortunately Broadcom doesn’t seem keen on releasing the documentation needed to write proper drivers to anyone other than Microsoft. Is this my imagination? Everyone else publishes the reference material, but Broadcom – I can’t find it.

If anyone can throw light on this one, please do. I’m still looking.

Update

Fitting a Realtek-based NIC in the Microserver and using that instead solves the problem. WOL just works. If you’re going to order one, remember it’s PCIe, not PCI, and that you really need one with a low-profile bracket option because a full-height card won’t fit.

 Further Update: See article here

Posted on

PAM authentication in PHP on FreeBSD

I have several groups of lusers who want to be able to set/change their mail vacation settings but aren’t up to using ssh to edit their .forward and .vacation.msg files. I thought I’d write a quick PHP application to allow them to do it in a luser-friendly way using a web browser. If this isn’t what PHP is for, I don’t know what good it is. The snag: you need to make sure the right user is editing the right file.

The obvious answer is to authenticate them with their mail user-name and password pair using PAM. (This is the system that will check user-name/password combinations against whatever authentication you see fit – by default /etc/passwd).

PHP has a module available for doing just this – it’s called “PAM” and there’s even a FreeBSD port of it you can install from /usr/ports/security/pecl-pam. If you want to use it, just “make” and “make install” – it’ll add it to the PHP extensions automatically, but don’t forget to restart Apache if you’re planning to use it there.

You’ll also have to configure PAM itself. This involves listing the authentication methods applicable to your module in /etc/pam.d/. In this case the php module will have the default name ‘php’ unless you’ve changed it in /etc/php.ini using a line like pam.servicename = "php";

Adding the above line above obviously does nothing as it’s the default, but it’s useful as a reminder of what the default is set to. I don’t like implicit defaults, but then again I don’t like a lot of the shortcuts taken by PHP.

The only thing you need to do to get it workings is to add a PAM module definition file called /etc/pam.d/php. The easy way to create this is copy an existing one, such as /etc/pam.d/ftp. This will be about right for most people, but read /etc/pam.d/README if you want to understand exactly what’s going on.

So – to test it. A quick PHP program such as the following will do the trick:

<?php
var_dump (pam_auth('auser','theirpassword',&$error,0));
print $error;
?>

If there’s an entry in /etc/passwd that matches then it’ll return true, otherwise false, and $error will contain the reason. Actually, it checks the file /etc/master.passwd – the one that isn’t world readable and therefore can contain the MD5 password hashes. And there’s the rub…

This works fine when run as root, but not as any other users; it always returns false. This makes it next to useless. It might be a bug in the code, but even if it isn’t it leads to interesting questions about security. For example, it would allow a PHP user to hammer away trying to brute-force guess passwords. I’ve seen it suggested to Linux users can overcome the need to run as root by making their shadow password group or world readable. Yikes!

If you’re going to use this with PHP inside Apache, you’re talking about giving the “limited” Apache user access to one of the most critical system files as far as security goes. I can see the LAMP lusers clamouring for for me to let them do this, but the answer is “no!” Pecl-pam is not a safe solution to this, especially on a shared machine. You could probably persuade it to use a different password file, but what’s the point? If the www user can read it, all web hosting users can and you might just as well read it from the disk directly (or use a database). PAM only makes sense for using system-wide passwords for authenticating real users.

I do now have a work-around: if you want your Apache PHP script to modify files in a user’s home directory you can do this using FTP. I’ve written some code to achieve this (not hard) and I’ll post it here if there’s any interest, and after I’ve decided it’s not another security nightmare.

 

Posted on

PHP PDO driver missing on FreeBSD

I got an email today – a FreeBSD 8.2 Web server installation with Apache and PHP 5.3 was missing its mySQL driver. No it wasn’t, I protested. But this error was appearing:

[error] [exception.CDbException] could not find driver
[error] [exception.CDbException] exception 'CDbException' with message 'CDbConnection failed to open the DB

“It’s not the database, it’s the PDO module”, came the explanation. Actually the pdo.so module was there and loaded, and it was installed in just the same way as it had been on earlier versions. So what was going on?

It turns out that as of PHP 5.3 there has been a change. You don’t get the important module – pdo_mysql – loaded when you compile /usr/ports/lang/php5-extensions. It’s not that it’s no longer checked by default – it’s not even there! I’m told this applies to Postgress too.

Further investigation revealed that PHP5.3 has a new native-mode driver (mysqlnd), and this is now outside the standard install. If you’re using sqlite for Persistant Data Objects, no problem, but if you’re using a more restrictive hosting account and therefore need to store them in a mySQL (or Postgress) database you seem to be out of luck.

For more information, see here.

However, the required module can still be found. Go to

/usr/ports/databases/php5-pdo_mysql

(If you’re using Postgress go to php5-pdo_pgsql instead – see Geoffrey McRae’s comment below).

make and make install what you find there. Also, make sure you compiled the original php5-extensions with mysql and mysqli, and don’t forget to restart Apache afterwards!

Note that it’s quite possible that none of the above relates to Linux. It’s to do with what gets installed from where with the FreeBSD ports tree.

Posted on

Large swap files on FreeBSD die with mystery “Killed” – howto add lots of swap space

Adding extra swap space to FreeBSD is easy, right? Just find a spare block storage device and run swapon with its name as an argument. I’ll put a step-by-step on how you actually do this at the end of the post in case this is news to you.

However, I’ve just found a very interesting gotcha, which could bite anyone running a 64-bit kernel and 8Gb+ of RAM.

From here we’re getting into the FreeBSD kernel – if you just want to know how to set up a lot of swap space, skip to the end…

I’ve been running a program to process a very large XML file into a large binary file – distilling 100Gb of XML into 1Gb of binary. This is the excuse for needing 16Gb of working storage (please excuse my 1970’s computer science terminology, but it’s a lot more precise than the modern “memory” and it makes a difference here).

I was using 2Gb of core and 8Gb of swap space, but this was too little so I added an extra 32Gb of swap file. Problem sorted? Well top and vmstat both reported 40Gb of swap space available so it looked good. However, on running the code it bombed out at random, with an enigmatic message “Killed” on the user console. Putting trace lines in the code narrowed it down to a random point while traversing a large array of pointers to pointers to the 15Gb heap, about an hour into the run. It looked for all the world like pointer corruption causing a Segmentation Fault or Bus Error, but if the process had a got that kind of signal it should have done a core dump, and it wasn’t happening. The output suggested a SIGKILL. But it wasn’t me sending it, and there were no other users logged in. Even a stack space error, which might have happened as qsort() was involved, was ruled out as the cause – and the kernel would have sent an ABORT, not a KILL in this case.

I finally tracked it down to a rather interesting “undocumented” feature. Within the kernel there is a structure called swblock in the John Dyson/Matthew Dillon VM handler, and a pointer called “swap” points to a chain of these these structures. Its size is limited by the value of kern.maxswzone, which you can tweak in /boot/loader.conf. The default (AMD64 8.2-Release) allows for about 14Gb of swap space, but because it’s a radix tree you’ll probably get a headache if you try to work it out directly. However, if you increase the swap space beyond this it’ll report as being there, but when when you try to use the excess, crunch!

Although this variable is tunable, it’s also hard-limited in include/param.h to 32M entries; each entry can manage 16 pages (if I’ve understood the code correctly). If you want to see exactly what’s happening, look at vm/swap_pager.c.

The hard limit to the size number of swblock entries is set as VM_SWZONE_SIZE_MAX in include/param.h. I have no idea why, and I haven’t yet tried messing with it as I have no need.

So, what was happening to my process? Well it was being killed by vm_pageout_oom() in vm/vm_pageout.c. This gets called when swap space OR the swblock space is exhausted, either in vm_pageout.c or swap_pager.c. In some circumstances it prints “swap zone exhausted, increase kern.maxswzone\n” beforehand, but not always. It’s effect is to find the largest running non-system process on the system and shoot it using killproc().

Mystery solved.

So, here’s how to set up 32Gb of USABLE swap space.

First, find your swap device. You can have as many as you want. This is either a disk slice available in /dev or, if you want to swap to a file, you need ramdisk to do the mapping. You can have as many swap devices as you like and FreeBSD will balance their use.

If you can’t easily add another drive, you’re best option is to add an extra swap file in the form of a ram disk on the existing filing system. You’ll need to be the root user for this.

To create a ram disk you’ll need a file to back it. The easy way to create one is using “dd”:

dd if=/dev/zero of=/var/swap0 bs=1G count=32

This creates a 32G file in /var filled with nulls – adjust as required.

It’s probably a good idea to make this file inaccessible to anyone other than root:

chmod 0600 /var/swap0

Next, create your temporary file-backed RAM disk:

mdconfig -a -t vnode -f /var/swap0 -u 0

This will create a device called /dev/md with a unit number specified by -u; in this case md0. The final step is to tell the system about it:

swapon /dev/md0

If you wish, you can make this permanent by adding the following to /etc/rc.conf:

swapfile="/var/swap0"

Now here’s the trick – if your total swap space is greater than 14Gb (as of FreeBSD 8.2) you’ll need to increase the value of kern.maxswzone in /boot/loader.conf. To check the current value use:

sysctl kern.maxswzone

The default output is:

kern.maxswzone: 33554432

That’s 0x2000000 32M. For 32Gb of VM I’m pretty sure you’d be okay with 0x5000000 (in round numbers), which translates to 83886080, so add this line to /boot/loader.conf (create the file if it doesn’t exist) and reboot.

kern.maxswzone="83886080"

 

Posted on

Spamassassin, spamd, FreeBSD and “autolearn: unavailable”

I recently built a mail server using FreeBSD 8.2 and compiled spamassassin from the current ports collection, to run globally. spamd looked okay and it was adding headers, but after a while I noticed the Baysian filtering didn’t seem to be working in spite of it having had enough samples through.

A closer look at the added headers showed “autolearn: no”, or “autolearn: unavailable” but never “autolearn: ham/spam”.

What was going on? RTFM and you’ll see spamassassin 3.0 and onwards has added three new autolearn return codes: disabled, failed and unavailable. The first two are pretty self-explanatory: either you’d set bayes_auto_learn 0 in the config file or there was some kind of error thrown up by the script. But I was getting the last one:

unavailable: autolearning not completed for any reason not covered above. It could be the message was already learned.

I knew perfectly well that the messages hadn’t already been learned, so was left with “any reason not covered by the above”. Unfortunately “the above” seemed to cover all bases already. There wasn’t any clue in /var/maillog or anywhere else likely.

I don’t much care for perl scripts, especially those that don’t work, so after an unpleasant rummage I discovered the problem. Simply put, it couldn’t access its database due to file permissions.

The files you need to sort are at /root/.spamassassin/bayes_* – only root will be able to write to them, not spamd – so a chmod is in order.

A better solution is to move the Bayesian database out of /root – /var would probably be more appropriate. You can achieve this by adding something like this to /etc/spamd.cf (which should link to /usr/local/etc/mail/spamassassin/local.cf):

bayes_path /var/spamassassin/bayes/bayes
bayes_file_mode 0666

I suspect that the lower-security Linux implementation avoids these problems by setting group-write-access as default, but FreeBSD, being a server OS, doesn’t. It’s also a bug in the error handling for the milter – it should clearly report as a “failed” and write something to the log file to tell you why.

You should be able to restart spamd after the edit with /usr/local/sbin/spamdreload, but to be on the safe side I use the following after shutting down Sendmail first.

/usr/local/etc/rc.d/spamass-milter restart
/usr/local/etc/rc.d/sa-spamd/restart

I don’t know if Sendmail can cope well with having spamass-milter unavailable, but why take the risk?

 

Posted on

OpenLDAP, Thunderbird and roving address books

IMAP is great. It lets you keep your mail synchronised between any number of machines, including webmail, and everything works just fine. The only snag is that your address book isn’t included. I’d always assumed this was what LDAP was for: a centralised directory of names, and other things, with the useful bit being the address book. Thunderbird, is my current favourite mail client on the basis that actaully works better than Outlook. It supports LDAP address books, and has offered to configure one for me many times. All I needed to do was configure slapd (the OpenLDAP server deamon) and point Thunderbird at it.

This blog entry isn’t a tutorial in configuring FreeBSD, OpenLDAP and Thunderbird to work together. I’m saving you from wasting a lot of your time trying. It does “work”, once you’ve sorted out schemas and got to grips with the arcane syntax of the configuration files and the hierarchical nature of the thing. It’s just that it’s useless even when it’s working because it’s READ-ONLY. Being able to add and amend entries in my address book is so fundamental to the nature of an address book that I didn’t bother to check that Thunderbird could do it. What’s the use of a read-only address book? Well there might be some point in a large organisation where a company-wide address book is needed, administered by a tame geek in the basement. For the rest of us it’s as fileofax with no pen.

So what are the good people at Mozilla playing at? The omission of read/write has been listed in their bug database for over ten years, and no one has tackled it. I thought about it for a while, but given the that Lightweight-DAP is a misnomer on a spectacular scale I thought again. Clearly no one who knows about LDAP actually likes it enough to want to help; either that or none actually understands it apart from the aforementioned geek in the basement, and he’s sitting tight because allowing users to edit address books might be detrimental to his pizza supply.

The time is right for a genuinely lightweight protocol for sharing address books in a sane and sensible manner; something like IMAP for addresses. I’m therefore writing one. Unfortunately I’m not so clued up on Thunderbird’s internal workings; if you are and wish to implement the front end please drop me a line and I’ll write a protocol and server that works.

Unfortunately this one issue is a killer app for Microsoft’s lightweight over-priced Mail system called Exchange. It’s a big of a dog (inflexible) but at least Microsoft seems to have got this fundamental functionality for sharing personal address books between mail clients sorted out. I believe it uses something similar LDAP underneath (along with IMAP for the mail itself); so it’s not impossible.

I’m very surprised to find myself having anything good to say about Outlook/Exchange Server. It might still be traumatised from the discovery that my assumption that the obvious LDAP solution was nothing of the sort. It’s just it’s so damn complex for no apparent reason that it gives the impression it must be great if you could only understand it.

Posted on

Fetchmail, Sendmail and oversized emails

There’s a tendency for lusers to try to email anything these days. If you though a few Gig of outgoing mail queue was enough you haven’t come across the luser who decided to email the contents of a CD (uncompressed) to all her friends. Quite what they’d have made of their iPhone trying to download it I’ll never know.

Sendmail has a method for limiting emails to a sensible size. As a reminder, inside host.example.com.mc you need to add:

# The following sets the maximum message to 5Mb - otherwise it's infinite
define(`confMAX_MESSAGE_SIZE', `5242880')

Then run “make” and “make install” and “make restart”. This will generate the sendmail.cf (and any hashmaps) before restarting. The bit you always forget when changing .mc files is the “make install”. This is all for FreeBSD – Linux types, please do it your own way.

So this is great – anyone sending an over-sized email is bounced from their server, and local users submitting email will be similarly clipped into the world of sane and sensible (if you regard something as large as 5Mb as sensible for an email).

But I came across one interesting issue recently and it could happen to you, too, if you’re using fetchmail.

For those who haven’t come across it before, fetchmail pulls emails from a POP3 box and delivers them to local users – dropping them into your local MTA by default. This is reasonable, as everything then goes through the spam filtering, procmail and anything else you have defined. It’s really useful for legacy situations where someone’s ended up with a POP3 box somewhere and you need to integrate it with the rest of their mail.

Fetchmail does plenty more besides, and has a config file to match the functionality. Presumably as a reaction against the complexity of the sendmail.cf syntax, this one tries to operate in plain English. I’ve never quite figured out the full syntax, but it’s designed to be “flexible” and figure out what you’re trying to say. Personally I don’t think it succeeds in being any more friendly then sendmail.cf in spite of being on the other end of the spectrum.

Anyway, the fun comes when fetchmail downloads an over-sized email from the POP3 box and delivers it locally via Sendmail. Sendmail will reject it, and send a bounce back to the original sender. So far, so good but f Sendmail is running as a cron job every five minutes, the luser gets a bounce back every five minutes because the outsized mail is stuck in the POP3 box. Opps! It may serve them right, but they shouldn’t be allowed to suffer for too long.

Fortunately one of fetchmail’s many options allows you to control the maximum download size, if you could figure out the syntax. It’s available as a command-line option –l , but if you prefer to keep things in the .fetchmailrc file (the best plan) you’ll need to proceed as per the following example. They keywords are “limit” and “limitflush”.

  • local-postmster-account is the login for your local postmaster – undelivered emails go there.
  • pop3.isp.co.uk – mail server with the POP3 box
  • users-domain.co.uk – Domain name who’s email ends up in POP3 box above
  • pop3-username, pop3-password – what you use to log into the POP3 box
  • Tom, Dick and Harry are local mailboxes, with tom being the default.
    set postmaster local-postmster-account

    poll pop3.isp.co.uk proto pop3 aka users-domain.co.uk no envelope no dns:
    user "pop3-username", with password "pop3-password",
    limit 5242368 limitflush to

    dick
    "dick@users-domain.co.uk " = dick
    "richard@users-domain.co.uk " = dick

    harry
    "harry@users-domain.co.uk " = harry

    tom
    "tom@users-domain.co.uk" = tom
    "*@ users-domain.co.uk " = tom

    here

    This isn’t intended as a tutorial in writing .fetchmailrc files – only an example of the use of limit and limitflush.

    So what’s going on? The limit keyword must be part of the poll statement, and is followed by the size (in bytes) of the maximum email to be retrieved. In the example it’s 512 bytes less than the 5Mb used in Sendmail (I feel I need a bit of slack on a boundary condition; it may be okay if they’re identical but I why push your luck?)

    Please read the fetchmail documentation for full details (although it’s light on examples). With just the “limit” keyword in use, over-sized mails will be left I the POP3 box. The following “limitflush” keyword will silently delete over-sized emails so they don’t bother you again. You may not want to do this! If you don’t, someone will have to retrieve or delete the emails form the POP3 box manually.

    Note that putting a limit on the download will prevent the bounce messages going to the original sender as it won’t get as far as sendmail.