Posted on

Rename file extensions in UNIX/Linux/FreeBSD

I had a directory with thousands of files from a Windoze environment with inconsistent file extension  Some ended in .hgt, others in .HGT. They all needed to be in lower case, for some Windows-written cross-compiled software to find them. UNIX is, of course, case-sensitive on such things but Windoze with its CP/M-like file system used upper-case only, and when the shift key was invented, decided to ignore case.

Anyway, rather than renaming thousands of files by hand I thought I’d write a quick script. Here it is. Remember, the old extension was  .HGT, but I needed them all to be .hgt:

for oldname in `find . -name "*.HGT"`
do
newname=`echo $oldname | tr .HGT .hgt`
mv $oldname $newname
done

Pretty straightforward  but I’d almost forgotten the tr (translate) command existed, so I’m now feeling pretty smug and thought I’d share it with the world. It’ll do more than a simple substitution – you could use “[A-Z] [a-z]” to convert all upper case characters in the file to lower case, but I wanted only the extensions done. I could probably have used -exec on the find command, but I’ll leave this as an exercise for the reader!

It could me more compact if you remove the $newname variable and substitute directly, but I used to have an echo line in there giving me confirmation I was doing the right thing.

 

Posted on

FreeBSD, Wake-on-LAN and HP Microservers – WOL compatible Ethernet

I’ve been having some difficulties getting Wake-on-LAN (WOL) to work with an HP Microserver thanks to its Broadcom Ethernet adapter not doing the business with the FreeBSD drivers – setting WOL in the Microserver BIOS doesn’t have any effect. I’m pleased to report a solution that works.

The on-board Broadcom Ethernet adaptor still refuses to play ball, for reasons described in my earlier post. The pragmatic solution is to use a better supported chip set and I’ve had no difficulties with Realtek (at the low end of the market) so it was an obvious choice. Just bung a cheap Realtek-based card in and use it as a remote “on” switch – what could possibly go wrong?

First off, the HP Microserver has PCI-Express slots, but weird looking ones. I’d assumed one was PCI when I’d glanced it, but it’s a PCIe 1-channel slot with something strange behind it – possibly a second 1-channel slot. The documentation says its for a remote management card; presumably one which doesn’t need access to the back. There’s a 16-channel PCIe next to it.All very curious but irrelevant here. The point is that you’ll need a PCIe Ethernet card – a surplus 100M PCI one with a well supported, bog-standard chip, won’t do. The PCIe cards tend to be 1Gb, and are therefore not as cheap.

The first card I bought was a TP-Link TG-3458, which has standard Realtek 8168B adapter chip. Or at least mine did; I note that there is a Mk2 version out there. Mine’s definitely a revision 1.2 PCB, but if you buy one now it may have the newer chip (which is a problem – read on below). Anyway, this Mk1 card worked like a charm. On sending it the magic packet and the Microserver bursts in to life. There’s only one snag: It has a full-height bracket and the Microserver has a half-height slot, so you have to leave the card floating in its socket. This works okay as long as no one trips over the cable.

My second attempt was an Edimax EN-9260TX-E, ordered because it was (a) cheap-ish; (b) had a Realtek chip; and (c) had the all-important half-height bracket. It fitted in the Microserver all right, but refused to act on a WOL, at least to begin with…

It turns out there was a little bug-ette in the driver code (prior to 8.3 or 9.1), spotted and fixed by the maintainer about a year ago. If you want to fix it yourself the patch is here. I decided I might as well use the latest drivers rather than re-working those shipped with 8.2, so pulled them, compiled a new if_re.ko and copied it to /boot/kernel in place of the old one. It didn’t work. Ha! Was I naive!

Further investigation revealed that it was completely ignoring this kernel module, as it was using a driver compiled in to the kernel directly. There was no point having the module there, all it does is trick you in to believing that it’s installed. I only realised “my” mistake when, to my astonishment, removing the file completely didn’t disable the network interface. I solved the problem by compiling a new kernel with the built-in Realtek driver commented out, and I’m currently loading the new driver specifically in loader.conf. It works a treat. I could have changed the kernel Realtek driver, but while it’s under review it’s easier to have it loaded separately. Incidentally, the driver is for 9.1 onwards but it works fine on 9.0 so far.

The next task is to fix the Broadcom driver so it works. I may be gone some time…

Posted on

Lighttpd in a FreeBSD Jail (and short review)

Lighttpd is an irritatingly-named http daemon that claims to be light, compared to Apache. Okay, the authors probably have a point although this puppy seems to like dragging perl in to everything and there’s nothing minuscule about that.

I thought it might be worth a look, as Apache is a bit creaky. It’s configuration is certainly a lot simpler than httpd.conf,although strangely, you tend to end up editing the same number of lines. But is it lighter? Basically, yes. If you want the figures it’s currently running (on AMD64) a size of 16M compared to Apache httpd instances of 196M.

But we’re not comparing like for like here, as Lighttpd doesn’t have PHP; only CGI. If you’re worried about that being slow, there’s FastCGI, which basically keeps instances of the CGI program running and Lightttpd hands tasks off to an instance when they crop up. Apache can do this (there’s the inevitable mod), but most people seem happy using the built-in PHP these days so I don’t think FastCGI is very popular. It’s a pity, as I’ve always felt CGI is under-rated and I’m very comfortable passing off to programs written in ‘C’ without there being an noticeable performance issues. Using CGI to run a perl script and all that entails is horrendous, of course. But FastCGI should level the playing field and allow instances of perl or any other script language of your dreams to remain on standby in much the same way PHP currently remains on standby in Apache. That doesn’t make perl or PHP good, but it levels their use with PHP on Apache, giving you the choice. And you can also choose  high-performance ‘C’.

This is all encouraging, but  I haven’t scrapped Apache just yet. One simple problem, with no obvious solution, is the lack of support for the .htaccess file much loved by the web developers and their content management systems. Another worry for me is security. Apache might be big and confusing, but it’s been out there a long time and has a good track record (lately). If it has holes, there are a lot of people looking for them.

Lighttpd doesn’t have a security pedigree. I’m not saying it’s got problems; it’s just that it hasn’t been thrashed in the same way as Apache and I get the feeling that the development team is much smaller. Sometimes this helps, as it’s cleaner code, but it’s statistically less likely to have members adept at spotting security flaws too. I’m a bit concerned about the FastCGI servers all running on the same level, for example.

Fortunately you can mitigate a lot of security worries by running in a jail on FreeBSD (it will also chroot on Linux, giving some degree of protection). It was fairly straightforward to compile from the ports collection, but it does have quite a few dependencies. Loads of dependencies, in fact. I saw it drag m4 in for some reason! Also the installation script didn’t work for me but it’s easy enough to tweak manually (find the directory with the script and run make in it to get most of the job done). The other thing you have to remember is that it will store local configurations in /usr/local on BSD, instead of the base system directories.

To get it running you’ll need to edit  /usr/local/etc/lighttpd/lighttpd.conf, and if you’re running in a jail be sure to configure the IP addresses to bind to correctly. Don’t be fooled: There’s a line at the bottom that sets the IP address and port but you must find the entry server.bind in the middle of the file and set that to the address you’ve configured for the jail to have passed through. This double-entry a real pooh trap, especially as it tries to bind to the loopback interface and barfs with a mysterious message. Other than that, it just works – and when it’s in the jail it will happily co-exist with Apache.

I’ve got it running experimentally on a production server now, and I’ve also cross-compiled to ARM and it runs on Raspberry Pi (still on FreeBSD), but it was more fun doing that with Apache.

When I get time I’ll do a full comparison with Hiawatha.

Posted on

Using ISO CD Images with Windows – Burn.Now problems

When CD-R drives first turned up you needed special software to write anything – originally produced by Adaptec but they were soon overtaken by Nero, with NTI and Ulead having lower cost options. Now, when you get a PC it will usually come with one of the above bundled, and Microsoft has added the functionally to Windows since XP (for CD, if not DVD). This is not good news for the independent producers, but Microsoft’s offering doesn’t quite cut the mustard, so most people will want something better.

My new Lenovo PC came bundled with Corel Burn.Now. Corel recently bought the struggling Ulead, and this is fundamentally the same product as Ulead burn.now. Unfortunately Burn.Now is also pretty feeble – it just can’t do the basics.

To duplicate a CD you need to copy all the data on it. Pretty obvious really. If you’re not copying drive-to-drive it makes sense to copy the data to a .ISO image on your hard disk. You can then transfer it to another machine, back it up or whatever; and write it to a new blank disk later. Burn.Now will create a CD from an ISO image, but if you ask it to copy a disk it uses its own weird and whacky .ixb format. Some versions of Burn.Now gave you the choice, but not the new Corel. It’s .ixb or nothing. This matters, because whilst everyone can write .ISO files, only Burn.Now can write from  .IXB format.

Burn.Now is crippled. What about Microsoft’s current built-in options? You can actually write an ISO image using Windows 7 – just right-click on the file and select “Burn disc image”. Unfortunately there is no way to create such a file with Windows. To do this you need add Alex Feinman’s excellent ISO Recorder, which basically does the opposite: Right-click on the CD drive and select Create Image from CD/DVD.

Unfortunately ISO Recorder doesn’t read all disks – it won’t handle Red Book for a start. This is a bit of a limitation – was its author, Mr Feinman concerned about music piracy? Given Windows Media Player can clone everything on an Audio CD without difficulty, his conciousness efforts won’t make a lot of difference.

So – Windows is its usual painful self. If you just want to simply create an image of a CD or DVD with no bells and whistles, go to UNIX where it’s been “built in” since the 1980’s (when CD-ROMs first appeared). Just use the original “dd” command:

# dd if=/dev/acd0 of=my-file-name.iso bs=2048

An ISO file is simply a straight copy of the data on the disk, so this will create one for you. You can write it back using:

# burncd -f /dev/acd0 data my-file-name.iso fixate
Or
# cdrecord dev=1,2,3 my-file-name.iso

Burncd is built in to FreeBSD (and Linux, IIRC), but only works with atapi drives. In the example it assumes the CD recorder is on /dev/acd0 (actually the default).

Cdrecord works with non atapi drives to, but has to be built from ports on FreeBSD and for other platforms it’s available here – along with lots of other good stuff. The example assumes the device is 1,2,3 – which is unlikely! Run cdrecord -scanbus to locate the parameters for your drive.

Once you have your ISO file, of course, you could use Windows to write it. The choice depends on whether you have strongly held views on whether Windows is a worthy desktop operating system. Corel Burn.Now is, however, a long way from being a worth CD/DVD writing utility.

Posted on

Can’t get PuTTY and FreeBSD with OpenSSH to do a Certificate Login – Myths

Following yesterday’s post about issues getting “Server Refused Our Key” errors when trying to use PuTTY to log in to FreeBSD with a certificate, I thought I’d just lay to rest a few myths I’ve seen on various web sites where people have tried to explain how to do this. It’s easy to see how these myths develop – I’ve laboured for years under the misapprehension that I needed to do something or other when it was just a coincidence it had started working the first time the idea came to me. So here goes with a few of the myths. If you’re not getting this to work, it’s not for one of these reasons:

Myth: You need to specify 0600 permissions for the authorized_keys file (or the .ssh directory)

Simply not true. It may be a good idea to stop others from reading your keys, although they are “public” keys and won’t let anyone else in anyway (unless a they have a suitable cracking tool and a lot of processing power – and I mean a lot). Only your private key needs to be a secret. The only stipulation is that they must only writeable by the user – 0644 is okay, 0664 or 0666 isn’t.

But as I mentioned yesterday, you MUST ensure that your home directory is also not world-writable! You mustn’t have 0777 permissions! 0755 is okay, as is 0711. I’ve not seen this documented anyway, but it’s true for FreeBSD 7.0 to 9.0.

Myth: OpenSSH requires the authorized_keys file to be owned by the user trying to log in

Again no – it simply doesn’t. It has to be readable to that user (not just root) – this may be because it’s world readable or group readable for the user in question. It might as well be owned by root:wheel as long as it’s Other read bit is set.

Myth: If you’re using SSH2, you need a file called authorized_keys2

This might be true on some installations, but not current ones! I’ve no reason to believe that this file would even be considered, never mind required. The file used is defined in the /etc/ssh/sshd_config, and on current versions of FreeBSD (7.0-9.0) it’s definitely authorized_keys

Myth: You must generate the keys using the OpenSSH keygen utility on FreeBSD – puttygen doesn’t work

Well, there’s a bit of truth in this, but not much. Put simply, the format is different, but this only extends as far as the header and comment.

OpenSSH keys look like this:

ssh-rsa AAAAB3NzaC1y… very long line … sXi+fF noone@example.com

PuTTYGen Keys look like this:

---- BEGIN SSH2 PUBLIC KEY ----
Comment: "no one@example.com"
AAAAB3NzaC1y … long line, possibly with breaks … sXi+fF
---- END SSH2 PUBLIC KEY ----

You can convert one to the other using any text editor of your choice, as long as it handles long lines properly (like vi).

I can see there could be all sorts of fun and games if you simply cut/pasted these end ended up with extra line breaks, spaces or truncation – but the key data and its encoding is exactly the same, and that’s the bit that makes it work or not.

If you generate your key using OpenSSH tools you will need to load it into PuTTY Gen and write a Private .ppk key on your Windoze box. Or not. It’s just a text file and you could put the appropriate wrapper on it, but you might as well just use PuTTY Gen.

Myth: You need to edit /etc/ssh/sshd_conf to enable certificate login

No you don’t. The default values as shipped work just fine. Because the file consists of commented out lines of parameters with their default values, I suspect people though that some have been confused about whether the ‘#’ needed to be removed before the parameter came in to effect. They don’t – you only need to remove the comment if you want to change the default value. If you do remove the comment, but don’t edit the value, it’ll make no difference to anything.

What’s Real

In my experience, problems are almost always down to either directory permissions (see above) or errors transcribing public keys from one machine to another – and chaos and confusion caused by the abovementioned myths!

 

Posted on

PuTTY, FreeBSD and SSH certificate logins

I’ve just gone crazy trying to figure out why PuTTY kept getting a “Server Refused Our Key” error when I tried to log in to a host using a certificate for the first time. Looking around the web, there are a lot of interesting theories about how to generate the certificates, and out of desperation I tried them all – nothing worked. So, for what it’s worth, here’s what does.

Generate your certificate on FreeBSD using the OpenSSH utility:

ssh-keygen -t rsa

With the default options this will create a couple of files in the .ssh directory within your home directory, and by default they’ll be called “id_rsa” and “id_rsa.pub”. In other words, if you’re user ID is fred the files will be in /usr/home/fred/.ssh/ with the above names. One’s private, the other is public.

You need to add the public key to the list of authorised keys in the .ssh directory:

cat id_rsa.pub >> ~/.ssh/authorized_keys

(The name authorized_keys with the American spelling is set in /etc/ssh/sshd_config)

Next you need to get the private key back to the machine running PuTTY. It’s just text – you can cut/paste it into a text editor and save it. For PuTTY to use it, however, it needs to be converted in to PuTTY’s own format, which you do using the PuTTY Key Generator, puttygen.exe. Run this, click on the Load button and read in your text file, then use the Save Private button to put write the .ppk file somewhere safe. You may wish to set a passphrase on it if there’s any chance someone else can get hold of it!

You may now get rid of the id_rsa.* files on the FreeBSD host, although you might want to add the public key to more than one user on more than one host – it’s a “public” key so there’s no harm in using it all over the place.

It is possible to use PuttyGen to make the keys and copy them to the FreeBSD host instead. A lot of people seem to have had trouble with this in the past (myself included), and it’s probably easier not to, especially if you’re going to use the keys in OpenSSH format for other purposes on the FreeBSD host anyway.

You’ll see a lot about setting the files in .ssh in some very restricted ways – basically all you need to do is ensure that they’re only writable by you. You can make your .ssh directory only readable by you if you wish but it won’t stop it from working. Also, the default /etc/ssh/sshd_config files is fine, and you don’t need to uncomment anything (in spite of what you might read). The default settings are all good, and all commented out, as it says on the top of the file. (Not quite true now – see 2024 update below)

Now, here’s the trick! What will cause a problem, as I eventually figured out, is if your home directory is writable by others. Don’t ask me how or why this should be true, but I tried this after I’d tried eliminating everything else on comparing working and non-working boxes. I know this for sure with FreeBSD 8.1 – ensure your home directory is drwxr-xr-x (or possibly less).

The final stage is to set up a session profile in PuTTY. This isn’t a tutorial for PuTTY, so I’ll be brief. In the options category open to Connection/Data and set the auto-login username you wish to use (if you haven’t already). Then under Connection/SSH/Auth select the private (.ppk) file you want to use. Remember, you can use this file with as many hosts and user accounts as you’ve added the public key to the .ssh/authorized_keys file. Save the session, and that’s it done. If it doesn’t do it for you, take a look in /var/log/auth.log.

Update 2024:
And finally, twelve years later, there’s a problem. newer versions of SSH will barf at RSA keys. You’ll get a “The server refused our key” message and something like this in auth.log…

 sshd[1539]: userauth_pubkey: signature algorithm ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]

Don’t worry – there’s a quick fix. In /etc/ssh/sshd_config add the following line somewhere that makes sense.

PubkeyAcceptedAlgorithms +ssh-rsa

You might want to use soemethign other than RSA keys going forward, but this is an update to a 2012 article – watch out for a new one.

Posted on

HP Microserver and WOL

Update: See article here

 

They just don’t seem to work. I’ve spent an annoying hour or so trying to get WOL to work with an HP Microserver – no joy whatsoever. I assumed it must be my code until I tried it on a few other machines but they worked just fine.

Now most of my machines are Realtek whereas HP are using Broadcom (as do the Dells). I’m not saying there’s anything wrong with Broadcom, but whenever I have a weird network problem they have a habit of being at the heart of it. Is it my magic packet? As far as I know it’s supposed to be 48-bits of ‘1’ followed by sixteen copies of the MAC address. Does it need a secure-on password? If so, how come you can’t set one in the BIOS.

I’ve asked an HP server expert: “Update the BIOS”. Perhaps, but these are brand new machines of an established design. They either turn on when they receive the packet, or they don’t work, and I can’t believe HP didn’t test them. Then again…

I’m told that these do support WOL on Windows, but not if you’re running anything else. On the face of it this is bonkers. Why should the OS the powered-off drive affect anything. The machine is off; the OS isn’t running. Well here’s a theory – before Windows shuts off it puts something in a register on the Broadcom chip to leave it in a WOL state. With the wrong drivers this doesn’t happen. Setting it in the BIOS doesn’t help, because it’s erased by the OS driver. The BIOS doesn’t restore it as the power is killed, but Windows hits the registers differently.

Unfortunately Broadcom doesn’t seem keen on releasing the documentation needed to write proper drivers to anyone other than Microsoft. Is this my imagination? Everyone else publishes the reference material, but Broadcom – I can’t find it.

If anyone can throw light on this one, please do. I’m still looking.

Update

Fitting a Realtek-based NIC in the Microserver and using that instead solves the problem. WOL just works. If you’re going to order one, remember it’s PCIe, not PCI, and that you really need one with a low-profile bracket option because a full-height card won’t fit.

 Further Update: See article here

Posted on

PAM authentication in PHP on FreeBSD

I have several groups of lusers who want to be able to set/change their mail vacation settings but aren’t up to using ssh to edit their .forward and .vacation.msg files. I thought I’d write a quick PHP application to allow them to do it in a luser-friendly way using a web browser. If this isn’t what PHP is for, I don’t know what good it is. The snag: you need to make sure the right user is editing the right file.

The obvious answer is to authenticate them with their mail user-name and password pair using PAM. (This is the system that will check user-name/password combinations against whatever authentication you see fit – by default /etc/passwd).

PHP has a module available for doing just this – it’s called “PAM” and there’s even a FreeBSD port of it you can install from /usr/ports/security/pecl-pam. If you want to use it, just “make” and “make install” – it’ll add it to the PHP extensions automatically, but don’t forget to restart Apache if you’re planning to use it there.

You’ll also have to configure PAM itself. This involves listing the authentication methods applicable to your module in /etc/pam.d/. In this case the php module will have the default name ‘php’ unless you’ve changed it in /etc/php.ini using a line like pam.servicename = "php";

Adding the above line above obviously does nothing as it’s the default, but it’s useful as a reminder of what the default is set to. I don’t like implicit defaults, but then again I don’t like a lot of the shortcuts taken by PHP.

The only thing you need to do to get it workings is to add a PAM module definition file called /etc/pam.d/php. The easy way to create this is copy an existing one, such as /etc/pam.d/ftp. This will be about right for most people, but read /etc/pam.d/README if you want to understand exactly what’s going on.

So – to test it. A quick PHP program such as the following will do the trick:

<?php
var_dump (pam_auth('auser','theirpassword',&$error,0));
print $error;
?>

If there’s an entry in /etc/passwd that matches then it’ll return true, otherwise false, and $error will contain the reason. Actually, it checks the file /etc/master.passwd – the one that isn’t world readable and therefore can contain the MD5 password hashes. And there’s the rub…

This works fine when run as root, but not as any other users; it always returns false. This makes it next to useless. It might be a bug in the code, but even if it isn’t it leads to interesting questions about security. For example, it would allow a PHP user to hammer away trying to brute-force guess passwords. I’ve seen it suggested to Linux users can overcome the need to run as root by making their shadow password group or world readable. Yikes!

If you’re going to use this with PHP inside Apache, you’re talking about giving the “limited” Apache user access to one of the most critical system files as far as security goes. I can see the LAMP lusers clamouring for for me to let them do this, but the answer is “no!” Pecl-pam is not a safe solution to this, especially on a shared machine. You could probably persuade it to use a different password file, but what’s the point? If the www user can read it, all web hosting users can and you might just as well read it from the disk directly (or use a database). PAM only makes sense for using system-wide passwords for authenticating real users.

I do now have a work-around: if you want your Apache PHP script to modify files in a user’s home directory you can do this using FTP. I’ve written some code to achieve this (not hard) and I’ll post it here if there’s any interest, and after I’ve decided it’s not another security nightmare.

 

Posted on

PHP PDO driver missing on FreeBSD

I got an email today – a FreeBSD 8.2 Web server installation with Apache and PHP 5.3 was missing its mySQL driver. No it wasn’t, I protested. But this error was appearing:

[error] [exception.CDbException] could not find driver
[error] [exception.CDbException] exception 'CDbException' with message 'CDbConnection failed to open the DB

“It’s not the database, it’s the PDO module”, came the explanation. Actually the pdo.so module was there and loaded, and it was installed in just the same way as it had been on earlier versions. So what was going on?

It turns out that as of PHP 5.3 there has been a change. You don’t get the important module – pdo_mysql – loaded when you compile /usr/ports/lang/php5-extensions. It’s not that it’s no longer checked by default – it’s not even there! I’m told this applies to Postgress too.

Further investigation revealed that PHP5.3 has a new native-mode driver (mysqlnd), and this is now outside the standard install. If you’re using sqlite for Persistant Data Objects, no problem, but if you’re using a more restrictive hosting account and therefore need to store them in a mySQL (or Postgress) database you seem to be out of luck.

For more information, see here.

However, the required module can still be found. Go to

/usr/ports/databases/php5-pdo_mysql

(If you’re using Postgress go to php5-pdo_pgsql instead – see Geoffrey McRae’s comment below).

make and make install what you find there. Also, make sure you compiled the original php5-extensions with mysql and mysqli, and don’t forget to restart Apache afterwards!

Note that it’s quite possible that none of the above relates to Linux. It’s to do with what gets installed from where with the FreeBSD ports tree.

Posted on

Large swap files on FreeBSD die with mystery “Killed” – howto add lots of swap space

Adding extra swap space to FreeBSD is easy, right? Just find a spare block storage device and run swapon with its name as an argument. I’ll put a step-by-step on how you actually do this at the end of the post in case this is news to you.

However, I’ve just found a very interesting gotcha, which could bite anyone running a 64-bit kernel and 8Gb+ of RAM.

From here we’re getting into the FreeBSD kernel – if you just want to know how to set up a lot of swap space, skip to the end…

I’ve been running a program to process a very large XML file into a large binary file – distilling 100Gb of XML into 1Gb of binary. This is the excuse for needing 16Gb of working storage (please excuse my 1970’s computer science terminology, but it’s a lot more precise than the modern “memory” and it makes a difference here).

I was using 2Gb of core and 8Gb of swap space, but this was too little so I added an extra 32Gb of swap file. Problem sorted? Well top and vmstat both reported 40Gb of swap space available so it looked good. However, on running the code it bombed out at random, with an enigmatic message “Killed” on the user console. Putting trace lines in the code narrowed it down to a random point while traversing a large array of pointers to pointers to the 15Gb heap, about an hour into the run. It looked for all the world like pointer corruption causing a Segmentation Fault or Bus Error, but if the process had a got that kind of signal it should have done a core dump, and it wasn’t happening. The output suggested a SIGKILL. But it wasn’t me sending it, and there were no other users logged in. Even a stack space error, which might have happened as qsort() was involved, was ruled out as the cause – and the kernel would have sent an ABORT, not a KILL in this case.

I finally tracked it down to a rather interesting “undocumented” feature. Within the kernel there is a structure called swblock in the John Dyson/Matthew Dillon VM handler, and a pointer called “swap” points to a chain of these these structures. Its size is limited by the value of kern.maxswzone, which you can tweak in /boot/loader.conf. The default (AMD64 8.2-Release) allows for about 14Gb of swap space, but because it’s a radix tree you’ll probably get a headache if you try to work it out directly. However, if you increase the swap space beyond this it’ll report as being there, but when when you try to use the excess, crunch!

Although this variable is tunable, it’s also hard-limited in include/param.h to 32M entries; each entry can manage 16 pages (if I’ve understood the code correctly). If you want to see exactly what’s happening, look at vm/swap_pager.c.

The hard limit to the size number of swblock entries is set as VM_SWZONE_SIZE_MAX in include/param.h. I have no idea why, and I haven’t yet tried messing with it as I have no need.

So, what was happening to my process? Well it was being killed by vm_pageout_oom() in vm/vm_pageout.c. This gets called when swap space OR the swblock space is exhausted, either in vm_pageout.c or swap_pager.c. In some circumstances it prints “swap zone exhausted, increase kern.maxswzone\n” beforehand, but not always. It’s effect is to find the largest running non-system process on the system and shoot it using killproc().

Mystery solved.

So, here’s how to set up 32Gb of USABLE swap space.

First, find your swap device. You can have as many as you want. This is either a disk slice available in /dev or, if you want to swap to a file, you need ramdisk to do the mapping. You can have as many swap devices as you like and FreeBSD will balance their use.

If you can’t easily add another drive, you’re best option is to add an extra swap file in the form of a ram disk on the existing filing system. You’ll need to be the root user for this.

To create a ram disk you’ll need a file to back it. The easy way to create one is using “dd”:

dd if=/dev/zero of=/var/swap0 bs=1G count=32

This creates a 32G file in /var filled with nulls – adjust as required.

It’s probably a good idea to make this file inaccessible to anyone other than root:

chmod 0600 /var/swap0

Next, create your temporary file-backed RAM disk:

mdconfig -a -t vnode -f /var/swap0 -u 0

This will create a device called /dev/md with a unit number specified by -u; in this case md0. The final step is to tell the system about it:

swapon /dev/md0

If you wish, you can make this permanent by adding the following to /etc/rc.conf:

swapfile="/var/swap0"

Now here’s the trick – if your total swap space is greater than 14Gb (as of FreeBSD 8.2) you’ll need to increase the value of kern.maxswzone in /boot/loader.conf. To check the current value use:

sysctl kern.maxswzone

The default output is:

kern.maxswzone: 33554432

That’s 0x2000000 32M. For 32Gb of VM I’m pretty sure you’d be okay with 0x5000000 (in round numbers), which translates to 83886080, so add this line to /boot/loader.conf (create the file if it doesn’t exist) and reboot.

kern.maxswzone="83886080"