Posted on

How to hack UNIX and Linux using wildcards

Leon Juranic from Croatian security research company Defensecode has written a rather good summary of some of the nasty tricks you can play on UNIX sysadmins by the careful choice of file names and the shell’s glob functionality.

The shell is the UNIX/Linux command line, and globbing is the shell’s wildcard argument expansion. Basically, when you type in a command with a wildcard character in the argument, the shell will expand it into any number of discrete arguments. For example, if you have a directory containing the files test, junk and foo, specifying cp * /somewhere-else will expand to cp test junk foo /somewhere else when it’s run. Go and read a shell tutorial if this is new to you.

Anyway, I’d thought most people knew about this kind of thing but I was probably naïve. Leon Juranic’s straw poll suggests that only 20% of Linux administrators are savvy.

The next alarming thing he points out is as follows:
Another interesting attack vector similar to previously described 'chown'
attack is 'chmod'.
Chmod also has --reference option that can be abused to specify arbitrary permissions on files selected with asterisk wildcard.

Chmod manual page (man chmod):
--reference=RFILE
use RFILE's mode instead of MODE values

 

Please generate and paste your ad code here. If left empty, the ad location will be highlighted on your blog pages with a reminder to enter your code. Mid-Post

Oh, er! Imagine what would happen if you created a file named “–reference=myfile”. When the root user ran “chmod 700 *” it’d end up setting the access permissions on everything to match those of “myfile”. chown has the same option, allowing you to take ownership of all the files as well.

It’s funny, but I didn’t remember seeing those options to chmod and chown. So I checked. They don’t actually exist on any UNIX system I’m aware of (including FreeBSD). On closer examination it’s an enhancement of the Linux bash shell, where many a good idea turns out to be a new vulnerability. That said, I know of quite a few people using bash on UNIX.

This doesn’t detract from his main point – people should take care over the consequences of wildcard expansion. The fact that those cool Linux guys didn’t see this one coming proves it.

This kind of stuff is (as he acknowledges) nothing new. One of the UNIX administrators I work with insists on putting a file called “-i” in every directory to stop wild-card file deletes (-i as an argument to rm forces an “Are you sure?” prompt on every file. And then there’s the old chestnut of how to remove a file with a name beginning with a ‘-‘. You can easily create one with:
echo test >-example
Come back tomorrow and I’ll tell you how to get rid of it!

Update 2nd July:

Try this:
rm ./-example

Posted on

Restoring cPanel backup to system without cPanel

cPanel is a web front end for “reseller” hosting accounts, and it’s very popular with web designers reselling hosting services. It’s very simple to use, and allows the web designers to set up virtual hosting accounts without giving them any real control over the server – self-service and fool proof. It’s also an expensive thing to license. It makes sense for a self-service low-cost hosting provider, where the customers do all the work, but for small-scale or “community” hosting providers you’re talking big money.

I’ve just had to rescue a number of web sites from a developer using one of these hosting services, and they’ve got a lot of sites. And the only access to the virtual server is through cPanel (and FTP to a home directory). I logged in to cPanel and there’s an option to create a backup of everything in one big tarball, and this looked like just what I wanted to get them all at once. However, it was designed to upload and unpack in another cPanel environment.

Getting out the home directories is pretty straightforward. They end up in a directory called “homedir”, and you just move it to where you want them – i.e. ~username/www/. But how about restoring the dump of the MySQL databases. Actually, that’s pretty simple too. They’re in a directory called “mysql”, but instead of it being one big dump, each is in it’s own file – and without the create commands, which are in another with the extension “.create” instead of “.sql”. Loading them all manually is going to be a time-wasting PITA, but I’ve worked out the the following shell script will do it for you if you run in while in the backup’s mysql directory:

for name in `find . -name “*.create”`; do
cat $name `echo $name | sed s/.create/.sql/` | mysql
done

You obviously have to be in the directory with the files (or edit find’s specification) and logged in as root (or add the root login as a parameter to the mysql utility).

You’ll also want to set the user/password combination on these. The tarball will have a file called mysql.sql in its root directory. Just feed it in thus:

mysql < mysql.sql

Please be aware that I figured this out looking at the files in the dump and NOT by reading any magic documentation. It works on the version of cPanel I encountered, and I was restoring to FreeBSD. By all means add a comment if you have a different experience when you try it, and don’t go this way if you’re not sure how to operate a MySQL database or you could do a lot of damage!

The final hurdle is configuring Apache for all these new sites. cPanel creates a directory in the dump called “userdata”, and this seems to contain a file with information about each web site. I decided to automate and wrote the following script:


#!/bin/sh

# Convert cPanel dump of "userdata" in to a series of Apache .conf files
# (c) F J Leonhardt 17 April 2014 - www.fjl.co.uk
# You may use this script for your own purposes, but must not distribute it without the copyright message above left intact

# Directory to write config files
# Normally /usr/local/etc/apache22/Include but you might want to write
# them somewhere else to check them first!

confdir=/usr/local/etc/apache22/Include

# oldhome and newhome are the old and new home directories (where the web sites are stored
# oldtestname and newtestname are used (together with a sub-domain) to implement test web sites before
# they have a real domain name pointed at them. They will be substituted in server names and aliases

oldhome=/data03/exampleuser/public_html
newhome=/home/exampleuser/www
oldtestname=exampleuser.oldisp.co.uk
newtestname=newuser.fjl.org.uk

# Now some static information to add to all virtual hosts
# vhost is the IP address or hostname you're using for virtual hosting (i.e. the actual name of the server)
# serveradmin is the email address of the server admin
# logfiles is the directory you want to put the log files in (assuming you're doing separate ones). If
# you do this you must uncomment the lines that write the .conf file

vhost=web.exampleuser.com
serveradmin=yourname@example.com
logdir=/var/log

getvalue()
{
grep ^$1: $name | sed s!$1:\ !! | sed s!$oldtestname!$newtestname!
}

# Start of main loop We DO NOT want to process a special file in the directory called "main" so
# a check is made.

for name in `ls`; do
if [ "$name" != "main" ]
then
echo -n "Processing $name "

if grep ^servername: $name >>/dev/null
then

# First we get some info from the file

sitename=`getvalue servername`
serveralias=`getvalue serveralias`
documentroot=`getvalue documentroot`

# Below we're setting the .conf pathname based on the first part of the file name (up to the first '.')
# This assumes that the file names are in the form websitename.isp.test.domain.com
#
# If the sitename in the source file is actually the name of the site (rather than a test alias) use
# this instead with something like:
#
# Basically, you want to end up with $givensitename as something meaningful when you see it
#
#givensitename=$sitename

givensitename=`echo $name | cut -d \. -f1`

confname=$confdir/$givensitename.conf

echo to $confname

echo "" >$confname
echo -e \\tServerAdmin $serveradmin >>$confname
echo -e \\tServerName $sitename >>$confname
for aname in $serveralias; do
echo -e \\tServerAlias $aname >>$confname
done
echo -e \\tDocumentRoot `echo $documentroot | sed s!$oldhome!$newhome!` >>$confname
echo -e \\tErrorLog $logdir/$givensitename-error.log >>$confname
echo -e \\tCustomLog $logdir/$givensitename-access.log combined >>$confname
echo "" >>$confname

#from check that servername present
else
echo "- ignoring file - no servername therefore wrong format?"
fi

#fi from check it wasn't called "main"
fi
done

All of the above assumes you’re familiar with setting up virtual hosting on an Apache 2.2 http server in an UNIX-like environment. It’s just too complicated to explain that in a single blog post. Drop me a line if you need assistance.

Posted on

FreeBSD 10.0 and ZFS

It’s finally here: FreeBSD 10.0 with ZFS. I’ve been pretty happy for many years with twin-drive systems protected using gmirror and UFS. It does what I want. If a disk fails it drops it out and sends me an email, but otherwise carries on. When I put a replacement blank disk it can re-build the mirror. If I take one disk out, put it into another machine and boot it, it’ll wake up happy. It’s robust!

So why mess around with ZFS, the system that puts your drives in to a pool and decides where things are stored, so you don’t have to worry your pretty little head about it? The snag is that the old ways are dying out, and sooner or later you’ll have no choice.

Unfortunately, the transition hasn’t been that smooth. First off you have to consider 2Tb+ drives and how you partition them. MBR partition tables have difficulties with the number of sectors, although AF drives with larger sectors can bodge around this. It can get messy though, as many systems expect 512b sectors, not 4k, so everything has to be AF-aware. In my experience, it’s not worth the hassle.

The snag with the new and limitless “GPT” scheme is that it keeps safe copies of the partition at the end of the disk, as well as the start. This tends to be where gmirror stores its meta-data too. You can’t mix gmirror and GPT. Although the code is hackable, I’ve got better things to do.

So the good new is that it does actually work as a replacement for gmirror. To test it I stuck two new 3Tb AF drives into a server and installed 10.0 using the new procedure, selecting the menu option zfs on root option and GPT partitioning. This is shown in the menu as “Experimental”, but seems to work. What you end up with, if you select two drives and say you want a zfs mirror, is just that.

Being the suspicious type, I pulled each of the drives in turn to see what had happened, and the system continues without a beat just like gmirror did. There were also a nice surprises when I stuck the drives back in and “onlined” them:

First-off the re-build was almost instant. Secondly, HP’s “non-hot-swap” drive bays work just fine for hot-swap under FreeBSD/ZFS. I’d always suspected this was a Windoze nonsense. All good news.

So why is the re-build so fast? It’s obvious when you consider what’s going on. The GEOM system works a block level. If the mirror is broken it has no way of telling which blocks are valid, so the only option is to copy them all. A major feature of ZFS, however, is that the directories and files have validation codes in the blocks above, going all the way to the root. Therefore, by starting at the root and chaining down, it’s easy to find the blocks containing changed data, and copy them. Nice! Getting rid of separate volume managers and file systems has its advantages.

So am I comfortable with ZFS? Not yet, but I’m a lot happier with it when its a complete, integrated solution. Previously I’d only been using on data drives in multi-drive configurations, as although it was possible to install root on ZFS, it was a real PITA.

Posted on

Pipe stdout to more than one process on FreeBSD

There are odd times when you may wish to use the stdout of a program as the stdin to more than one follow-on. bash lets you do this using a > to a command instead of just a file. I think the idea is that you just have to make sure the command is in ( ) and it works. One up to bash, but what about somthing that will work on standard shells?

The answer is to use a named pipe or fifo, and it’s a bit more hassle – but not too much more.

As an example, lets stick to the “hello world” theme. The command sed s/greeting/hello/ will replace the word “greeting” on stdin with “world” on stdout – everything else will pass through unchanged. Are you okay with that? Try it if you’re not comfortable with sed

Now I’m going to send a stdout to two different sed instances at once:

sed s/greeting/hello/
sed s/greeting/world/

To get my stdout for test purposes I’ll just use “echo greeting”. To pipe it to a single process we would use:

echo greeting | sed s/greeting/hi/

Our friend for the next part is the tee command (as in a T in plumbing). It copies stdin to two different places, stdout and (unfortunately for us) a file. Actually it can copy it to as many files as you specify, so it should probably have been called “manifold”, but this is too much to ask for an OS design that spells create without the training ‘e’.

Tee won’t send output to another processes stdin directly, but the files can be a fifos (named pipes). In older versions of UNIX you created a pipe with the mknod command, but since FreeBSD moved to character only files this is deprecated and you should use mkfifo instead. Solaris also uses mkfifo, and it came in as far back as 4.4BSD, but if you’re using something old or weird check the documentation. It’ll probably be something like mknod <pipename> .

Here’s an example of it in action, solving our problem:

mkfifo mypipe
sed s/greeting/hello/ < mypipe &
echo greeting | tee mypipe | sed s/greeting/world/  
rm mypipe

It works like this: First off we create a named pipe called mypipe. Next (and this is the trick), we run the first version of sed, specifying its input to come from “mypipe”. The trailing ‘&’ is very important. In case it had passed you by until now, it means run this command asynchronously – or in background mode. If we omitted it, sed would sit there waiting for input it would never receive, and we wouldn’t get the command prompt back to enter the further commands.

The third line has the tee command added to send a copy of stdout to the pipe (where the first sedis still waiting). The first copy is piped in the normal way to the second sed.

Finally we remove the pipe. It’s good to be tidy but it doesn’t matter if you want to leave it there use it again.

As a refinement, pipes with names like “mypipe” in the working directory could lead to trouble if you forgot to delete it or if another job picks the same name. Therefore it’s better to create them in the /tmp directory and add the current process ID to the name in order to avoid a clash. e.g.:

mkfifo /tmp/mypipe.1.$$

$$ expands to the process-ID, and I added a .1. in the example so I can expand the scheme to have multiple processes receiving the output – not just one. You can use tee to send to as many pipes and files as you wish.

If you run the example, you’ll probably get “hello world” output on two lines, but you might get “world hello”. The jobs have equal status, so there’s no way to of knowing which one will complete first, unless you decided to put sleep at the start of one for force the issue.

When I get around to it a more elaborate example might appear here.

Posted on

Using MX records to create backup mail server

There’s a widely held misunderstanding about “main” and “backup” MX records in the web developer world. The fact is that there no such thing! Anyone who tells you different is plain wrong, but there are a lot of web developers who believe there is the case, and some ISPs give in and provide them as it’s simpler than arguing. It’s possible to use two MX records in some crazy scheme that looks like a backup server, in practice it does very little to help and quite possibly rather a lot to hinder. It won’t make your email more robust in practical terms.

If you are using an email server at a data centre, with reasonable expectation of an always-on connection, you need a single MX record. If your processing requirements are great you can have multiple records at the same level to spread the load between peered servers, but none would be a backup any more than any other. Senders simply get one server at random. I have a single MX record.

“But you must have a backup!”, is the usual response. I do, of course, but it has nothing to do with having multiple MX records. Let me explain:

A domain’s MX record gives the address of the server to which its email should be sent. In practice, this means the company’s mail sever; or if they have multiple servers, the incoming one. Most companies have one mail server address, and this is fine. If that mail server dies it needs to be repaired or replaced, and the replacement gets the same address.

But what of having a second MX record with an alternative, lower-priority server? It may sound good, but it’s nuts. Think about it – the company’s mail server is where the mail ends up. It’s where the users expect to log in and read it. If you have an alternative server, the mail will go there instead, but the user’s won’t be able to read it. This assumes that the backup is on a different site, available if only the first site goes down. If it’s on the same site it’s even more pointless, as it will be affected by the same connectivity issues that took the first one offline. Users will have their existing mail on the broken server, new mail will be on a different server, and you’ll be in a real bugger’s muddle trying to reconcile the two later. It makes much more sense to just fix the broken one, or switch in a backup at the same location and on the existing IP address. In extremis, you can change the MX record to point to a replacement server elsewhere.

There’s a vague idea that if you don’t have a second MX, mail will be lost. Nothing can be further from the truth. If your one and only mail server is off-line, the sender’s server will queue up the message and keep trying until it comes back – it will normally do this for a week. It won’t lose it. Most mail servers will report back to the sender if it hasn’t been able to get through for four hours, so they’ll know there’s a problem and won’t worry that you haven’t replied.

If you have two mail servers, one on a different site, the secondary server will start receiving emails when the first one goes off-line. It’ll just queue them up, waiting to forward them to the primary one, but in this case the sender won’t get notification of the delay. Okay, if the primary server is off-line for more than a week it will prevent mail loss – but why would the primary server possibly be off-line for a week – the company won’t function unless it’s repaired quickly.

In the old days of dial-up, before POP3 came in to being, some people did use SMTP in a way where a server in a data centre forwarded to the remote site when it connected. I remember Cliff Stanford had a PC mail client called Turnpike that did just this in the early days of Demon. But SMTP was designed for always-on connections and POP3 was designed for dial-up, so POP3 won out.

Let’s get real: There are two likely scenarios for having a mail server off-line. Firstly, the hardware could be dead. If so, get it repaired, and in less than a week. Secondly, the line to the server could be down, and this could be medium-term if someone with a JCB has done a particularly “good job” on it. This gives you a week to make alternative arrangements and direct the mail down another line, which is plenty of time.

So, the idea of having a “backup” MX is pointless. It can only send mail to an off-site server; it doesn’t prevent any realistic mail loss and your email ends up where your users can’t get it until the primary server is repaired. But is there any harm in having one if it makes you feel better?

Actually, in practice, yes. It does make matters worse. In theory mail will just pile up on a spare server and get forwarded later. However, this spare server probably isn’t going to be up to the same specification as the primary one. They never are – they sit there idling, with nothing to do nearly all the time. They won’t necessary have the fastest line; their spam and virus filtering will be out-of-date or non-existent and they have a finite amount of disk space to absorb mail. This can really matter if you end up storing and forwarding a large amount of spam, as is often the case these days. The primary server can be configured to discard it quickly, but this isn’t a job appropriate for the secondary one. So it builds up until it’s ancient and meagre disk space is exhausted, and then it tells the sender to give up trying due to a “disk full” error – and the email is bounced off in to the ether. It’d have been much better to leave it on the sender’s server is the first place.

There are other security issues to having a secondary server. One problem comes with spam filtering. This is best done at the end of the line; it’s not the place of a secondary server to determine what gets delivered and what doesn’t. For starters, it doesn’t see the corpus of legitimate emails, so won’t be so adept at comparing and sorting. It’s probably going to be some old spare kit that’s under-powered for modern spam processing anyway. However, when it stores and forwards it, the primary server will see it comes from a “friend” rather than a dubious source in a lawless part Internet. Spammers do use secondary MX records as a back door to get around virus and spam filters for this very reason.

You could, of course, specify and configure a secondary mail server to be up to the job, with loads of disk space to prevent a DoS attack and fully functional spam filters, regularly maintained and sharing Bayesian analysis data and local rules with the actual server. And then have this expensive resource sitting there doing nothing all day but converting electricity in to heat. Realistically, it’s not going to happen.

By now you may be wondering, if multiple MX records are so pointless, why they exist? It’s one of these Internet myths; a paradigm that users feel comfortable with, without questioning the technology behind it. There is a purpose, but it’s not for “backup”.

When universal Internet email was new, messages would be sent to a user “@” computer, and computers were normally shared, so each would have multiple possible users. The computer would receive the email and put it in the mailbox corresponding to the user part of the address.

When the idea of sending email to a domain rather than a specific server came in to being, MD and MF records also came in to being. A MD record gave the IP address of the server where mail should end up (the Mail Destination). An MF record, if it existed, allowed the mail to be forwarded through another machine first (Mail Forward). This was sometimes necessary, for example if the MD was on a dial-up connection or behind a firewall and unable to accept direct connections over the Internet. The mail would go to the MF instead, and the MF would send it to the MD – presumably by batching it up and opening a line, transiting a firewall or using some other non-public mechanism.

In the mid 1980’s it was felt that having both MD and MF records placed double the load on DNS servers, so unified MX records, which could be read with a single lookup, were born. To allow for multiple levels of mail forwarding through firewalls, they were prioritised to 99 levels, although if you need more than three for any scheme you’re just being silly.

Unfortunately, the operation of MX records, rather than the explicitly named MF and MD, is a bit subtle. So subtle it’s often very misunderstood.

The first thing you need to understand is that email delivery should be controlled from the DNS for the domain, NOT from the individual mail servers that exist on that domain. This may not be obvious, but this is how it’s designed to work, and when you think of it, a central point of control is a good thing.

Secondly, DNS records should be universal. Every computer on the Internet, making the same DNS query, should get the same result. With the later addition of asymmetric NAT, there is now an excuse for varying this, but you can come unstuck if you get it wrong and that’s not what it was designed for.

If you want to reconfigure the route that mail takes to a domain, you do it by editing the single master DNS record (zone file) for that domain – you leave the multiple mail servers alone,

Now consider this problem: an organisation (called “theorganisation”) has a mail server called A. It’s inside the theorganisation’s firewall, for its own protection. Servers on the Internet can’t talk to A directly, because the firewall won’t let them through, but local users send and receive mail through it all day long. To receive external mail there’s another server called B, this time outside the firewall. A rule on the firewall allows specific traffic from B to get to A. The relevant part of the zone file may look something like this (at least logically):

MX 1 A.theorganisation
MX 2 B.theorganisation

So how do these simple lines tell the world, and servers A and B, how to operate? You need to understand the rules…

When another server, which I shall call C, sends a message to alice@theorganisation it will look up the MX records for theorganisation, and see the records above. C will then attempt to contact alice at the lowest numbered MX it finds, which points to server A. If C is located within the same department, it will be within the firewall and mail can be delivered directly; otherwise the firewall will block it. If C can’t contact A because of the firewall it will try the next highest on the list, in this case B. B is on the Internet, and will accept connections from C (and anyone else). The message arrives at B for Alice, but alice is not a user of B. However, B knows that it’s not the final destination for mail to theorganisation because the MX record says there’s a lower numbered server called A, so it attempts to forward it there. As B is allowed through the firewall, it can deliver the message to A, where it’s finally put in alice’s mailbox.

This may sound a bit complicated, but the rules for MX records can be made to route mail through complex paths simply by editing the DNS zone file, and this is how it’s supposed to work. The DNS zone file MX records control the path the mail will take. If you try to use the system for some contrary purpose (like a poor-man’s backup), you’re going to come unstuck.

There is another situation where you might want multiple MX records: If your mail server has multiple network interfaces on different (redundant) lines. If the MX priority value is the same for both, each IP address will (or should) be used at random, but if you have high-cost and low-cost lines you can change the priority to favour one route over another. With modern routing this artifice may not be necessary – let the router choose the line and mangle the IP addresses in to one for you. But sometimes a simple solution works just as well.

In summary, MX record forwarding is not designed for implementing backup mail servers and any attempt to use them for the purpose is going to do more harm than good. The ideas that this is what they’re all about is a myth.

 

Posted on

FreeBSD 8.4 released today

FreeBSD 8.4 has just been released. But I thought we were up to 9.1? Actually version 8 is still being maintained for those who don’t want to change too much in one go, as is the way for FreeBSD.

FreeBSD Logo
FreeBSD 8.4 released

Given this conservatism, why bother upgrading from 8.3 to 8.4? If you want the latest, why not go straight to  9.1; otherwise be conservative and leave well alone? This time I might upgrade, because 8.4 contains fixed versions of BIND and OpenSSL. Certain high-profile DDoS attacks amplified by a feature of BIND are a good enough reason to suggest everyone keeps up with the latest BIND release.

You could, of course, update BIND and OpenSSL by just pulling them from the repository but there are a number of other good bug fixes in there anyway, especially in several on the Ethernet drivers. And ZFS has been improved, if you want crazy disks.

I’m not expecting 9.2 to show up until early next year, if convention holds, which is a pity because some of the BIND and OpenSSL problems were found after 9.1 was released. Don’t wait until January, apply the patches now! (Follow the links above).

 

Posted on

Rename file extensions in UNIX/Linux/FreeBSD

I had a directory with thousands of files from a Windoze environment with inconsistent file extension  Some ended in .hgt, others in .HGT. They all needed to be in lower case, for some Windows-written cross-compiled software to find them. UNIX is, of course, case-sensitive on such things but Windoze with its CP/M-like file system used upper-case only, and when the shift key was invented, decided to ignore case.

Anyway, rather than renaming thousands of files by hand I thought I’d write a quick script. Here it is. Remember, the old extension was  .HGT, but I needed them all to be .hgt:

for oldname in `find . -name "*.HGT"`
do
newname=`echo $oldname | tr .HGT .hgt`
mv $oldname $newname
done

Pretty straightforward  but I’d almost forgotten the tr (translate) command existed, so I’m now feeling pretty smug and thought I’d share it with the world. It’ll do more than a simple substitution – you could use “[A-Z] [a-z]” to convert all upper case characters in the file to lower case, but I wanted only the extensions done. I could probably have used -exec on the find command, but I’ll leave this as an exercise for the reader!

It could me more compact if you remove the $newname variable and substitute directly, but I used to have an echo line in there giving me confirmation I was doing the right thing.

 

Posted on

FreeBSD, Wake-on-LAN and HP Microservers – WOL compatible Ethernet

I’ve been having some difficulties getting Wake-on-LAN (WOL) to work with an HP Microserver thanks to its Broadcom Ethernet adapter not doing the business with the FreeBSD drivers – setting WOL in the Microserver BIOS doesn’t have any effect. I’m pleased to report a solution that works.

The on-board Broadcom Ethernet adaptor still refuses to play ball, for reasons described in my earlier post. The pragmatic solution is to use a better supported chip set and I’ve had no difficulties with Realtek (at the low end of the market) so it was an obvious choice. Just bung a cheap Realtek-based card in and use it as a remote “on” switch – what could possibly go wrong?

First off, the HP Microserver has PCI-Express slots, but weird looking ones. I’d assumed one was PCI when I’d glanced it, but it’s a PCIe 1-channel slot with something strange behind it – possibly a second 1-channel slot. The documentation says its for a remote management card; presumably one which doesn’t need access to the back. There’s a 16-channel PCIe next to it.All very curious but irrelevant here. The point is that you’ll need a PCIe Ethernet card – a surplus 100M PCI one with a well supported, bog-standard chip, won’t do. The PCIe cards tend to be 1Gb, and are therefore not as cheap.

The first card I bought was a TP-Link TG-3458, which has standard Realtek 8168B adapter chip. Or at least mine did; I note that there is a Mk2 version out there. Mine’s definitely a revision 1.2 PCB, but if you buy one now it may have the newer chip (which is a problem – read on below). Anyway, this Mk1 card worked like a charm. On sending it the magic packet and the Microserver bursts in to life. There’s only one snag: It has a full-height bracket and the Microserver has a half-height slot, so you have to leave the card floating in its socket. This works okay as long as no one trips over the cable.

My second attempt was an Edimax EN-9260TX-E, ordered because it was (a) cheap-ish; (b) had a Realtek chip; and (c) had the all-important half-height bracket. It fitted in the Microserver all right, but refused to act on a WOL, at least to begin with…

It turns out there was a little bug-ette in the driver code (prior to 8.3 or 9.1), spotted and fixed by the maintainer about a year ago. If you want to fix it yourself the patch is here. I decided I might as well use the latest drivers rather than re-working those shipped with 8.2, so pulled them, compiled a new if_re.ko and copied it to /boot/kernel in place of the old one. It didn’t work. Ha! Was I naive!

Further investigation revealed that it was completely ignoring this kernel module, as it was using a driver compiled in to the kernel directly. There was no point having the module there, all it does is trick you in to believing that it’s installed. I only realised “my” mistake when, to my astonishment, removing the file completely didn’t disable the network interface. I solved the problem by compiling a new kernel with the built-in Realtek driver commented out, and I’m currently loading the new driver specifically in loader.conf. It works a treat. I could have changed the kernel Realtek driver, but while it’s under review it’s easier to have it loaded separately. Incidentally, the driver is for 9.1 onwards but it works fine on 9.0 so far.

The next task is to fix the Broadcom driver so it works. I may be gone some time…

Posted on

Lighttpd in a FreeBSD Jail (and short review)

Lighttpd is an irritatingly-named http daemon that claims to be light, compared to Apache. Okay, the authors probably have a point although this puppy seems to like dragging perl in to everything and there’s nothing minuscule about that.

I thought it might be worth a look, as Apache is a bit creaky. It’s configuration is certainly a lot simpler than httpd.conf,although strangely, you tend to end up editing the same number of lines. But is it lighter? Basically, yes. If you want the figures it’s currently running (on AMD64) a size of 16M compared to Apache httpd instances of 196M.

But we’re not comparing like for like here, as Lighttpd doesn’t have PHP; only CGI. If you’re worried about that being slow, there’s FastCGI, which basically keeps instances of the CGI program running and Lightttpd hands tasks off to an instance when they crop up. Apache can do this (there’s the inevitable mod), but most people seem happy using the built-in PHP these days so I don’t think FastCGI is very popular. It’s a pity, as I’ve always felt CGI is under-rated and I’m very comfortable passing off to programs written in ‘C’ without there being an noticeable performance issues. Using CGI to run a perl script and all that entails is horrendous, of course. But FastCGI should level the playing field and allow instances of perl or any other script language of your dreams to remain on standby in much the same way PHP currently remains on standby in Apache. That doesn’t make perl or PHP good, but it levels their use with PHP on Apache, giving you the choice. And you can also choose  high-performance ‘C’.

This is all encouraging, but  I haven’t scrapped Apache just yet. One simple problem, with no obvious solution, is the lack of support for the .htaccess file much loved by the web developers and their content management systems. Another worry for me is security. Apache might be big and confusing, but it’s been out there a long time and has a good track record (lately). If it has holes, there are a lot of people looking for them.

Lighttpd doesn’t have a security pedigree. I’m not saying it’s got problems; it’s just that it hasn’t been thrashed in the same way as Apache and I get the feeling that the development team is much smaller. Sometimes this helps, as it’s cleaner code, but it’s statistically less likely to have members adept at spotting security flaws too. I’m a bit concerned about the FastCGI servers all running on the same level, for example.

Fortunately you can mitigate a lot of security worries by running in a jail on FreeBSD (it will also chroot on Linux, giving some degree of protection). It was fairly straightforward to compile from the ports collection, but it does have quite a few dependencies. Loads of dependencies, in fact. I saw it drag m4 in for some reason! Also the installation script didn’t work for me but it’s easy enough to tweak manually (find the directory with the script and run make in it to get most of the job done). The other thing you have to remember is that it will store local configurations in /usr/local on BSD, instead of the base system directories.

To get it running you’ll need to edit  /usr/local/etc/lighttpd/lighttpd.conf, and if you’re running in a jail be sure to configure the IP addresses to bind to correctly. Don’t be fooled: There’s a line at the bottom that sets the IP address and port but you must find the entry server.bind in the middle of the file and set that to the address you’ve configured for the jail to have passed through. This double-entry a real pooh trap, especially as it tries to bind to the loopback interface and barfs with a mysterious message. Other than that, it just works – and when it’s in the jail it will happily co-exist with Apache.

I’ve got it running experimentally on a production server now, and I’ve also cross-compiled to ARM and it runs on Raspberry Pi (still on FreeBSD), but it was more fun doing that with Apache.

When I get time I’ll do a full comparison with Hiawatha.