Posted on

Using ddrescue to recover data from a USB flash drive

If you’re in the data recovery, forensics or just storage maintenance business (including as an amateur) you probably already know about ddrescue. Released about twenty years ago by Antonio Diaz Diaz, it was a big improvement over the original concept dd_rescue from Kurt Garloff in 1999. They copy disk images (which are just files in Unix) trying to get as much data extracted when the drive itself has faults.

If you’re using Windows rather than Unix/Linux then you probably want to get someone else to recover your data. This article assumes FreeBSD.

The advantage of using either of these over dd or cp is that they expect to find bad blocks in a device and can retry or skip over them. File copy utilities like dd ignore errors and continue, and cp will just stop. ddrescue is particularly good at retrying failed blocks, and reducing the block size to recover every last readable scrap – and it treats mechanical drives that are on their last legs as gently as possible.

If you’re new to it, the manual for ddrescue can be found here. https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html

However, for most use cases the command is simple. Assuming the device you want to copy is /dev/da1 and you’re calling it thumbdrive the command would be:

ddrescue /dev/da1
thumbdrive.img thumbdrive.map

The device data would be stored in thumbdrive.img, with ongoing state information stored in thumbdrive.map. This state information is important, as it allows ddrescue to pick up where it left off.

However, ddrescue was written before USB flash drives (pen drives, thumb drives or whatever). That’s not to say it doesn’t work, but they have a few foibles of their own. It’s still good enough that I haven’t modified ddrescue base code to cope, but by using a bit of a shell script to do the necessary.

USB flash drives seem to fail in a different way to Winchester disks. If a block of Flash EPROM can’t be read it’s going to produce a read error – fair enough. But they have complex management software running on them that attempts to make Flash EPROM look like a disk drive, and this isn’t always that great in failure mode. In fact I’ve found plenty of examples where they come across a fault and crash rather than returning an error, meaning you have to turn them off and on to get anything going again (i.e. unplug them and put them back in).

So it doesn’t matter how clever ddrescue is – if it hits a bad block and the USB drive controller crashes the it’s going to be waiting forever for a response and you’ll just have come reset everything manually and resume. One of the great features of ddrescue is that it can be stopped and restarted at any time, so continuing after this happens is “built in”.

In reality you’re going to end up unplugging your USB flash drive many times during recovery. But fortunately, it is possible to turn a USB device off and on again without unplugging it using software. Most USB hardware has software control over its power output, and it’s particularly easy on operating systems like FreeBSD to do this from within a shell script. But first you have to figure out what’s where in the device map – specifically which device represents your USB drive in /dev and which USB device it is on the system. Unfortunately I can’t find a way of determining it automatically, even on FreeBSD. Here’s how you do it manually; if you’re using a version of Linux it’ll be similar.

When you plug a USB storage device into the system it will appear as /dev/da0 for the first one; /dev/da1 for the second and so on. You can read/write to this device like a file. Normally you’d mount it so you can read the files stored on it, but for data recovery this isn’t necessary.

So how do you know which /dev/da## is your media? This easy way to tell is that it’ll appear on the console when you first plug it in. If you don’t have access to the console it’ll be in /var/log/messages. You’ll see something like this.

Jun 10 17:54:24 datarec kernel: umass0 on uhub5
kernel: umass0: <vendor 0x13fe USB DISK 3.0, class 0/0, rev 2.10/1.00, addr 2> on usbus1
kernel:  umass0 on uhub5
kernel:  umass0:  on usbus1
kernel:  umass0:  SCSI over Bulk-Only; quirks = 0x8100
kernel:  umass0:7:0: Attached to scbus7
kernel:  da0 at umass-sim0 bus 0 scbus7 target 0 lun 0
< USB DISK 3.0 PMAP> Removable Direct Access SPC-4 SCSI device
kernel:  da0: Serial Number 070B7126D1170F34
kernel:  da0: 40.000MB/s transfers
kernel:  da0: 59088MB (121012224 512 byte sectors)
kernel: da0: quirks=0x3
kernel: da0: Write Protected

So this is telling us that it’s da0 (i.e /dev/da0)

The hardware identification is “<vendor 0x13fe USB DISK 3.0, class 0/0, rev 2.10/1.00, addr 2> on usbus1” which means it’s on USB bus 1, address 2.

You can confirm this using the usbconfig utility with no arguments:

ugen5.1:  at usbus5, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (0mA)
 ...snip...
ugen1.1:  at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (0mA)
ugen1.2:  at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (300mA)

There it is again, last line.

usbconfig has lots of useful commands, but the ones we’re interested are power_off and power_on. No prizes for guessing what they do. However, unless you specify a target then it’ll switch off every USB device on the system – including your keyboard, probably.

There are two ways of specifying the target, but I’m using the -d method. We’re after device 1.2 so the target is -d 1.2

Try it and make sure you can turn your USB device off and on again. You’ll have to wait for it to come back online, of course.

There are ways of doing this on Linux by installing extra utilities such as hub-ctrl. You may also be able to do it by writing stuff to /sys/bus/usb/devices/usb#/power/level” – see the manual that came with your favourite Linux distro.

The next thing we need to do is provide an option for ddrescue so that it actually times out if the memory stick crashes. The default is to wait forever. The –timeout=25 or -T 25 option (depending on your optional taste) sees to that, making it exit if it hasn’t been able to read anything for 25 seconds. This isn’t entirely what we’re after, as a failed read would also indicate that the drive hadn’t crashed. Unfortunately there’s no such tweak for ddrescue, but failed reads tend to be quick so you’d expect a good read within a reasonable time anyway.

So as an example of putting it all into action, here’s a script for recovering a memory stick called duracell (because it’s made by Duracell) on USB bus 1 address 2.

#!/bin/sh
while ! ddrescue -T 25 -u /dev/da0 duracell.img duracell.map
 do
     echo ddrescue returned $?
     usbconfig -d 1.2 power_off
     sleep 5
     usbconfig -d 1.2 power_on
     sleep 15
     echo Restarting
 done

A few notes on the above. Firstly, ddrescue’s return code isn’t defined. However, it appears to do what one might expect so the above loop will drop out if it ever completes. I’ve set the timeout for time since last good read to 25 seconds, which seems about right. Turning off the power for 5 seconds and then waiting for 15 seconds for the system to recognise it may be a bit long – tune as required. I’m also using the -u option to tell ddrescue to only go forward through the drive as it’s easier to read the status when it’s always incrementing. Going backwards and forwards makes sense with mechanical drives, but not flash memory.

Aficionados of ddrescue might want to consider disabling scraping and/or trimming (probably trimming) but I’ve seen it recover data with both enabled. Data recovery is an art, so tweak away as you see fit – I wanted to keep this example simple.

Now this system isn’t prefect. I’m repurposing ddrescue, which does a fine job on mechanical drives, to recover data from a very different animal. I may well write a special version for USB Flash drives but this method does actually work quite well. Let me know how you get on.

Posted on

Proper Case in a shell script

How do you force a string into proper case in a Unix shell script? (That is to say, capitalise the first letter and make the rest lower case). Bash4 has a special feature for doing it, but I’d avoid using it because, well, I want to be Unix/POSIX compatible.

It’s actually very easy once you’ve realised tr won’t do it all for you. The tr utility has no concept on where in the input stream it is, but combining tr with cut works a treat.

I came across this problem when I was writing a few lines to automatically create directory layouts for interpreted languages (in this case the Laminas framework). Languages of this type like capitalisation of class names, but other names have be lower case.

Before I get started, I note about expressing character ranges in tr. Unfortunately different systems have done it in different ways. The following examples assume BSD Unix (and POSIX). Unix System V required ranges to be in square brackets – e.g. A-Z becomes “[A-Z]”. And the quotes are absolutely necessary to stop the shell globing once you’ve introduced the square brackets!

Also, if you’re using a strange character set, consider using \[:lower:\] and \[:upper:\] instead of A-Z if your version of tr supports it (most do). It’s more compatible with foreign character sets although I’d argue it’s not so easy on the eye!

Anyway, these examples use A-Z to specify ASCII characters 0x41 to 0x5A – adjust to suit your tr if your Unix is really old.

To convert a string ($1) into lower case, use this:

lower=$(echo $1 | tr A-Z a-z)

To convert it into upper case, use the reverse:

upper=$(echo $1 | tr a-z A-Z)

To capitalise the first letter and force the rest to lower case, split using cut and force the first character to be upper and the rest lower:

proper=$(echo $1 | cut -c 1 | tr a-z A-Z)$(echo $1 | cut -c 2- | tr A-Z a-z)

A safer version would be:

proper=$(echo $1 | cut -c 1 | tr "[:lower:]" "[:upper:]")$(echo $1 | cut -c 2- | tr "[:upper:]" [":lower:"])

This is tested on FreeBSD in /bin/sh, but should work on all BSD and bash-based Linux systems using international character sets.

You could, if you wanted to, use sed to split up a multi-word string and change each word to proper case, but I’ll leave that as an exercise to the reader.

Posted on

Systemd Network Configuration

Unless you’ve been living in a very Linux-free environment for a while, you’ll know about systemd – the collection of daemons intended to replace the System V init system commonly found on Linux, with something more complicated. I’m not a fan of System V startup, but they might have done better by going for the Research Unix or BSD /etc/rc approach for robustness, simplicity and compatibility. But Linux, to many, is a launcher stub for graphical desktops running LibreOffice and games, and these probably work better with systemd syntax when controlled by a simple GUI.

Systemd is more than an init system – in fact it has daemons from everything from the keyboard to DNS resolution – and network interface configuration (networkd)

This nightmare came out of Red Hat, and Linux distributions like Debian, Ubuntu, openSUSE, Arch, and their derivatives have started using it. One result, amongst other things, is that it’s suddenly not possible to configure networks the way you used to using ifconfig and /etc/resolv.conf.

You can install the missing ifconfig and suchlike using a package called net-tools, which is present on most major Linux distributions and is installed in the appropriate way (dnf, apt, yum etc). This may be the best way to keep scripts working.

Otherwise, you might be hoping systemd-networkd has simplified things, with less to type. But I’m afraid not.

So for those who are struggling, here’s a cheat sheet.

Names

The first think you’ll have to remember is that systemd-networkd doesn’t call your Ethernet interfaces eth0:, eth1. It doesn’t even call them by their driver name+enum BSD style. Instead it mungs a name from indices provided by the firmware, PCIe slot number and even the MAC address. Look out for some very strange interface names.

The idea is that the NIC/port has a predictable name, which is great in theory. I can see two problems: Firstly this doesn’t really help you find the RJ45 any better unless you have a schematic. Secondly, if you pull the system from one host and put it in another it all goes to hell in a handcart anyway. On the plus side I guess it means that adding or removing a NIC isn’t going to change the name of the existing ports.

For what it’s worth, eno# is an onboard device, ens# is a PCI slot index, enp#s# is port number on NIC and PCI slot index. enx1122334455667788 is the MAC address but this behaviour seems to be turned off on most systems. If it can’t determine anything it will fall back to eth#.

There are ways of selecting the old behaviour using kernel parameters or knobbling the /etc/systemd/network/… something “default” depending on system but you should check that out in the man page. Oh, hang on, this is Linux there probably no man pages.

Cheat Sheet

OldNew
ifconfig eth0 192.168.1.2/24ip addr add 192.168.1.2/24 dev eth0
ifconfig eth0 192.168.1.2 deleteip addr del 192.168.1.2/24 dev eth0
ifconfig eth0 netmask 255.255.255.0? Set address and netmask together ?
ifconfig eth0 mtu 5000ip link set eht0 mtu 5000
ifconfig eth0 down (or up)ip link set eth0 down (or up)
ifconfigip a
netstatss
netstat -rip route show
routeip r
route add default 192.168.1.254ip route add default via 192.168.1.254
arp -aip n
ifconfig eht0 name wan1? Not possible from command line ?

The last entry in the table is about renaming an interface, which given the user-hostile names now generated is even more useful. I haven’t figured out how to do this from the command line, but the assumption is that all interface configuration is done in configuration files by default, which brings us neatly on to these.

Configuring at startup

At one time you could just edit /etc/network/interfaces, and it might still work (it does int he latest Debian, for example). In BSD stick simple definitions in rc.conf, but that’s too easy. Anyway, /etc/network/interfaces could look something like this:

auto eth0
iface eth0 inet static
address 192.168.1.2
netmask 255.255.255.0
gateway 192.168.1.253

auto eth1
iface eth1 inet dhcp

After editing the configuration files(s) you could restart:

/etc/init.d/networking [start | stop | restart]

But some systemd Linux distributions are different. Systemd-networkd has a directory tree full of configuration stuff and I can only scratch the surface here.

Basically a load of *.network stored in /etc/systemd/network/ get run in sort order. It’s normal to prefix each file with two digits and a dash to set this order. I don’t think there’s any reason not to use a single file, but in the Linux world people don’t, often choosing to make the rest of the filename the NIC name, such as “04-enp0s5.network“, although the name you choose is only for your reference (or that of some GUI configuration tool).

To force every NIC to configure using dhcp create a file 02-dhcpall.network:

[Match]
Name=en*
[Network]
DHCP=yes

Note the wildcard on the NIC Name=*

On the other hand if you want to make one specific card static, have a file which you might want to call 01-enp5s2.network:

[Match]
Name=enp5s2
 
[Network]
Address=192.168.1.2/24
Gateway=192.168.1.254
DNS=192.168.1.254 8.8.8.8
Domains=example.com test.example.com

This should be fairly self-explanatory. You can specify multiple Address= lines (aliases) but for some reason DNS servers tend to be listed on one line, although multiple lines do work in my experience. I’ve used IPv4 in the examples but IPv6 works too.

Domains=example.com test.example.com is basically the DNS search domains (as normally found in resolv.conf). As systemd has its own resolver, systemd-resolved, it’s not just a matter of edit one file any longer, and is also less flexible.

You can restart systemd-networkd with:

systemctl restart systemd-networkd

If you haven’t made any mistakes you might still be connected to your server.

Posted on

Reply-To: gmail spam and Spamassassin

Over the last few months I’ve noticed huge increase is spam with a “Reply To:” field set to a gmail address. What the miscreants are doing is hijacking a legitimate mail server (usually a Microsoft one) and pumping out spam advertising a service of some kind. These missives only work if the mark is able to reply, and as even a Microsoft server will be locked down sooner or later, so they’ll never get the reply.

The reason for sending this way is, of course, spam from a legitimate mail server isn’t going to be blacklisted or blocked. SPF and other flags will be good. So these spams are likely to land in inboxes, and a few marks will reply based on the law of numbers.

To get the reply they’re using the email “Reply-To:” field, which will direct the reply to an alternative address – one which Google is happy to supply them for nothing.

The obvious way of detecting this would be to examine the Reply-To: field, and if it’s gmail whereas the original sender isn’t, flag it as highly suspect.

I was about to write a Spamassassin rule to do just this, when I discovered there is one already – and it’s always been there. The original idea came from Henrik Krohns in 2009, but it’s time has now definitely arrived. However, in a default install, it’s not enabled – and for a good reason (see later). The rule you want is FREEMAIL_FORGED_REPLYTO, and it’s found in 20_freemail.cf

Enabling FREEMAIL_FORGED_REPLYTO in Spamassassin

If you check 20_freemail.cf you’ll see the rules require Mail::SpamAssassin::Plugin::FreeMail, The FreeMail.pm plugin is part of the standard install, but it’s very likely disabled. To enable this (or any other plugin) edit the init.pre file in /usr/local/etc/mail/spamassassin/ Just add the following to the end of the file:

# Freemail checks
#
loadplugin Mail::SpamAssassin::Plugin::FreeMail FreeMail.pm

You’ll then need to add a list of what you consider to be freemail accounts in your local.cf (/usr/local/etc/mail/spamassassin/local.cf). As an example:

freemail_domains aol.* gmail.* gmail.*.* outlook.com hotmail.* hotmail.*.*

Note the use of ‘*’ as a wildcard. ‘?’ matches a single character, but neither match a ‘.’. It’s not a regex! There’s also a local.cf setting “freemail_whitelist”, and other things documented in FreeMail.pm.

Then restart spamd (FreeBSD: service spamd restart) and you’re away. Except…

The problem with this Rule

If you look at 20_freemail.cf you’ll see the weighting is very low (currently 0.1). If this is such a good rule, why so little? The fact is that there’s a lot of spam appearing in this form, and it’s the best heuristic for detecting it, but it’s also going to lead to false positives in some cases.

Consider those silly “contact forms” beloved by PHP Web Developers. They send an email from a web server but with a “faked” reply address to the person filling in the form. This becomes indistinguishable from the heuristic used to spot the spammers.

If you know this is going to happen you can, of course add an exception. You can even have the web site use a local submission port and send it to a local mailbox without filtering. But in a commercial hosting environment this gets a bit complicated – you don’t know what Web Developers are doing. (How could you? They often don’t).

If you have control over your users, it’s probably safe to up the weighting. I’d say 3.0 is a good starting point. But it may be safer to leave it at 0.1 and examine the results for what would have been false positives.

Posted on

Nothing new with Intel SDSi

Intel’s latest wheeze for its CPUs is Software Defined Silicone (SDSi). The deal is that you buy the CPU at one price and then pay extra for a license to enable more stuff.

If you want the geeky stuff about how it’s supposed to work in Linux, see here. https://github.com/intel/intel-sdsi

Basically, the CPU has an interface that you can access if you have an Authentication Key Certificate (AKC) and have purchased a Capability Activation Payload (CAP) code. This will then enable extra stuff that was previously disabled. Quite what the extra stuff is remains to be seen – it could be extra instructions or enabling extra cores on a multi-core chip, or enabling more of the cache. In other words, you buy extra hardware that’s disabled, and pay extra to use it. What’s even more chilling is that you could be continuously paying licenses for the hardware you’ve bought or it’ll stop working.

It’s not actually defining the silicone in software like a FPGA, as you’d expect from euphemistic name. Software Defined Uncrippling would be more honest, but a harder sell.

But this is nothing new. I remember IBM doing this with disk drives in the 1970’s. If you upgraded your drive to double the capacity an IBM tech turned up and removed a jumper, enabling the remaining cylinders. Their justification was that double the capacity meant double the support risk – and this stuff was leased.

Fast forward 20 years to Intel CPUS. Before the Intel 80486 chips you could provide whatever input clock you wanted to your 80386, just choosing how fast it went. Intel would guarantee the chip to run at a certain speed, but that was the only limiting factor. Exceed this speed at your own risk.

The thing was that the fast and slow CPUs were theoretically identical. It’s often the case with electronic components. However, manufacturing tolerances mean that not all components end up being the same, so they’re batch tested when the come off the line. Those that pass the toughest test get stamped with a higher speed and go in the fast bucket, where they’re sold for more. Those that work just fine at a lower speed go into the slower bucket and sell for less. Fair enough. Except…

It’s also the nature of chip manufacture that the process improves over time, so more of the output meets the higher test – eventually every chip is a winner. You don’t get any of the early-run slow chips, but you’re contracted to sell them anyway. The answer is to throw some of the fast chips into the slow bucket and sell them cheap, whilst selling others at premium price to maintain your margins.

In the early 1990’s I wrote several articles about how to take advantage of this in PCW, after real-world testing of many CPUs. It later became known as overclocking. I also took the matter up with Intel at the time, and they explained that their pricing had nothing to do with manufacturing costs, and everything to do with supply and demand. Fair enough – they were honest about it. This is why AMD gives you more bang-per-buck – they choose to make things slightly better and cheaper because that maximises their profits too.

With the introduction of the 80486, the CPU clock speed was set in the package so the chip would only run at the speed you paid for. SDSi is similar, except you can adjust the setting by paying more at a later date. It also makes technical sense – producing large quantities of just one chip has huge economies of scale. The yield improves, and you just keep the fab working. In order to have a product range you simply knobble some chips to make them less desirable. And using software to knobble them is the ultimate, as you can decide at the very last minute how much you want to sell the chip for, long after it’s packaged and has left the factory.

All good? Well not by me. This only works if you’re in a near monopoly position in the first place. Microsoft scalps its customers with licenses and residual income, and Intel wants in on that game. It’s nothing about being best, it’s about holding your customers to ransom for buying into your tech in the first place. This hasn’t hurt Microsoft’s bottom line, and I doubt it’ll hurt Intel’s either.

Posted on

STARTTLS is not a protocol

As regular readers will know, I’m not a fan of STARTTLS but today I realised that some people are confused as to what it even means. And there’s a perfectly good reason for this – some graphical email software is actually listing STARTTLS as a protocol for talking to mail servers and people are jumping to conclusions.

So what is STARTTLS all about if you go back to basics?

Originally, when only nice people had access to computers, network traffic was unencrypted. If you had physical access to the network you could pretty much read anything you wanted to, as everything connected to the same network saw the same data. This isn’t true now, but encryption you data is a good idea just in case it can be intercepted – and if it’s going over the Internet that’s definitely the case.

In the mid 1990s, the original mass-market web browser, Netscape, decided to do something about it and they (or more specifically their chief scientist Taher Elgamal, invented a protocol called Secure Sockets Layer (SSL) to protect HTTP (web) traffic. Actually, several times as the first couple of attempts weren’t very secure at all.

SSL didn’t really fit in with the OSI model; it runs on top of the transport protocol (usually TCP) but under the presentation layer, which would logically handle encryption but doesn’t usually. To use it you need an SSL layer added to the stack to transparently do the deed on a particular port.

But, as a solution to the encryption problem, SSL took off and pretty much every major protocol has an SSL port along with its original cleartext one. So clear HTTP is on port 80, HTTPS is on port 443. Clear POP3 is on port 110, encrypted on 995. Clear IMAP is on port 143, encrypted on 993.

As is the way of genius ideas in cybersecurity, even the third version of SSL was found to be full of holes. SSL version 3.1, which was renamed TLS, continued plugging the leaks and by TLS 1.2 it’s considered pretty much secure now. TLS 1.3, which interoperates with TLS 1.2, simply deprecates certain cyphers and hashes on the suspicion they might be insecure; although anyone into cybersecurity should tell you that everything is secure only until it’s broken.

Unfortunately, because different levels of TLS use different cyphers and reject others, TLS levels are by no means interoperable. And neither is it the case that a newer version is more secure; bugs have been introduced and later fixed. This’d be fine if everything and everyone used the same version of TLS, but in the real world this isn’t practical – old hardware, in particular, bakes in old versions of SSL or TLS and if you decided to deprecate older cyphers and not work with them, you loose the ability to talk to your hardware.

But apart from this, things were going along pretty well; and then someone had the bright idea of operating encrypted and unencrypted connections on the same port by hacking it at the application layer instead. This was achieved by modifying the application protocol to include a STARTTLS command. If this is received, the application then negotiates a TLS connection. If the receiving host didn’t understand what STARTTLS meant it’d send back and error, and things could continue unencrypted.

In other words, if you’re implementing an SMTP server with STARTTLS, this keyword is added to the protocol and the SMTP server does something about it when it sees it.

What could go wrong?

Well quite a lot of things, actually. Because TLS doesn’t fit in to the OSI model, it’s actually very difficult to deal with the situation where a TLS connection is requested and agreed to but the TLS layer fails to agree on a cypher with an older or newer version on the other end. There’s no mechanism for passing this to the application to say “okay, let’s revert to Plan A”, and the connection tends to hang.

There’s also a problem with name-based virtual servers must all use the same host certificate because the TLS connection must be established before the application layer headers are transferred.

But perhaps my biggest gripe is that enabling STARTTLS makes encryption optional. You’re not enforcing encryption when you need to, and even if you think you are, STARTTLS connection are obviously vulnerable to a man-in-the-middle attack. You have no idea how many times TLS has been turned on and off between the two endpoints.

You might be tempted to think that optional encryption is better than none at all, but in reality it means you don’t care – and if you don’t care, don’t bother. It just leads to a false sense of security. And it can lead to interoperability problems. My advice is to use “always TLS” ports for sensitive data and turn off the old port.

Posted on

Minecraft server in a FreeBSD Jail

You may have no interest in the game Minecraft, but that won’t stop people asking you to set up a server. Having read about how to do this on various forums and Minecraft fan sites (e.g. this one) I came to the conclusion that no one knew how to do it on current FreeBSD. So here is how you do it, jailed or otherwise.

First off, there isn’t a pre-compiled package. The best way to install it is from the ports, where it exists as /usr/ports/games/minecraft-server

Be warned – this one’s a monster! Run “make config-recursive” first, or it’ll go on stopping for options all the way through. Then run “make install”. It’s going to take quite some time.

The first configuration option screen asks if you want to make it as a service or stand-alone. I picked “service”, which sets up the start-up scripts for you but doesn’t actually tell you it’s done it. It does, however, stop it trying to run in graphics mode on your data centre server so I’m not complaining too much.

The good news is that this all works perfectly in a jail, so while it’s compiling (it could be hours) you can set up the required routing, assuming you’re using an internal network between jails – in this case 192.168.2.0/24. Using pf this will look something like:

externalip="123.123.123.123"
minecraft="192.168.2.3"
extinterface="fx0"
scrub in all
nat pass on $extinterface from 192.168.2.0/24 to any -> $externalip
rdr pass on $extinterface proto tcp from any to $externalip port 25565 -> $minecraft
rdr pass on $extinterface proto tcp from any to $externalip port  
{19132,19133,25565} -> $minecraft

And that’s it. You’re basically forwarding on TCP and three UDP ports. If you’re not using a jail, you obviously don’t need to forward anything. For instructions on setting up jails properly, see here, and for networking jails see elsewhere on this blog.

One thing that’s very important – this is written in Java, so as part of the build you’ll end up with OpenJDK. This requires some special file systems are mounted – and if you’re using a jail this will have to be in the host fstab, not the jails!

# Needed for OpenJDK
fdesc                   /dev/fd         fdescfs rw              0       0 
proc                    /proc           procfs  rw              0       0

If you’re using a jail, make sure the jail definition includes the following, or Java still won’t see them:

mount.devfs;
mount.procfs;

Once you’ve finished building you might bet tempted to follow some of these erroneous instructions in forums and try to run “minecraft-server”. It won’t exist!

To create the basic configuration files run “service minecraft onestart”. This will create the configuration files for you in /usr/local/etc/minecraft-server. It will also create a file called eula.txt. You need to edit this change “eula=false” to “eula=true”.

You can make the minecraft service run on startup with the usual “minecraft_enable=yes” in /etc/rc.conf

And that’s really it. There are plenty of fan guides on tweaking the server settings to your requirements, and they should apply to any installation.

This assumes you’re handy with FreeBSD, understand jails and networking; if you’re not so handy then please leave a comment or contact me. Everyone has to start somewhere, and it’s hard to know what level to pitch instructions like this. Blame me for assuming to much!

Posted on

USB Ethernet on FreeBSD

Why would you even want to cripple your network connection by using USB instead of a proper NIC? Well if you have no free PCIe slots you might have to, as is the case if you’re using a very small computer as a router/gateway.

But why would you use a small computer as a FreeBSD router? Well FreeBSD doesn’t keep crashing like the purpose-built routers we all know and hate. Yes, that’s right – you can use FreeBSD in your router and it won’t crash. Having to turn your network router on/off periodically is, it turns out, entirely optional.

So back to these small computers: the problem is they tend to have but one Ethernet port, and no slots to add a second. They do tend use USB as the only way of expanding the hardware. But assuming you don’t have a crazy fast WAN you can still get away with a USB network adapter on that. Perhaps don’t bother using a bunch of them to make a switch though, but for a single second port and not much else going on, it works just fine.

But here’s the thing – FreeBSD doesn’t support USB3 Ethernet chips very well. In fact the only one I know of is the ASIX88179, and there are mixed reports about how well this works these. I’ve ordered one to take a look at.

In the mean time, the smart money is on USB2; and although you can get 1Gb USB2->Ethernet chips, a quick calculation will show you can’t do a sustained 1Gb transfer through it. But if your WAN is <100Gbs, this won’t actually matter.

There are, fortunately, plenty of USB2->100baseT chips to choose from, and these are the safest options. I’ve extracted this list from the documentation:

ADMtek AN986 Pegasus
ASIX AX88172,AX88178,AX88772,AX88772A,AX88772B and AX88760
CATC EL1210A (10Mb only!)
Kawasaki LSI KL5KLUSB101B
RealTek RTL8150
SMSC LAN95xxx
Davicom DM9601

I’ll be doing some experiments with a few of these to see which work best, but if you’ve got any further information please comment!

Incidentally, thus far I’ve been happy with ASIX chips. That said, I’ve not tested them, or the others, to their limits yet. They do seem stable though.

Posted on

Networking FreeBSD Jails

Or port forwarding to a jail

I’ve already explained how easy FreeBSD jails are to set up and use without resorting to installing heavy management tools, but today I thought I’d add a bit about networking. Specifically, how do you pass traffic arriving on a particular port to a service running inside a jail?

It’s actually very easy. All you need is a very local network inside FreeBSD, natted to the one outside.

Suppose you have your jail.conf set up as per my previous article. Here’s an excerpt:

tom { ip4.addr = 192.168.0.2 ; }
dick { ip4.addr = 192.168.0.3 ;  }
harry { ip4.addr = 192.168.0.4 ;  }

The defaults were set earlier in the file; the only thing that’s unique about each jail is the IP4 address and the name. What I didn’t say at the time was that 192.168.0.0 could have been on an internal network.

To define your local network just define it in rc.conf:

cloned_interfaces="lo1"
ipv4_addrs_lo1="192.168.0.1-14/28"

This creates another local loopback interface and assigns a range of IPv4 addresses to it. This can be as large as you wish, but I’ve defined 1..14 (with appropriate subnet mask) because they’ll be listed every time you run ifconfig!

If you’re doing this as a one off or don’t want to reboot just yet, the equivalent commands are:

ifconfig lo1 create
ifconfig lo1 inet 192.168.0.1/28
ifconfig lo1 inet 192.168.0.2/32 alias
ifconfig lo1 inet 192.168.0.3/32 alias
...

Next you’re going to need something to do the natting. pf is your friend here. I struggled for years using ipfw before I discovered pf.

Enable pf in rc.conf too:

pf_enable="yes"

And you’ll need an /etc/pf.conf file to do the magic. I like pf – it’s easier for my brain to understand than most firewalls. Here’s an example file:

PUB_IP="192.168.1.217"
INT="bge0"
JAIL_NET="192.168.0.0/24"
TOM="192.168.0.2"
DICK="192.168.0.3" 
HARRY="192.168.0.4" 
scrub in all
nat pass on $INT from $JAIL_NET to any -> $PUB_IP
block on $INT proto tcp from any to $PUBIP port 111
rdr pass on $INT proto tcp from any to $PUBIP port 3306  -> $TOM
rdr pass on $INT proto tcp from any to $PUBIP port {21,80,443} -> $DICK
rdr pass on $INT proto tcp from any to $PUBIP port 81 -> $HARRY port 80

So what’s going on?

I’ve used a few macros. PUB_IP is your public IP address, and INT is the interface it’s on. pf may figure some of this out, but I’m being explicit.

TOM, DICK and HARRY are the IPv4 addresses of the jails.

Next I’m scrubbing all interfaces (normally a good idea, but you don’t have to). But the next line is important – it uses nat to allow stuff on your jail network to talk to the outside world. If your host has more than one interface, add extra lines as appropriate (e.g. if it’s a gateway, you might want to NAT to LAN addresses and the WAN).

The following line is where you might want to block more stuff – in this case NFS on port 111. Then we’re back to jail things for the final three lines. They’re pretty self-explanatory, but here’s an explanation anyway.

Let’s say the tom jail is running a MariaDB server on port 3306. The first line takes anything arriving on port 3306 and sends it to tom’s jail IP. Simple. It can reply because of the nat line earlier.

dick is running a web and ftp server, so ports 21,80 and 443 are sent there. The pf syntax lets you do nice stuff like this with the {..}

Finally we come to harry. Here we’re running an http server on port 80, but to make it accessible externally we’re mapping it to port 81 as otherwise it would clash with dick. In other words, if you don’t specify a destination port in the redirect it will assume the same as the source port.

And that’s it! When you jail is started you will see an interface lo1 with the IP address defined in /etc/jail.conf and assuming you have something sensible in /etc/resolv.conf you’ll have a jail that looks like it’s running behind a NAT router with port forwarding.

Of course, if you don’t need to map a jailed service to an external IP address, don’t! Jails can access services on each other using their own virtual network.

Posted on

Wake on LAN with FreeBSD and Broadcom Ethernet

To be fair, the Broadcom Ethernet adapter chips aren’t the best supported by FreeBSD. They’re okay, but for FreeBSD Intel rules the roost at 1-Gig and Chelsio at 10-Gig plus.

Unfortunately you may have no choice, as Broadcom chips are built in to the motherboards on some HP servers; notably the Microservers.

The Microserver Gen 7 is a great little box. I love them. I’m not so keen on the Gen 8 or Gen 10, as they’ve dropped a lot of the features that make the Gen 7 so adaptable.
So after much investigation, I decided to either forego WoL support, or bung in a cheap Realtek card simply to wake them up, but with only two PCIe slots, it was a pity to waste one on a better supported Ethernet NIC. That was a long time ago, and I’ve had one eye out for a fix for a while.

And then last year someone looked at it, and I found references to this in bug tracker. Unfortunately I don’t know who to credit with the fix, and it hasn’t made it to the kernel yet as it’s not 100%, but if you need it I’ve patched the 12.1 driver based on all the notes I could find.

There’s a question about the new driver’s correct operation on laptops. Why you’d be using WoL on a laptop eludes me; but for the Microservers it just works.

if_bge.c for 12.1Download
if_bge..c for 12.2Download

Download the patched driver ‘C’ file above (as edited by yours truly – no guarantees), and copy it to /usr/src/sys/dev/bge/if_bge.c, rebuild the kernel and away you go. As it was a bug with the state the chip was left on shutdown, you have to reboot the NEW kernel to shut down in the correct state for WoL to work. Don’t forget to enable WoL in the Microserver BIOS too.

Although this fixes the WoL issue, I know the Broadcom chips are capable of more than the driver – for example jumbo packets. If you want better network performance with a Microserver, add a dual-port Intel Ethernet card instead. Sorry, HP/Broadcom.