Posted on

Wake on LAN with FreeBSD and Broadcom Ethernet

To be fair, the Broadcom Ethernet adapter chips aren’t the best supported by FreeBSD. They’re okay, but for FreeBSD Intel rules the roost at 1-Gig and Chelsio at 10-Gig plus.

Unfortunately you may have no choice, as Broadcom chips are built in to the motherboards on some HP servers; notably the Microservers.

The Microserver Gen 7 is a great little box. I love them. I’m not so keen on the Gen 8 or Gen 10, as they’ve dropped a lot of the features that make the Gen 7 so adaptable.
So after much investigation, I decided to either forego WoL support, or bung in a cheap Realtek card simply to wake them up, but with only two PCIe slots, it was a pity to waste one on a better supported Ethernet NIC. That was a long time ago, and I’ve had one eye out for a fix for a while.

And then last year someone looked at it, and I found references to this in bug tracker. Unfortunately I don’t know who to credit with the fix, and it hasn’t made it to the kernel yet as it’s not 100%, but if you need it I’ve patched the 12.1 driver based on all the notes I could find.

There’s a question about the new driver’s correct operation on laptops. Why you’d be using WoL on a laptop eludes me; but for the Microservers it just works.

if_bge.c for 12.1Download
if_bge..c for 12.2Download

Download the patched driver ‘C’ file above (as edited by yours truly – no guarantees), and copy it to /usr/src/sys/dev/bge/if_bge.c, rebuild the kernel and away you go. As it was a bug with the state the chip was left on shutdown, you have to reboot the NEW kernel to shut down in the correct state for WoL to work. Don’t forget to enable WoL in the Microserver BIOS too.

Although this fixes the WoL issue, I know the Broadcom chips are capable of more than the driver – for example jumbo packets. If you want better network performance with a Microserver, add a dual-port Intel Ethernet card instead. Sorry, HP/Broadcom.

Posted on

FreeBSD 10.0 and ZFS




It’s finally here: FreeBSD 10.0 with ZFS. I’ve been pretty happy for many years with twin-drive systems protected using gmirror and UFS. It does what I want. If a disk fails it drops it out and sends me an email, but otherwise carries on. When I put a replacement blank disk it can re-build the mirror. If I take one disk out, put it into another machine and boot it, it’ll wake up happy. It’s robust!

So why mess around with ZFS, the system that puts your drives in to a pool and decides where things are stored, so you don’t have to worry your pretty little head about it? The snag is that the old ways are dying out, and sooner or later you’ll have no choice.

Unfortunately, the transition hasn’t been that smooth. First off you have to consider 2Tb+ drives and how you partition them. MBR partition tables have difficulties with the number of sectors, although AF drives with larger sectors can bodge around this. It can get messy though, as many systems expect 512b sectors, not 4k, so everything has to be AF-aware. In my experience, it’s not worth the hassle.

The snag with the new and limitless “GPT” scheme is that it keeps safe copies of the partition at the end of the disk, as well as the start. This tends to be where gmirror stores its meta-data too. You can’t mix gmirror and GPT. Although the code is hackable, I’ve got better things to do.

So the good new is that it does actually work as a replacement for gmirror. To test it I stuck two new 3Tb AF drives into a server and installed 10.0 using the new procedure, selecting the menu option zfs on root option and GPT partitioning. This is shown in the menu as “Experimental”, but seems to work. What you end up with, if you select two drives and say you want a zfs mirror, is just that.

Being the suspicious type, I pulled each of the drives in turn to see what had happened, and the system continues without a beat just like gmirror did. There were also a nice surprises when I stuck the drives back in and “onlined” them:

First-off the re-build was almost instant. Secondly, HP’s “non-hot-swap” drive bays work just fine for hot-swap under FreeBSD/ZFS. I’d always suspected this was a Windoze nonsense. All good news.

So why is the re-build so fast? It’s obvious when you consider what’s going on. The GEOM system works a block level. If the mirror is broken it has no way of telling which blocks are valid, so the only option is to copy them all. A major feature of ZFS, however, is that the directories and files have validation codes in the blocks above, going all the way to the root. Therefore, by starting at the root and chaining down, it’s easy to find the blocks containing changed data, and copy them. Nice! Getting rid of separate volume managers and file systems has its advantages.

So am I comfortable with ZFS? Not yet, but I’m a lot happier with it when its a complete, integrated solution. Previously I’d only been using on data drives in multi-drive configurations, as although it was possible to install root on ZFS, it was a real PITA.

Posted on

FreeBSD, Wake-on-LAN and HP Microservers – WOL compatible Ethernet





I’ve been having some difficulties getting Wake-on-LAN (WOL) to work with an HP Microserver thanks to its Broadcom Ethernet adapter not doing the business with the FreeBSD drivers – setting WOL in the Microserver BIOS doesn’t have any effect. I’m pleased to report a solution that works.

The on-board Broadcom Ethernet adaptor still refuses to play ball, for reasons described in my earlier post. The pragmatic solution is to use a better supported chip set and I’ve had no difficulties with Realtek (at the low end of the market) so it was an obvious choice. Just bung a cheap Realtek-based card in and use it as a remote “on” switch – what could possibly go wrong?

First off, the HP Microserver has PCI-Express slots, but weird looking ones. I’d assumed one was PCI when I’d glanced it, but it’s a PCIe 1-channel slot with something strange behind it – possibly a second 1-channel slot. The documentation says its for a remote management card; presumably one which doesn’t need access to the back. There’s a 16-channel PCIe next to it.All very curious but irrelevant here. The point is that you’ll need a PCIe Ethernet card – a surplus 100M PCI one with a well supported, bog-standard chip, won’t do. The PCIe cards tend to be 1Gb, and are therefore not as cheap.

The first card I bought was a TP-Link TG-3458, which has standard Realtek 8168B adapter chip. Or at least mine did; I note that there is a Mk2 version out there. Mine’s definitely a revision 1.2 PCB, but if you buy one now it may have the newer chip (which is a problem – read on below). Anyway, this Mk1 card worked like a charm. On sending it the magic packet and the Microserver bursts in to life. There’s only one snag: It has a full-height bracket and the Microserver has a half-height slot, so you have to leave the card floating in its socket. This works okay as long as no one trips over the cable.

My second attempt was an Edimax EN-9260TX-E, ordered because it was (a) cheap-ish; (b) had a Realtek chip; and (c) had the all-important half-height bracket. It fitted in the Microserver all right, but refused to act on a WOL, at least to begin with…

It turns out there was a little bug-ette in the driver code (prior to 8.3 or 9.1), spotted and fixed by the maintainer about a year ago. If you want to fix it yourself the patch is here. I decided I might as well use the latest drivers rather than re-working those shipped with 8.2, so pulled them, compiled a new if_re.ko and copied it to /boot/kernel in place of the old one. It didn’t work. Ha! Was I naive!

Further investigation revealed that it was completely ignoring this kernel module, as it was using a driver compiled in to the kernel directly. There was no point having the module there, all it does is trick you in to believing that it’s installed. I only realised “my” mistake when, to my astonishment, removing the file completely didn’t disable the network interface. I solved the problem by compiling a new kernel with the built-in Realtek driver commented out, and I’m currently loading the new driver specifically in loader.conf. It works a treat. I could have changed the kernel Realtek driver, but while it’s under review it’s easier to have it loaded separately. Incidentally, the driver is for 9.1 onwards but it works fine on 9.0 so far.

The next task is to fix the Broadcom driver so it works. I may be gone some time…

Posted on

HP Microserver and WOL

Update: See article here

 

They just don’t seem to work. I’ve spent an annoying hour or so trying to get WOL to work with an HP Microserver – no joy whatsoever. I assumed it must be my code until I tried it on a few other machines but they worked just fine.

Now most of my machines are Realtek whereas HP are using Broadcom (as do the Dells). I’m not saying there’s anything wrong with Broadcom, but whenever I have a weird network problem they have a habit of being at the heart of it. Is it my magic packet? As far as I know it’s supposed to be 48-bits of ‘1’ followed by sixteen copies of the MAC address. Does it need a secure-on password? If so, how come you can’t set one in the BIOS.

I’ve asked an HP server expert: “Update the BIOS”. Perhaps, but these are brand new machines of an established design. They either turn on when they receive the packet, or they don’t work, and I can’t believe HP didn’t test them. Then again…

I’m told that these do support WOL on Windows, but not if you’re running anything else. On the face of it this is bonkers. Why should the OS the powered-off drive affect anything. The machine is off; the OS isn’t running. Well here’s a theory – before Windows shuts off it puts something in a register on the Broadcom chip to leave it in a WOL state. With the wrong drivers this doesn’t happen. Setting it in the BIOS doesn’t help, because it’s erased by the OS driver. The BIOS doesn’t restore it as the power is killed, but Windows hits the registers differently.

Unfortunately Broadcom doesn’t seem keen on releasing the documentation needed to write proper drivers to anyone other than Microsoft. Is this my imagination? Everyone else publishes the reference material, but Broadcom – I can’t find it.

If anyone can throw light on this one, please do. I’m still looking.

Update

Fitting a Realtek-based NIC in the Microserver and using that instead solves the problem. WOL just works. If you’re going to order one, remember it’s PCIe, not PCI, and that you really need one with a low-profile bracket option because a full-height card won’t fit.

 Further Update: See article here