FreeBSD – Page 3 – Frank Leonhardt's Blog

ESXi, NFS, ZFS and vfs.nfsd.async

So there I was, reading the source code to FreeBSD’s nfsd (as you do), trying to figure out why ESXi’s performance was so bad when used with an NFS datastore in a ZFS dataset. Actually, I had some idea. There’s a lot out there on the interweb about whether it’s safe to tweak it to ignore requests to flush the write cache using the sysctl tunable vfs.zfs.cache_flush_disable. (For what it’s worth, I’d say that if your drives are on a UPS it’s fine).

But why does ESXis suck so badly in this respect with NFS connected datastores? What is this excessive cache flushing all about? I decided to install it on an HP Microserver and get to some serious debugging.

Okay, here is how ZFS writes work. When you write something it doesn’t actually write, it puts it in the ZIL. This is an Intent Log – i.e. writes intended to happen. Not exactly a write cache, but it has the same effect, and because of the way ZFS works it’s perfectly safe for avoiding data corruption. If a transaction is waiting in the ZIL when the music stops, the transaction is lots but the disk isn’t trashed. (NB. It’s also possible to put a ZIL on a log drive rather than RAM – I won’t discuss this here).

This should speed things up, right? Normally it does, but not when NFS is being abused. Let me explain. NFS has a transaction commit instruction. The client can tell NFS to flush everything in a transaction to permanent storage and not return until it’s finished. Sometimes you really need this, like if you’re updating the super-block in a database structure. Most of the time you don’t.

Enter ESXi running brain-dead Windows guest machines. How does it know when they’re writing something it isn’t a super-block? It doesn’t. So its solution (as far as I can tell) is to send NFS a commit after every single write and hang around waiting until it’s done it. There’s no point in having the ZIL at all, as it needs to be flushed every time. Putting the ZIL on disk is even worse, as you get an extra write/read for each transaction. I’ve seen people trying to put fast SSDs on the system to try and overcome this – best of luck with that.

As you move further down the chain, FreeBSD, being POSIX compliant whenever possible, will pass on the request for a synchronous write all the way to the disk. Send a block to a SATA or SAS drive and it will initially be cached, right? The write will then complete and the data actually written in the background while the rest of the system zips along. Except that it then issues a SATA or SAS “flush cache” command and waits until everything in its cache has been committed.

In tests this paranoid behaviour lead to running at 20% throughput or less.

Now, if you’re backing an emulated Windows disk you’re always at risk of data corruption, because FAT and NTFS are corruptable. And, dare I say it, crash rather too often. Let’s face it, if you’re worried about stuff like that you wouldn’t be running Windows – never mind as a VM, So lets be sensible about it.

So why was I reading the nfsd code? Well the obvious answer to this performance problem would be to simply ignore NTFS commit commands coming from the client. This is better than killing off all synchronous writes using the tunable vfs.zfs.cache_flush_disable because ZFS itself might be updating its uberblock and have a valid reason for doing it.

My plan was to hack the code – I’ve seen this done elsewhere. But wanting to do things properly I thought I should make it a system tunable. So I took a look at where the synchronous writes were happening – vdev_disk.c and vdev_geom.c (depending on whether you were hitting the raw drive or the GEOM). Lo and behold there was a global called nfs_sync that was compared along with the SYNC flag, and if either were true the sync request was ignored. So where did nfs_async come from? Digging further back it comes from nfs_nfsdserv.c , where it’s set by a system tuneable – vfs.nfsd.async. Now that’s an interesting name! Follow the stable auto variable in nfsrvd_write() and the nfs_async global if you want to see what I’m on about.

A quick Google for vfs.nfsd.async revealed – nothing. I seem to have found another useful tunable that’s yet to be documented. although it’s been in the source since at least 10.0. So I’ll get on to documenting after I’ve done a few more tests.

But if you’re having Windows/NFS problems, especially with ESXi, try setting vfs.nfsd.async instead of crudely disabling cache flushing with vfs.zfs.cache_flush_disable. Let me know how you get on.

Incidentally, you can disable synchronous writes to a dataset using the “sync=disabled” ZFS option. It helps, but not much. I’m still digging to find out why.
Or you could just use Virtualbox instead.

25-September-1625-September-16

FreeBSD 11 is here. Sort of.

According to freebsd.org, 11.0-RELEASE is still in the build stage and awaiting a release announcement this Wednesday. However, release builds have appeared on the FTP site:

ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/11.0/

This is obviously unofficial, and use them at your own risk.

At time of writing, the ARM and i386 versions is also available but not yet the PPC, ia64 or Sparc.

23-May-1623-November-21

FreeBSD, ZFS and Denial of Service

I’ve been using ZFS since FreeBSD 8, and it has it’s uses. It’s pretty be wonderful and all that, but I was actually pretty happy with UFS, and switching to ZFS isn’t a no-brainer.

So what’s the up-side to ZFS? Well you get more error checking and correction and it’s great for managing huge filing systems. You can snapshot and roll back, and do lots of other wonderful stuff with datasets and rive arrays. And it’s more “auto” when it comes to allocating disk space. But call me old fashioned if you like; I don’t like “auto” if I can avoid it.

Penguinistas might not “get” this next bit, but on a UNIX system you didn’t normally have One Big Disk. Instead you had several, and even if you only had one, you’d partition the slice it up so it looked like several. And then, of course, you’d mount disks or partitions on to the root filing system wherever you wanted them to appear.

For reliability, you could also create mirrors and striped RAIDs, put a FS on them and mount them wherever you wanted. And demount them, and mount them somewhere else, and so on.

ZFS does all this good stuff, but automatically, and often as One Big Disk. A good thing? Well… if you must. But there are a few points you might want to consider before diving in.

First off, I like to know where and on which disk my data actually resides. I’m really uneasy with ZFS deciding for me. If ZFS loses it, I want to know where to find it. I also like having a FS on each drive or partition, so I can pull the drive out and mount it wherever I want to get data off – or move it from machine to machine. It’s my data, I’ll do what I want to with it, dammit! You can do this virtually with ZFS datasets, but you can’t unplug a dataset and hold it in your hand. Datasets, of course, are fluid rather than fixed in size, so you don’t need to guess how much space to allocate.

Secondly, with UFS I get to decide what hardware is used for each kind of file. Parts of the FS that are rarely used can be put on slow, cheap, huge disks. The database goes on a velociraptor or better, and the swap partitions – well! Okay, you can use multiple zpools for difference performance situations but then you’re using it like UFS.

Thirdly, there’s a price for all this ZFS wonderfulness. Apart from the software overhead, the Copy-on-Write business needs a lot of RAM to maintain good performance. Fragmentation no the physical drive is guaranteed. If you’re running software (e.g. a database) that uses random access files and lots of transaction, UFS with its in-place modification wins out. A DBMS will take care of its own consistency and storage optimisation, and it has the edge as it knows what the data represents at the application level.

But what of the Denial of Service problem in the headline? Okay, it’s been a bit of a ramble, but this is something you must consider.

There are always management issues with One Big Disk. Linux users seem oblivious to this, but this doesn’t mean putting everything on a big partition is a great plan – even if you’re using a single disk in practice.

With the old way of having multiple partitions, each with an FS, mounted on the directory tree, when an FS on a partition/drive filled up, it is was full. You couldn’t create more files on it. You either have to delete unwanted stuff, or you can mount a bigger drive in its place. With One Big Disk, when it’s full, it’s also full. The difference is that you can’t write any data anywhere on the entire FS. And this is where DoS comes it.

Take, for example, /var/log. Any UNIX admin with a bit of sense will have this in its own partition. If some script kiddie then did something that caused a lot of log file activity, eventually you’d run out of space in /var/log. But the rest of the system would still be alive. With UFS the default installation process created partitions with sensible sizes. Using the One Big Disk principle, ZFS satisfies the requests of any disk-eating process until there isn’t a single byte left anywhere, and then rolls over saying the zpool is full. Or it would say it if there was a monitor connected to the server in a data centre miles away, and there was someone there to look at it.

With ZFS you can set a limit to the size on a dataset-by-dataset basis and prevent this sort of thing from happening. But it doesn’t happen by default, so set your quotas manually if you’re plonking the OS, and in particular /var on it.

Okay, this might sound a bit anti-ZFS, and I’ve yet to have a disaster with a ZFS system that’s required me to move drives around, so I don’t really know how possible it is when the chips are down. And ZFS has is a nice unified way of doing stuff, rather than fiddling around with geom and the FS separately. But after a couple of years with FreeBSD 10, where it became practical to boot from ZFS, shouldn’t I be feeling a bit more enthusiastic about it?

Having a ZFS pool attached as a data store rather than as a boot device is, of course, a different story. That’s when you see the benefits. But it does also eat resources, so I want the benefits to be worth it for the particular application. For the time being I’m putting the OS on UFS, usually with a data partition for databases to thrash, and userland putting simple files on ZFS – best of both worlds.

25-April-1627-April-16

UbuntuBSD – lovechild of Linux and FreeBSD

It’s no secret that Linux users with good taste have viewed the FreeBSD kernel with envious eyes for many years. A while back Debian distributions started having the FreeBSD kernel as an option instead of the Linux one. (Yes, you read that correctly). But now things seem to have been turned up a notch with UbuntuBSD.

It seems a group of penguinistas regard the Ubuntu world’s adoption of systemd as a step too far, and forked. And rather than keeping with Linux, they’ve opted to dump the whole kernel and bolt the Ubuntu front-end on to FreeBSD instead, getting kernel technology like ZFS and jails but “…keeping the familiarity of Ubuntu”.

Where could this be going? We already have PC-BSD for a “shrink wrapped” graphical desktop environment. Is anyone actually using it? I’m not. I’m sure we’ve all downloaded it out of curiosity, but if I want a Windows PC I’ll have a Windows PC. With BSD I’m more than happy with a command line, thank you very much.

UbuntuBSD could be different. Linux users actually use the graphical desktop, and most can’t cope with a command line. If they were to switch to FreeBSD instead, UbuntuBSD would make a lot of sense.

Although it’s only been around a month, in early beta form, its Sourceforge page is showing a lot of downloads. If I wanted to run a graphical desktop on top of FreeBSD, UbuntuBSD would make a lot of sense over PC-BSD, because I get the vibes that Ubuntu has desktop applications more together.

The project has just launched its own web site too, at www.ubuntubsd.org.

So does this spell the end of PC-BSD, Ubuntu Linux, Windows 10 or none of the above? It’s surely a strong vote against systemd.

20-April-1627-April-16

FreeBSD 10.3 hangs on upgrade – beware!

There seems to be a bit of a problem with upgrades to FreeBSD 10.3-RELEASE. Basically, shutdown -r is hanging, requiring you to manually reset the machine (turn it off and on again). This is annoying unless the machine in question happens to be at a data centre on a different continent, in which “annoying” really doesn’t cut it.

This was a known issue with 10.3-STABLE., but it appears to have made it in to -RELEASE too.

I suggest not using freebsd-update. Basically if you follow the official instructions you may need someone on hand to reboot the old fashioned way.

6-April-166-April-16

FreeBSD Device Driver Memory Allocation

Yesterday someone asked me how to allocate memory in a FreeBSD device driver. Although not quite as simple as a user-space malloc(), it’s relatively simple – but could I remember the name/parameter order? Not confidently, so I suggested RTFM.

A quick look at the manual doesn’t actually cover it very well. Basically there are special versions of malloc()/free() and they’re have exactly the same names, except the parameters are different. For example, malloc() has two extra parameters; one is the memory type (used for kernal statistics purposes), and one is a flags field, with options whether you’re prepared to wait, or is this a critical situation and using the reserve pool is okay.

For details, see “man 9 malloc”. The ‘9’ is important, as otherwise you’ll get the user-land version in libc. (Incidentally, a read through the libc code should put you off algorithms making wanton use dynamic memory allocation if you weren’t already).

Now what the FreeBSD documentation doesn’t tell you (and something for my to-do list) is how to actually make use of this in a device driver. I had to go back to code I’d written ten years ago to remind me, as I’m just as guilty of copying and tweaking my standard code many times over without really remembering what it does.

But before you go worrying about allocating dynamic memory in a device driver, consider that there’s no reason why you can’t just use static memory – just allocate in BSS in the normal way. Okay, this won’t suit every eventuality but on on most of my simple drivers, which have been to mess with custom hardware for a single process, it’s not actually a problem.

Okay, so you still want to use dynamic? Well to get the kernel versions instead of the the libc ones you need to include instead. As I mentioned above, for some reason using the same names must have seemed like a good idea at the time, but the parameters are different.

The other thing you should be aware of is when about allocating kernel memory you are talking about non-paged. Don’t go crazy.

There is also a memory allocation tracker and statistics dumper available in the libc version (see /etc/malloc.conf), which will help you out if you’ve messed up memory allocation. Don’t expect any such help with the kernel. However, if you compile the kernel with the INVARIANTS option set it will scrub freed memory with 0xdeadc0de, which is handy if you find yourself using unallocated or free kernel RAM. Actually, this is a pretty good idea if you’re writing KLDs anyway, as it stops and does a core dump at the first sign you’ve screwed up any kernel structures.

The documentation in “man 9 malloc” should be enough to cope with the extra parameters; basically the malloc_type. Note that the first parameter to the MALLOC_DEFINE macro is actually a name you make up! By convention it’s in the form M_XXXXX, in upper case.

Also note that when you’re freeing memory it’s not normally zeroed. Therefore someone else using kernel memory might be able to allocate it and read what your driver wrote. Okay, bug deal – if the bad guys are installing kernel modules it’s game anyway. But… consider the bad guys cause a kernel panic and get a core dump.

22-March-16

FreeBSD sysarch kernel panic vulnerability

A bug has been found and fixed in the FreeBSD kernel that would allow someone with malicious intent to crash a running system. It’d be difficult to achieve unless the attacker had console access. However it’s been patched for all supported systems. See here for all the details (which I won’t repeat).

The problem was found by Core Security, and they have provided an excellent write-up here.

But if you want it in plain English:

The sysarch() system call is used to get/set processor-specific stuff. You’re not supposed to call it directly; you’re supposed to call a processor-specific library if you want to do things like that, but you still can call it if you want to. On processors that support memory segments, such as i386, there is a Local Descriptor Table (LDT) to manage them if you want to mess with specific stuff like that. However, for security reasons, you can only modify the LDT using the sysarch() call, which checks what you’re trying to do and prevents applications from doing anything crazy.

Unfortunately the AMD64 implementation of the code gets the checking wrong. If you use a signed integer it’s always going to be less than another unsigned value, and when it compares the two parameters to make sure that one is less than the other it passes when it shouldn’t, and the rogue parameter causes it to go funky-deux and overwrite a shed load of stuff.

This is in all in:

/sys/amd64/amd64/sys_machdep.c

in the function:

int amd64_set_ldt(td, uap, descs)

The FreeBSD advisory contains a patch for all “supported” versions; but what if you’re using an older one? Using the information from Core it’s easy enough to patch. But what else is affected?

To save you the trouble, I’ve looked back at earlier versions. The problem code definitely exists in the AMD64 versions for 8.x, but isn’t present in any 7.x, as far as I can tell. The system call simply doesn’t exist. On i386 versions, I can’t see any obvious problem with the code.

How worried should we be? If someone breaks in to a system with shell access, they will be able to crash it. However, I think it’s very unlikely that any service is written in such a way that malicious data could cause the necessary parameters to be sent to sysarch() call. In fact, on checking the ports collection, it’s not exactly used all over the place. You’re highly unlikely to be running any application that even makes the call.

7-December-157-December-15

How to stop Samba users deleting their home directory and email

UNIX permissions can send you around the twist sometimes. You can set them up to do anything, not. Here’s a good case in point…

Imagine you have Samba set up to provide users with a home directory. This is a useful feature; if you log in to the server with the name “fred” you (and only you) will see a network share called “fred”, which contains the files in your UNIX/Linux home directory. This is great for knowledgeable computer types, but is it such a great idea for normal lusers? If you’re running IMAP email it’s going to expose your mail directory, .forward and a load of other files that Windoze users might delete on a whim, and really screw things up.

Is there a Samba option to share home directories but to leave certain subdirectories alone? No. Can you just change the ownership and permissions of the critical files to root and deny write access? No! (Because mail systems require such files to be owned by their user for security reasons). Can you use permission bits or even an ACL? Possibly, but you’ll go insane trying.

A bit of lateral thinking is called for here. Let’s start with the standard section in smb.conf for creating automatic shares for home directories:

[homes]
    comment = Home Directories
    browseable = no
    writable = yes

The “homes” section is special – the name “homes” is reserved to make it so. Basically it auto-creates a share with a name matching the user when someone logs in, so that they can get to their home directory.

First off, you could make it non-writable (i.e. set writable = no). Not much use to use luser, but it does the job of stopping them deleting anything. If read-only access is good enough, it’s an option.

The next idea, if you want it to be useful, is to use the directive “hide dot files” in the definition. This basically returns files beginning in a ‘.’ as “hidden” to Windoze users, hiding the UNIX user configuration files and other stuff you don’t want deleted. Unfortunately the “mail” directory, containing all your loverly IMAP folders is still available for wonton destruction, but you can hide this too by renaming it .mail. All you then need to do is tell your mail server to use the new name. For example, in dovecot.conf, uncomment and edit the line thus:

mail_location = mbox:~/.mail/:INBOX=/var/mail/%u

(Note the ‘.’ added at the front of ~/mail/)

You then have to rename each of the user’s “mail” folders to “.mail”, restart dovecot and the job is done.

Except when you have lusers who have turned on the “Show Hidden Files” option in Windoze, of course. A surprising number seem to think this is a good idea. You could decide that hidden files allows advanced users control of their mail and configuration, and anyone messing with a hidden file can presumably be trusted to know what you’re doing. You could even mess with Windoze policies to stop them doing this (ha!). Or you may take the view that all lusers and dangerous and if there is a way to mess things up, they’ll find it and do it. In this case, here’s Plan B.

The trick is to know that the default path to shares in [homes] is ‘~’, but you can actually override this! For example:

[homes]
    path = /usr/data/flubnutz
    ...

This maps users’ home directories in a single directory called ‘flubnutz’. This is not that useful, and I haven’t even bothered to try it myself. When it becomes interesting is when you can add a macro to the path name. %S is a good one to use because it’s the name as the user who has logged in (the service name). %u, likewise. You can then do stuff like:

[homes]
     path = /usr/samba-files/%S
     ....

This stores the user’s home directory files in a completely different location, in a directory matching their name. If you prefer to keep the user’s account files together (like a sensible UNIX admin) you can use:

[homes]
     comment = Home Directories
     path = /usr/home/%S/samba-files
     browseable = no
     writable = yes<

As you can imagine, this stores their Windows home directory files in a sub-directory to their home directory; one which they can’t escape from. You have to create “~/samba-files” and give them ownership of it for this to work. If you don’t want to use the explicit path, %h/samba-files should do instead.

I’ve written a few scripts to create directories and set permissions, which I might add to this if anyone expresses an interest.

5-November-156-November-15

FreeBSD on 96-core 64-bit ARMv8

A couple of year’s back I managed to compile and run FreeBSD/Apache/BIND on an ARM-based Raspberry Pi. It was fun, but I have to admit it’s been left on the shelf ever since. A solution waiting for a problem.

Since then the ARM has been a specific target for FreeBSD 11. Do you really need FreeBSD on your smartphone? However much I like BSD, the Linux-based Android does well enough. But wait…

ARM has a 64-bit turbo-nutter-bastard version waiting in the wings, for server use. The ARMv8 is scalable to at least 48 cores per socket and intended to go like the clappers in SMP applications. FreeBSD has long been considered to have the edge over the Linux kernel when it comes to SMP. This is getting interesting.

A team including Semihalf has now got FreeBSD 11 stable running on a twin-CPU monster using the Cavium ThunderX ARM chips, each of which has 48-cores. For details see their blog. With a lot of serious web applications running FreeBSD in preference to the freewheeling Linux, there could be a very ready market for this kind of box.

I would be in danger of being extremely jealous, as my budget for playing with ARM chips doesn’t stretch much past the Raspberry Pi. However, in the 1980’s Atari Research gave me an ATW transputer box with 128 discrete CPUs to help implement an OS on, so I’m still 32-cores ahead. There wasn’t much of a market for the ATW back then, but Cavium could be on to a winner with the approach nearly thirty years later.

Footnote

Does anyone know what happened to the big transputer box prototype that was knocking around the Cambridge office? When the ATW/Abaq was released it was greatly scaled down with no more than 13 transputers, and lacked the glass case with all the flashing lights.

12-October-1512-October-15

Installing Apache 2.4 with PHP on FreeBSD for Drupal 8. It’s a Nightmare

I’ve been playing about the Drupal 8 (still in Beta) and one of its features is that it needs the latest version of PHP (5.5.9 or later). I have a server I keep for testing the latest whatever, and this includes Apache 2.4. So how hard can it be to compile in PHP?

Actually, it’s not straightforward. Apache 2.4 is fine, but PHP is another matter. First off, installing lang/php55 does not include mod_php for Apache. It’s not that the option to compile it hasn’t been set – the option has gone. With a bit of digging around you can find it elsewhere – in www/mod_php55. Don’t be fooled in to thinking you need to just build and install that though…

You’ll probably end up with stuff like this in your httpd error log:

Call to undefined function session_name() Call to undefined function hash()

Digging further you’ll find www/php55-session and security/php55-hash in there, and go off to build those too. Then wonder why it still isn’t working.

The clue can be found with this log file error:

PHP Warning: PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20121212-zts/session.so' - Cannot open &quote;/usr/local/lib/php/20121212-zts/session.so&quote; in Unknown on line 0

(NB. The &quote appears in the log file itself!)

Basically, mod_php expects you to compile the ZTS (Zend Thread Safe) version of everything. And why wouldn’t you? Well it turns out that this important option is actually turned off by default so you need to configure the build to include it. Any extensions you’ve compiled up until now will not have been placed in a directory tagged with -zts, which is why it’s looking in the wrong place as shown by the error log.

If you’re reading this following a Google search, you’ve probably already fallen down the Pooh trap. You need to go back to lang/php55 and start again with the correct options. The best way to do this (in case you didn’t know) is:

make clean make config make make install

When you run make config it’ll give you a chance to select ZTS, so do it.

Repeat this for compiling www/mod_php55 and then go back and compile www/php55-session, security/php55-hash and anything else you got wrong the first time, You don’t have the option to configured them, but they must be compiled again once the core of PHP has been compiled using ZTS.

Incidentally, if you haven’t had this pain before, you will probably need to switch to using the new pkg system if you haven’t already. Trying to build without it, it’ll put up a curt little note about it and go in to sulk mode until you do. Unfortunately, on an older FreeBSD, any attempt to compile this will result in an O_CLOEXEC symbol undefined error in pkg.c. This is actually a flag to the open() kernel function that was added to POSIX in 2008. What it means is that if your process subsequently makes exec call, the file handle will be automatically closed. It saves leaking fds if your execution path goes awry. But what’s the solution?

Well, if you’re using an older version of the kernel then it won’t support O_CLOEXEC anyway, so my fix is to delete it from the source and try again. It only appears once, and if the code is so sloppy that it doesn’t close the handle, it’s not the end of the world. The official answer is, of course, to upgrade your kernel.

If you are running Drupal 8, here’s a complete list of the ports you’ll need to compile:

lang/php55 (select ZTS option in the configuration dialogue)
www/mod_php55 (select ZTS option in the configuration dialogue)
www/php55-session
security/php55-hash
security/php55-filter
devel/php55-json
devel/php55-tokenizer (for Drupal 8)
databases/php55-pdo
databases/php55-pdo_mysql
textproc/php55-ctype
textproc/php55-dom
textproc/php55-simplexml
graphics/php55-gd
converters/php55-mbstring (not tested during setup)

All good fun! This relates to Drupal 8.0.0 RC1 – it may be different with the final release, of course.