Facebook has user data slurped

The following has just appeared on Facebook’s press release page:

Security Update

“On the afternoon of Tuesday, September 25, our engineering team discovered a security issue affecting almost 50 million accounts….”

“Our investigation is still in its early stages. But it’s clear that attackers exploited a vulnerability in Facebook’s code that impacted… a feature that lets people see what their own profile looks like to someone else.”

Mark Zuckerberg’s understated response to the incident was “I’m glad we found this and fixed the vulnerability. It definitely is an issue that this happened in the first place. I think this underscores the attacks that our community and our services faces.”

Wall Street’s response so far has been a 3% drop in Facebook’s stock.

I’m now waiting to see which of my sock puppets is affected.

Tesco Bank hit with £30m fine for computer breach

According to a Sky News exclusive, the FCA is set to clobber Tesco Bank with a fine of £30m over the data breach in late 2016, where £2.5m was snaffled from thousands of its customer’s current accounts. Except it turned out it wasn’t; only fifty accounts were actually plundered, not for very much, and it was all sorted.

So how does this warrant such a huge fine? It’s hard to see, but the first two theories I have are that Sky News has got of wrong, or the FCA has gone seriously bonkers. If they’re touching miscreant institutions for £600K per customer inconvenienced, RBS and NatWest are toast.

So what’s it all about? Well we don’t know what Tesco Bank actually did. My best guess is that someone cloned cards and cashed out at ATMs. That’s the easiest way, and there is no evidence this was widespread or sophisticated. And its interesting that only current accounts were hit; not credit – which is where the big money is in retail banking fraud.

But that’s just a guess. Why would the FCA be so exercised about some card fraud?

There is not shortage of other theories. There is the usual criticism of the patent company and its insecure non-banking systems. The usual unpatched server card is played. Yes, everyone knows Tesco self-checkouts use Windows XP. There ate criticisms of the lack of protective monitoring. Lack of AV. But this comes from commentators whose employer’s business is selling such things. There is talk of an inside job, which is possible but they didn’t take them for much if it was.

So if the FCA is really that cross with Tesco Bank, why?

The question no one is asking is why Tesco Bank announced a major breach, affecting so many people? Here I’m stacking guesses, but just for fun…

If I’m right about it being ATM bandits, could it be that staff investigating found something horrible and hairy, and jumped to the conclusion it was behind it? They did the right thing, and told everyone about the vulnerability, but the black hats hadn’t. The FCA would have been unimpressed, regardless of the consequences, and whacked them according.

If I’m right, it’s a bit rough on Tesco Bank, fined as a result of being robbed. But this is all one guess based on another. The truth may be still stranger.

Facebook shares worth a punt

The confected row about Facebook and CA’s mining of the latter’s users’ data beggars belief. Facebook’s raison d’être is to profile its users and sell the information to anyone needing to target messages (adverts). The punters sign up to this because access is free. They might not understand what they’re agreeing to; a quick look at Facebook shows that many users are far from the brightest lights in the harbour. Buy hey, it’s free!

This is basically how Web 2.0 works. Get the punters to provide the content for you, collect information of value to sell to advertisers, and use the money to pay for the platform. Then trouser a load of tax-free profit by exploiting the international nature of the Internet.

So why the brouhaha now? Where has the moral outrage been for the last ten years? How come punters have only just started talking of a boycott (about twelve years after I did)? What’s changed?

The media has suddenly taken notice because some messages were sent on behalf of Donald Trump’s presidential campaign. What might broadly be called “left-wing” politicians have been exploiting unregulated social media to sway opinion for a very long time. Some became very uncomfortable when Trump gained traction by “speaking directly to his supporters” on Twitter. And now they’ve finally woken up to the way that the simple majority using a social media platform are able to propagate fake news and reinforce their simplistic beliefs.

But it wasn’t until the recent revaluations that Donald Trump was using it that anyone batted an eyelid.

This rabbit hole goes very deep.

Does this spell the end of Facebook? I somehow doubt it. Social media addicts are just that. They don’t want to lose all their virtual “friends”. They want people to “like” them. Those that realise it’s a load of fluff try to cut back, or “detox” for a few weeks, but they always come back for more. And for those who see social media for what it and have nothing to do with it are constantly pressured by the addicts, like a drug user turned pusher.

“You don’t use Facebook? How are we supposed to contact you?”

No. This row doesn’t spell the end of Facebook. I know MySpace, bix, CompuServe, Geocities and the rest went out of fashion, but Facebook and Twitter are too well established, and even promoted on the BBC. And if the addicts were outraged enough to move to a different platform, where would they go? Part of their addiction comes from Facebook being “free”, and no one has come up with an alternative business model that works. They’ll stick with the devil they know.

Meanwhile investors have the jitters and the share price has fallen. This won’t last.

 

Look at the size of my virtual pipe!

Many years ago I decried the new mania for virtual servers as a fix for Windows’ limitations in allowing services to be moved from one host to another. They’re also being used in the Linux world (particularly) in the form of “appliance architecture”, where services are not run on operating systems but whole systems are run within systems. I guess this allows non-technical people to visualise them better or something.

The situation is getting out-of-hand. People don’t understand they’re using a paradigm, and not a computer. This is leading to a lot of nuttery.

I’ve seen an instance when two virtual servers (running on one host) were running a service between them with a virtual load balancer in front in an attempt to improve performance. This was in a production environment. I only hope that whoever designed the system assumed it was going to run on real hardware, and then some muppet came along and simply copied a prototype to “the cloud”.

Reality check people: You may have something that looks like lots of small computers, but underneath there’s just one of them – and you’re sharing it with other customers. By virtualizing lots of small servers you’re just burning cycles on the big one, and retarding its disk performance. It’s a bonkers as a perpetual motion machine; it’s never going to run as fast as it would have directly on the host.

I’ve even heard people comparing one virtual host with another as if it was real hardware. Mine’s got 64Gb of RAM! Well mine is all SSD and a 16-core Xeon!

No you haven’t! You’ve got a software emulation of whatever your provider has sold you, running at whatever speed is left after the other customers have taken their chunk. You don’t have any RAM at all. Your OS thinks it has, but the whole OS could be swapped out. It’s disk accesses go through the hypervisor cache, and to its backing store at whatever speed it goes at. It may not look like your memory is paged, but the hypervisor is certainly going to be paging it anyway. If you feel better thinking you’ve got all the RAM you need, please continue in your virtual wonderland.

Ah, but you’ve got Elastic Computing, and can inflate the size of your RAM number of CPUs as demand increases. Let me tell you, an inflatable is never as good as the real thing. And your high demand may coincide with someone else’s. So you “reserve” the resources needed to cope with your peak demand. Hmm. Sounds a bit like having your own hardware to me.

I use one cloud server provider – vultr.com. It’s a bit of a love-hate relationship as, in case you didn’t realise, I don’t think much of cloud computing and anyway, I can afford to have my own. But if you need a small service on the end of an IP address on the other side of the world, they’re just what you need. I was amused to note that my “512Mb/20Gb” virtual server believed it came equipped with a 10Gb NIC talking to the Internet. Software emulation of 10Gbps anyone? And then there’s the contention ratio to worry about.

 

If I had a pound for every time someone asked me about BitCoin

IFrank Leonhardtt was no surprise when people started asking me about Bitcoin. Money is of great interest to a lot of people; mix it with technology and they want to talk about it.

The main question asked is “Should I buy some?”, closely followed by “Is it safe?”, and “Do you think it’s a bubble?”

To answer the last one first: “Of course it’s a bubble you idiot”. I don’t think there’s anyone who believes it isn’t, but greed conquers common sense. And investing in a bubble can be a rational strategy as long as you make sure you take your capital out before it bursts. You could say the same about any form of investment to some extent. The value of shares will rise and fall in the long term, and everyone knows you should spread the risk. Seeing the return for a punt on Bitcoin at the moment persuades some to abandon this golden rule and put all their funds at risk.

As to whether the technology is safe: No way! It’s as safe as the security of the computers it is stored on, and the integrity of those storing it. Good luck with that. Technically, blockchain technology itself looks very secure but that isn’t where the risk lies.

And now we get back to the main question: Should I buy some? Well I wouldn’t, simply because it’s immoral.

Yes folks, if you can see beyond the chance of a fast buck, Bitcoin is sleaze. There are a few fundamental truths about cash it might be worth reiterating.

Back at the dawn of history, humans realised they’d be better off if they traded. If you had a lot of grain but no apples, find someone with apples and no grain who wanted to do a swap. Cash emerged so you could defer a transaction; or enter in to multi-party deals more easily by extracting the value from the item and placing it in to something more convenient (small pieces of soft shiny metal).

A coin’s value depends on whether you can buy what you need with it at a later date. If you exchange your grain for a coin you have to be convinced that the apple dealer will exchange the coin for your apples. Coins are a matter of confidence; confidence that they can be exchanged for something useful later.

If coins were easy to make, people would just make coins and the apple dealer would end up with a load of inedible shiny metal fragments; so there must be a finite supply for cash to work if the cash has representative rather than commodity value. Prisoners have often used cigarettes as they also have commodity value in that you can smoke them. Leaves, on the other hand, are a poor choice of currency as they grow on trees.

With no commodity value, you might ask why Bitcoin works at all? There are effectively a finite number of valid bitcoins, so you can’t make your own. And people have confidence that they can be exchanged for the goods they need at a later date. Perhaps not as much confidence as they do with regulated currencies, but their big advantage is that they are outside the regulatory system, and like cash or cigarettes, are ideal for black market transactions.

The bottom line is that criminals accept Bitcoin for the purchase of drugs, weapons and extortion payments. Like the legitimate world using BACS/CHAPS/CHIPS (electronic Bank payments), organised crime in the 21st Century benefits from a black money clearing system: Bitcoin. Cryptocurrency has a value because it can be used for buying drugs in large quantities across international borders far more conveniently than using the old-school suitcase of dollar bills. No questions asked. If you want to buy narcotics, you need to buy Bitcoin to pay the dealers with.

Like any currency with a floating exchange rate, the value of a Bitcoin should fluctuate based on the supply and demand for the illegal goods and services it represents. If the demand goes up and supply remains the same, the value of Bitcoin would rise as purchasers out-bid each other to secure enough Bitcoin to pay their dealer. I strongly suspect that knee-jerk (or just jerk) investors are seeing a rise in cost, and not looking too deeply at the tangible commodities backing it. Or perhaps city speculators are not being greedy and stupid; perhaps they really do need Bitcoin to pay for their coke habits.

So, as to whether I think Bitcoin is a good investment, they only answer is: “Yes – it’s can be just as profitable other parts of the drugs trade if you can get it right.”

What’s the point of Docker on FreeBSD or Solaris?

Penguinisters are very keen on their docker, but for the rest of us it may be difficult to see what the fuss is all about – it’s only been around a few years and everyone’s talking about it. And someone asked again today. What are we missing?

Well docker is a solution to a Linux (and Windows) problem that FreeBSD/Solaris doesn’t have. Until recently, the Linux kernel only implemented the original user isolation model involving chroot. More recent kernels have had Control Groups added, which are intended to provide isolation for a group of processes (namespaces). This came out of Google, and they’ve extended to concept to include processor resource allocation as one of the knobs, which could be a good idea for FreeBSD. The scheduler is aware of the JID of the process it’s about to schedule, and I might take a look in the forthcoming winter evenings. But I digress.

So if isolation (containerisation in Linux terms) is in the Linux kernel, what is Docker bringing to the party? The only thing I can think of is standardisation and an easy user interface (at the expense of having Python installed). You might think of it in similar terms to ezjail – a complex system intended to do something that is otherwise very simple.

To make a jail in FreeBSD all you need do is copy the files for your system  to a directory. This can even be a whole server’s system disk if you like, and jails can run inside jails.  You then create a very simple config file, giving the jail a name, the path to your files and an what IP addresses to pass through (if any) and you’re done. Just type “service jail nameofjal start”, and off it goes.

Is there any advantage in running Docker? Well, in a way, there is. Docker has a repository of system images that you can just install and run, and this is what a lot of people want. They’re a bit like virtual appliances, but not mind-numbingly inefficient.

You can actually run docker on FreeBSD. A port was done a couple of years ago, but it relies on the 64-bit Linux emulation that started to appear in 10.x. The newer the version of FreeBSD the better.

Docker is in ports/sysutils/docker-freebsd. It makes uses of jails instead of Linux cgroups, and requires ZFS rather than UFS for file system isolation. I believe the Linux version uses Union FS but I could be completely wrong on that.

The FreeBSD port works with the Docker hub repository, giving you access to thousands of pre-packaged system images to play with. And that’s about as far as I’ve ever tested it. If you want to run the really tricky stuff (like Windows) you probably want full hardware emulation and something like  Xen. If you want to deploy or migrate FreeBSD or Solaris systems, just copy a new tarball in to the directory and go. It’s a non-problem, so why make it more complicated?

Given the increasing frequency Docker turns up in conversations, it’s probably worth taking seriously as Linux applications get packaged up in to images for easy access. Jails/Zones may be more efficient, and Docker images are limited to binary, but convenience tends to win in many environments.

Del FS12-NV7 and other 2U server (e.g. C6100) disk system hacking

(Photographs to follow)

A while back I reviewed the Dell FS12-NV7 – a 2U rack server being sold cheap by all and sundry. It’s a powerful box, even by modern standards, but one of its big drawbacks is the disk system it comes with. But it needn’t be.

There are two viable solutions, depending on what you want to do. You can make use of the SAS backplane, using SAS and/or SATA drives, or you can go for fewer SATA drives and free up one or more PCIe slots as Plan B. You probably have an FS12 because it looks good for building a drive array (or even FreeNAS) so I’ll deal with Plan A first.

Like most Dell servers, this comes with a Dell PERC RAID SAS controller – a PERC6/i to be precise. This ‘I’ means it has internal connectors; the /E is the same but its sockets are external.

The PERC connects to a twelve-slot backplane forming a drive array at the front of the box. More on the backplane later; it’s the PERCs you need to worry about.

The PERC6 is actually an LSI Megaraid 1078 card, which is just the thing you need if you’re running an operating system like Windows that doesn’t support a volume manager, striping and other grown-up stuff. Or if your OS does have these features, but you just don’t trust it. If you are running such an OS you may as well stick to the PERC6, and good luck to you. If you’re using BSD (including FreeNAS), Solaris or a Linux distribution that handles disk arrays, read on. The PERC6 is a solution to a problem you probably don’t have, but in all other respects its a turkey. You really want a straightforward HBA (Host Bus Adapter) that allows your clever operating system to talk directly with the drives.

Any SAS card based on the 1078 (such as the PERC6) is likely to have problems with drives larger than 2Tb. I’m not completely sure why, but I suspect it only applies to SATA. Unfortunately I don’t have any very large SAS drives to test this theory. A 2Tb limit isn’t really such a problem when you’re talking about a high performance array, as lots of small drives are a better option anyway. But it does matter if you’re building a very large datastore and don’t mind slower access and very significant resilvering times when you replace a drive. And for large datastores, very large SATA drives save you a whole lot of cash. The best capacity/cost ratio is for 5Gb SATA drives

Some Dell PERCs can be re-flashed with LSI firmware and used as a normal HBA. Unfortunately the PERC6 isn’t one of them. I believe the PERC6/R can be, but those I’ve seen in a FS12 are just a bit too old. So the first thing you’ll need to do is dump them in the recycling or try and sell them on eBay.

There are actually two PERC6 cards in most machine, and they each support eight SAS channels through two SFF-8484 connectors on each card. Given there are twelve drives slots, one of the PERCs is only half used. Sometimes they have a cable going off to a battery located near the fans. This is used in a desperate attempt to keep the data in the card’s cache safe in order to avoid write holes corrupting NTFS during a power failure, although the data on the on-drive caches won’t be so lucky. If you’re using a file system like that, make sure you have a UPS for the whole lot.

But we’re going to put the PERCs out of our misery and replace them with some nice new LSI HBAs that will do our operating system’s bidding and let it talk to the drives as it knows best. But which to pick? First we need to know what we’re connecting.

Moving to the front of the case there are twelve metal drive slots with a backplane behind. Dell makes machines with either backplanes or expanders. A backplane has a 1:1 SAS channel to drive connection; an expander takes one SAS channel and multiplexes it to (usually) four drives. You could always swap the blackplane with an expander, but I like the 1:1 nature of a backplane. It’s faster, especially if you’re configured as an array. And besides, we don’t want to spend more money than we need to, otherwise we wouldn’t be hot-rodding a cheap 2U server in the first place – expanders are expensive. Bizarrely, HBAs are cheap in comparison. So we need twelve channels of SAS that will connect to the sockets on the backplane.

The HBA you will probably want to go with is an LSI, as these have great OS support. Other cards are available, but check that the drivers are also available. The obvious choice for SAS aficionados is the LSI 9211-8i, which has eight internal channels. This is based on an LSI 2000 series chip, the 2008, which is the de-facto standard. There’s also four-channel -4i version, so you could get your twelve channels using one of each – but the price difference is small these days, so you might as well go for two -8i cards. If you want cheaper there are 1068-based equivalent cards, and these work just fine at about half the price. They probably won’t work with larger disks, only operate at 3Gb and the original SAS standard. However, the 2000 series is only about £25 extra and gives you more options for the future. A good investment. Conversely, the latest 3000 series cards can do some extra stuff (particularly to do with active cables) but I can’t see any great advantage in paying megabucks for one unless you’re going really high-end – in which case the NV12 isn’t the box for you anyway. And you’d need some very fast drives and a faster backplane to see any speed advantage. And probably a new motherboard….

Whether the 6Gb SAS2 of the 9211-8i is any use on the backplane, which was designed for 3Gb, I don’t know. If it matters that much to you you probably need to spend a lot more money. A drive array with a direct 3Gb to each drive is going to shift fast enough for most purposes.

Once you have removed the PERCs and plugged in your modern-ish 9211 HBAs, your next problem is going to be the cable. Both the PERCs and the backplane have SFF-8484 multi-lane connectors, which you might not recognise. SAS is a point-to-point system, the same as SATA, and a multi-lane cable is simply four single cables in a bundle with one plug. (Newer versions of SAS have more). SFF-8484 multi-lane connectors are somewhat rare, (but unfortunately this doesn’t make them valuable if you were hoping to flog them on eBay). The world switched quickly to the SFF-8087 for multi-lane SAS. The signals are electrically the same, but the connector is not.

So there are two snags with this backplane. Firstly it’s designed to work with PERC controllers; secondly it has the old SFF-8484 connectors on the back, and any SAS cables you find are likely to have SFF-8087.

First things first – there is actually a jumper on the backplane to tell it whether it’s talking to a PERC or a standard LSI HBA. All you need to do is find it and change it. Fortunately there are very few jumpers to choose from (i.e. two), and you know the link is already in the wrong place. So try them one at a time until it works. The one you want may be labelled J15, but I wouldn’t like to say this was the same on every variant.

Second problem: the cable. You can get cables with an SFF-8087 on one end and an SFF-8484 on the other. These should work. But they’re usually rather expensive. If you want to make your own, it’s a PITA but at least you have the connectors already (assuming you didn’t bin the ones on the PERC cables).

I don’t know what committee designed SAS cable connectors, but ease of construction wasn’t foremost in their collective minds. You’re basically soldering twisted pair to a tiny PCB. This is mechanically rubbish, of course, as the slightest force on the cable will lift the track. Therefore its usual to cover the whole joint in solidified gunk (technical term) to protect it. Rewiring SAS connectors is definitely not easy.

I’ve tried various ways of soldering to them, none of which were satisfactory or rewarding. One method is to clip the all bare wires you wish to solder using something like a bulldog clip so they’re at lined up horizontally and then press then adjust the clamp so they’re gently pressed to the tracks on the board, making final adjustments with a strong magnifying glass and a fine tweezers. You can then either solder them with a fine temperature-controlled iron, or have pre-coated the pads with solder paste and flash across it with an SMD rework station. I’d love to know how they’re actually manufactured – using a precision jig I assume.

The “easy” way is to avoid soldering the connectors at all; simply cut existing cables in half and join one to the other. I’ve used prototyping matrix board for this. Strip and twist the conductors, push them through a hole and solder. This keeps things compact but manageable. We’re dealing with twisted pair here, so maintain the twists as close as possible to the board – it actually works quite well.

However, I’ve now found a reasonably-priced source of the appropriate cable so I don’t do this any more. Contact me if you need some in the UK.

So all that remains is to plug your HBAs to the backplane, shove in some drives and you’re away. If you’re at this stage, it “just works”. The access lights for all the drives do their thing as they should. The only mystery is how you can get the ident LED to come on; this may be controlled by the PERC when it detects a failure using the so-called sideband channel, or it may be operated by the electronics on the backplane. It’s workings are, I’m afraid, something of a mystery still – it’s got too much electronics on board to be a completely passive backplane.

Plan B: SATA

If you plan to use only SATA drives, especially if you don’t intend using more than six, it makes little sense to bother with SAS at all. The Gigabyte motherboard comes with half a dozen perfectly good 3Gb SATA channels, and if you need more you can always put another controller in a PCIe slot, or even USB. The advantages are lower cost and you get to free up two PCIe slots for more interesting things.

The down-side is that you can’t use the SAS backplane, but you can still use the mounting bays.

Removing the backplane looks tricky, but it really isn’t when you look a bit closer. Take out the fans first (held in place by rubber blocks), undo a couple of screws and it just lifts and slides out. You can then slot and lock in the drives and connect the SATA connectors directly to the back of the drives. You could even slide them out again without opening the case, as long as the cable was long enough and you manually detached the cable it when it was withdrawn. And let’s face it – drives are likely to last for years so even with half a dozen it’s not that great a hardship to open the case occasionally.

Next comes power. The PSU has a special connector for the backplane and two standard SATA power plugs. You could split these three ways using an adapter, but if you have a lot of drives you might want to re-wire the cables going to the backplane plug. It can definitely power twelve drives.

And that’s almost all there is to it. Unfortunately the main fans are connected to the backplane, which you’ve just removed. You can power them from an adapter on the drive power cables, but there are unused fan connectors on the motherboard. I’m doing a bit more research on cooling options, but this approach has promising possibilities for noise reduction.

Dell FS12-NV7 Review – Bargain FreeBSD/ZFS box

It seems just about everyone selling refurbished data centre kit has a load of Dell FS12-NV7’s to flog. Dell FS-what? You won’t find them in the Dell catalogue, that’s for sure. They look a bit like C2100s of some vintage, and they have a lot in common. But on closer inspection they’re obviously a “special” for an important customer. Given the number of them knocking around, it’s obviously a customer with big data, centres stuffed full of servers with a lot of processing to do. Here’s a hint: It’s not Google or Amazon.

So, should you be buying a weirdo box with no documentation whatsoever? I’d say yes, definitely. If you’re interests are anything like mine. In a 2U box you can get twin 4-core CPUs and 64Gb of RAM for £150 or less. What’s not to like? Ah yes, the complete lack of documentation.

Over the next few weeks I intend to cover that. And to start off this is my first PC review for nearly twenty years.

So the Dell FS12-NV7:

FS-12 looking at the back panel. Note the cowling across the CPUs

As I mentioned, it’s a 2U full length heavy metal box on rails. On the back there are the usual I/O ports: a 9-way RS-232, VGA, two 1Gb Ethernet, two USB2 and a PS/2 keyboard and mouse. The front is taken up by twelve 3.5″ hard drive bays, with the status lights and power button on one of the mounting ears to make room. Unlike other Dell servers, all the connections are on the back, only.

If you want to play with the metalwork, the rear panel is modular and can easily be unscrewed although in practice there’s not much scope for enhancement without changing the  motherboard.

The FS12 has a single 1U PSU

Speaking of metalwork, it comes with  a single 1U PSU. There’s space above it for a second, but the back panel behind the PSU bay would need swapping – or removing – if you wanted to add a second. The area above the existing unit is just about the only space left in the box, and I have thought of piling up a load of 2.5″ drives there.

Taking the top off is where the fun starts. Inside there’s large Gigabyte EATX motherboard – a Gigabyte GA-3CESL-RH. All the ones I’ve seen are rev 1.7, which is a custom version but its similar to a rev 1.4. It does have, of all things, a floppy disk controller and an IDE (PATA) connector. More generally usefully, there are two more USB headers, a second RS-232 and six SATA sockets (3Gb). At the back there’s either a BMC module, or a socket where it used to be. If you like DRAC, knock yourself out (you’re likely to be barely concious to begin with). Seriously, this is old DRAC and probably only works with IE 2.0 or something. (You can probably tell I haven’t bothered to try it). The BIOS also allows you to redirect the console to the serial port for remote starting.

The Ethernet ports are Marvel 88E1116 1Gb, and haven’t given me any trouble. The firmware supports PXE, and I’m pleased to say that WoL works with the FreeBSD drives.

Somebody has pinched the slots!

Unfortunately, while the original Gigabyte model sported twin PCI and three PCIe sockets, the connectors are missing from these examples. It’s hard to find anything with a bit of grunt that can also use with your old but interesting PCI cards. It should be possible to rework it by adding the sockets and smoothing caps and sockets; fortunately the SMD decoupling caps are already still there.  On the other had, you could find another motherboard with PCI sockets if that’s what you really want.

But grunt is what this box is all about, and there’s plenty of that.

This is board was designed for Opteron Socket-F processors; specifically the 2000 series (Barcelona and Shanghi). The first digit refers to the number of physical CPUs that work together (either 2 or 8), the second is a code for the number of cores (1=1, 2=2, 3=4, 4=6, 5=8). The last two digits are a speed code. It’s not the frequency, it’s the benchmark speed.  I’ve heard rumours that some of FS-12s contain six-core CPUs, but I’ve only seen the 2373EE myself. The EE is the low power consumption version. Sweet.

If I could choose any Opeteron Socket-F CPU, the 2373EE is almost as good as it gets. It’s a tad slower than some of the other models running at 2.1GHz , but has significantly lower power and cooling requirements and was one of the last they produced in the 45nm process. It would be possible to change it for a 2.3GHz version, or one with six cores, but otherwise pretty much every other Opteron would be a downgrade. In other words, don’t think you can hot-rod it with a faster processor – you’re unlikely to find a Socket-F CPU anyway. After these, AMD switched to the Bulldozer line in an AM3+ socket.

This isn’t to say the CPU is modern. It does have the AMD virtualisation instructions, so it’s good news if you want to run nested 64-bit operating systems or hypervisors. The thing it lacks that I’d like most are the AES instructions that appeared in Bulldozer onwards. If you’re doing a lot of crypto, this matters. If you’re not, it doesn’t. Naturally, it implements the AMD64 instruction set, as now used by Intel, and all the media processing bit-twiddle stuff if you can use it. AMD has traditionally been at the forefront of processing smarter, whereas Intel goes for brute force and cranks up the clock speed. This is why AMD has, in my opinion, made assembler programming fun again.

Eight very capable Opteron cores: a good start. This generation supported DDR2 ECC RAM, and these boxes have 16 sockets (eight per CPU). They should be able to support 8Gb DIMMs, although I haven’t been able to verify this. Gigabyte’s documentation on similar motherboards is inconclusive as the earlier boards were from an time when 4Gb was all you could get. Again, I haven’t tried this but they are designed to handle 512Mb DIMMs. 1Gb and 4Gb certainly work and these tend to be available with any FS-12 you buy. At one time DDR2 ECC RAM was rather expensive. Not now. It’s much cheaper than DDR3 because, to be blunt, you can’t use it in very much these days.

And this is what makes the FS12 such a good buy: For about £150 you can get an eight-core processor with 64Gb of RAM. Bargain! And that’s before you look at the disk options.

The FS12, like most Dell Servers, is set up to run Windows and as a result requires a separate volume manager, on hardware designed to pretend Windows is looking at a disk. So-called “hardware” RAID. This takes the form of two PERC6/i cards occupying both PCIe cards on a riser. Fine if you want to run Windows or some other lightweight operating system, but PERC cards are about as naff as you can get for anything Unix-like. They work in RAID mode only, hiding the drives from the OS, and these are just a bit to old to be re-flashed in to anything useful.

The drives fit into a front-loading 12-way array with a SAS/SATA backplane. This is built in to the case; you can’t detach it and use it separately. Not without an angle grinder anyway, although if you really wanted to this would be a practical proposition. Note well that this is a backplane; not an expander, enclosure or anything so complex. Some Dell 2U servers like this do have an expander, which takes four SAS channels of SAS on a single cable and expands them to twelve, but this is the 1:1 version. And it’s an old one at that, using SFF-8484 connectors. If you’ve been using SAS for years you may still never have seen an SFF-8484 (AKA 32-pin Multi-lane). These didn’t last long and were quickly replaced with  the far more sensible SFF-8487(AKA 36-pin Mini-SAS). However, if you can sort out the cables (as I will explain in a later post), this backplane has possibilities.

But as it stands you get a the PERCs and a 12-slot drive array that’s only good for Windows or Linux. Unless, that is, you remove the backplane and the PERCs and make use of the six 3Gb SATA sockets on the motherboard. You’ll have to leave the drives in place and connect the cables directly back, but how many drives do you need?

There is one unfortunate feature of these boxes that is hard to ignore: the cooling. It’s effective, but when you turn it on it sounds like a jet engine spooling up. And then it gets even louder. There a lot you can do about this and I’m experimenting with options, which I’ll explain in a later post, but in the mean time you need to give everyone ear defenders, or install it in an outbuilding and use a KVM extender. I’ve been knocking around data centres for over twenty years and I’ve never heard one this bad.

The cooling is actually accomplished by five fans. Two are 1U size in the PSU, and are probably as annoying as any other ~40mm fan. The real screamers are two 80mm and one 60mm fan positioned between the drive cage and the motherboard. A cowling directs the one 80mm fan across each CPU and its DIMMs and the 60mm gives airflow over the Northbridge and PCI slots. They all spin really fast – in excess of 10,000rpm, and although they have sense and control wires nothing seems to be adjusting them downwards to the required rate.

My suspicion is that either the customer didn’t care about noise but wanted to keep everything as cool as possible, or that whatever operating system was installed (ESX I suspect) had a custom daemon to control their speed via the SAS backplane. I shall be going in to cooling options later, but note that the motherboard has five monitored and software adjustable fan connectors that are currently not used.

So, in summary, you’re getting a lot for your money if its the kind of thing you want. It’s ideal as a high-performance Unix box with plenty of drive bays (preferably running BSD and ZFS). In this configuration it really shifts. Major bang-per-buck. Another idea I’ve had is using it for a flight simulator. That’s a lot of RAM and processors for the money. If you forego the SAS controllers in the PCIe slots and dump in a decent graphics card and sound board, it’s hard to see what’s could be better (and you get jet engine sound effects without a speaker).

So who should buy one of these? BSD geeks is the obvious answer. With a bit of tweaking they’re a dream. It can build-absolutely-everything in 20-30 minutes. For storage you can put fast SAS drives in and it goes like the wind, even at 3Gb bandwidth per drive. I don’t know if it works with FreeNAS but I can’t see why not – I’m using mostly FreeBSD 11.1 and the generic kernel is fine. And if you want to run a load of weird operating systems (like Windows XP) in VM format, it seems to work very well with the Xen hypervisor and Dom0 under FreeBSD. Or CentOS if you prefer.

So I shall end this review in true PCW style:

Pros:

  • Cheap
  • Lots of CPUs,
  • Lots of RAM
  • Lots of HD slots
  • Great for BSD/ZFS or VMs

Cons:

  • Noisy
  • no AES-NI
  • SAS needs upgrading
  • Limited PCI slots

As I’ve mentioned, the noise and SAS are easy and relatively cheap to fix, and thanks to BitCoin miners, even the PCI slot problem can be sorted. I’ll talk about this in a later post.

 

Don’t blame Amazon, it’s Corporation Tax that’s broken

Well it looks like Amazon has only paid £1.3M UK tax, based on turnover of £Sqillions. Much wringing of hands and cries of “Something should be done!”. The same goes for Google, Starbucks or any other international company doing well in the UK. But nothing is being done to solve the problem, and for various reasons depending on your economic policy outlook.

First off, it’s not true to say Amazon pays very little tax in the UK. It pays VAT and PAYE. Lots of it. What it doesn’t pay much of is corporation tax, which is the tax on profits. And if you were an international company, you wouldn’t either. For international companies, corporation tax is, for practical purposes, optional. Companies may opt to pay as much or as little as suits their purpose.

If this is news to you, it works like this: Take Starbucks, for example. They managed to make very little profit in the UK. Because of this they were paying little or no corporation tax, which may seem odd when consider their ubiquitous presence in the high street. The reason was simple: Starbucks in the UK bought its coffee from its Dutch operation and the price was so high it wiped out the profits here. In Holland they were minting it, selling coffee to the UK, but the Dutch government took a liberal view on how much tax it should pay on these profits. Basically they were allowing Starbucks to pay a cut of what should have been UK corporation tax, and trouser the rest.

If Starbucks can do this simply by finding a foreign government prepared to sell out for a share of the profits, how easy is it for a Internet company with no physical product?

Basically, corporation tax would be a farce, were it not so serious. The problem is that it’s still paid in full by our local companies, putting them at an obvious disadvantage to foreign competition. It does more damage than good.

There are two solutions:

The left-wing idea is to make more new law against tax dodging. Somehow. And if international companies don’t like it, they can take their jobs, investment, VAT payments, PAYE payments and business rates and go somewhere else (e.g. Ireland). They’ll be gutted.

Back in the real world, if you have an unenforceable tax that damages local companies the smart thing to do is abandon it. But there is a problem with this – how do you make up the revenue you’re currently collecting from UK businesses (those that remain)? The obvious answer, and one the Conservatives won’t stomach, is to raise personal income tax. This isn’t actually a problem, because foreign companies will just have to cover it to keep take-home incomes stable (or lose staff) and local companies can afford to give everyone a pay rise out of the money that would have gone in corporation tax. Levelling the playing field won’t be painless in the short term, but this no reason to avoid it.

So Labour has a busted ideological plan and the Conservatives would be annihilated if they raised taxes. Something needs to break the deadlock, because newspapers naming and shaming global companies that are simply playing by the rules we gave them is no answer. Labour banging on about alleged “tax cuts for the rich” isn’t going to help. Neither will Conservative pledges not to raise any taxes. It’s not a question of raising or reducing taxes, it’s a question of balancing them properly.

Meanwhile the Irish government is laughing at us, all the way to the bank.

 

ZFS is not always the answer. Bring back gmirror!

The ZFS bandwaggon has momentum, but ZFS isn’t for everyone. UFS2 has a number of killer advantages in some applications.

ZFS is great if you want to store a very large number of normal files safely. It’s copy-on-write (COW) is a major advantage for backup, archiving and general data safety, and datasets allow you to fine-tune almost any way you can think of. However, in a few circumstances, UFS2 is better. In particular, large random-access files do badly with COW.

Unlike traditional systems, a block in a file isn’t overwritten in place, it always ends up at a different location. If a file started off contiguous it’ll pretty soon be fragmented to hell and performance will go off a cliff. Obvious victims will be databases and VM hard disk images. You can tune for these, but to get acceptable performance you need to throw money and resources to bring ZFS up to the same level. Basically you need huge RAM caches, possibly an SLOG, and never let your pool get more than 50% full. If you’re unlucky enough to end up at 80% full ZFS turns off speed optimisations to devote more RAM to caching as things are going to get very bad fragmentation-wise.

If these costs are a problem, stuck with UFS. And for redundancy, there is still good old GEOM Mirror (gmirror). Unfortunately the documentation of this now-poor relation has lagged a bit, and what once worked as standard, doesn’t. So here are some tips.

The most common use of gmirror (with me anyway) is a twin-drive host. Basically I don’t want things to fail when a hard disk dies, so I add a second redundant drive. Such hosts (often 1U servers) don’t have space for more than two drives anyway – and it pays to keep things simple.

Setting up a gmirror is really simple. You create one using the “gmirror label” command. There is no “gmirror create” command; it really is called “label”, and it writes the necessary metadata label so that mirror will recognise it (“gmirror destroy” is present and does exactly what you might expect).

So something like:

gmirror label gm0 ada1 ada2

will create a device called /dev/mirror/gm0 and it’ll contain ada1’s contents mirrored on to ada2 (once it’s copied it all in the background). Just use /dev/mirror/gm0 as any other GEOM (i.e. disk). Instead of calling it gm0 I could have called it gm1, system, data, flubnutz or anything else that made sense, but gm0 is a handy reminder that it’s the first geom mirror on the system and it’s shorter to type.

The eagle eyed might have noticed I used ada1 and ada2 above. You’ve booted off ada0, right? So what happens if you try mirroring yourself with “gmirror label gm0 ada0 ada1“? Well this used to work, but in my experience it doesn’t any more. And on a twin-drive system, this is exactly what you want to do. But it is still possible, read on…

How to set up a twin-drive host booting from a geom mirror

First off, before you do anything (even installing FreeBSD) you need to set up your disks. Since the IBM XT, hard disks have been partitioned using an MBR (Master Boot Record) at the start. This is really old, naff, clunky and Microsoft. Those in the know have been using the far superior GPT system for ages, and it’s pretty cross-platform now. However, it doesn’t play nice with gmirror, so we’re going to use MBR instead. Trust me on this.

For the curious, know that GPT keeps a copy of the partition table at the beginning and end of the disk, but MBR only has one, stored at the front. gmirror keeps its metadata at the end of the disk, well away from the MBR but unfortunately in exactly the same spot as the spare GPT. You can hack the gmirror code so it doesn’t do this, or frig around with mirroring geoms rather than whole disks and somehow get it to boot, but my advice is to stick to MBR partitioning or BSDlabels, which is an extension. There’s not a lot of point in ever mounting your BSD boot drive on a non-BSD system, so you’re not losing much whatever you choose.

Speaking of metadata, both GPT and gmirror can get confused if they find any old tables or labels on a “new” disk. GPT will find old backup partition tables and try to restore them for you, and gmirror will recognise old drives as containing precious data and dig its heels in when you try to overwrite it. Both gpart and gmirror have commands to erase their metadata, but I prefer to use dd to overwrite the whole disk with zeros anyway before re-use. This checks that the disk is actually good, which is nice to know up-front. You could just erase the start and end if you were in a hurry and wanted to calculate the offsets.

The next thing you’ll need to do is load the geom_mirror kernel module. Either recompile the kernel with it added, or if this fills you with horror,  just add ‘load_geom_mirror=”yes”‘ to /boot/loader.conf. This does bring it in early enough in the process to let you boot from it. The loader will boot from one drive or the other and then switch to mirror mode when it’s done.

So, at this point, you’ve set up FreeBSD as you like on one drive (ada0), selecting BSDlabels or MBR as the partition method and UFS as the file system. You’ve set it to load the geom_mirror module in loader.conf.  You’re now looking at a root prompt on the console, and I’m assuming your drives are ada0 and ada1, and you want to call your mirror gm0.

Try this:

gmirror label gm0 ada0

Did it work? Well it used to once, but now you’ll probably get an error message saying it could not write metadata to ada0. If (when) this happens I know of one answer, which I found after trying everything else. Don’t be tempted to try everything else yourself (such as seeing if it works with ada1). Anything you do will either fail if you’re lucky, or make things worse. So just reboot, and select single-user mode from the loader menu.

Once you’re at the prompt, type the command again, and this time it should say that gm0 is created. My advice is to now reboot rather than getting clever.

When you do reboot it will fail to mount the root partition and stop, asking for help to find it. Don’t panic. We know where it’s gone. Mount it with “ufs:/dev/mirror/gm0s1a” or whatever slice you had it on if you’ve tried to be clever. Forgot to make a note? Don’t worry, somewhere on the boot long visible on the screen it actually tell you the name of the partition it couldn’t find.

After this you should be “in”. And to avoid this inconvenience next time you boot you’ll need to tweak /etc/fstab using an editor of your choice, although real computer nerds only use vi. What you need to do is replace all references to the actual drive with the gm0 version. Therefore /dev/ada0s1a should be edited to read /dev/mirror/gm0s1a. On a current default install, which no longer partitions the drive, this will only apply the root mount point and the swap file.

Save this, reboot (to test) and you should be looking good. Now all that remains is to add the second drive (ada1 in the example) with the line:

gmirror insert gm0 ada1

You can see the effect by running:

gmirror status

Unless your drive is very small, gm0 will be DEGRADED and it will say something about being rebuilt. The precise wording has changed over time. Rebuilding takes hours, not seconds so leave it. Did I mention it’s a good idea to do this when the system isn’t busy?