Posted on

The Crazy Politics of Age Verification

The UK government required age verification for pornographic website to protect children. Australia was the first country in the world to ban children from social media. Children, of course, circumvented all such restrictions immediately. And now the UK government and others around the world are thinking “This plays well with the public” and is trying to follow suit, in spite of the fact the teenagers are laughing at the restrictions.

Rather than getting an informed opinion on the practicality of such measures from technology experts, the politicians, as usual, get their technical understanding from some random teenagers on social media.

This lack of tech savvy would be hilarious if it wasn’t such a serious matter. It’s nothing new. And these proposed technological solutions to problems in society are doing nothing to fix the very real issues they’re trying to address. You cannot fix social problems using technology. Leave a comment if you can think of any example where this has been the case.

Next Monday there’s a vote in the commons on just this, and I fear grandstanding arts-graduate politicians will let it through, regardless of the technical impossibility of complying. And there’s no end to this nonsense.

There’s crazy, then there’s California

And just when you think it can’t get any worse, the Californians have taken it to the next level with a new bill called Age Verification signals: software applications and online services.

Here’s an extract:

1798.501. (a) An operating system provider shall do all of the following:

(1) Provide an accessible interface at account setup that requires an account holder to indicate the birth date, age, or both, of the user of that device for the purpose of providing a signal regarding the user’s age bracket to applications available in a covered application store.

(2) Provide a developer who has requested a signal with respect to a particular user with a digital signal via a reasonably consistent real-time application programming interface that identifies, at a minimum, which of the following categories pertains to the user:

(A) Under 13 years of age.
(B) At least 13 years of age and under 16 years of age.
(C) At least 16 years of age and under 18 years of age.
(D) At least 18 years of age.

Having an handy OS function to determine the age of the user seems like a good idea, doesn’t it? Naughty software can refuse to run if the user is under-age. But it’s technical nonsense; not thought through at all. For example, it’s referring to a “real-time application programming interface” – so this only applies to a RTOS? Obviously they saw the phrase, thought it sounded good and included it.

But the concept has some merit – rather than everything doing age verification, why not dump the problem on Microsoft, Apple and Google? They’ve got the resources, and with their penchant for knowing everything about their customers, their OS could easily provide age verification information along with their shoe size and taste in pot plants.

Except… you need to consider the chilling effect this will have on anything called an Operating System. The lawmakers probably wouldn’t recognise an OS if it jumped out at them waving a flag, but this is important here. For starters, every embedded system has software that fits the definition of Operating System. Do they have to provide age verification?

And what about, say, servers? They don’t have a user as such, but they have people using them (administrators). How can you tell the age of the administrator at any given time? The account may well be shared by a number of people.

But the most damage from this nonsense will be to FOSS operating systems like Linux. They don’t have a KYC culture like Microsoft/Apple/Google to make it easy and don’t have the time, money or resources to solve the problem. An open source operating system is community owned, and the community has no incentive to track the personal details of who might be using it.

Although Linux is free, it’s still licensed using the GPL and has a vendor – even if no money changes hands. If this bill ever passes, the State of California may well go after the vendor for non-compliance and try to force them into it by fines and suchlike.

The logical solution to the problem for Linux distributions (and FreeBSD and whatever) is to change the terms of their license saying “You may not download or use this software in the State of California”. Microsoft, Apple and Google will love that.

Posted on

Does the new AWS Euro Cloud solve the Data Sovereignty problem?

No, obviously not. It’s just fine words, along with every other promise made by cloud computing vendors. I’m using AWS as an exemplar, as they’ve just hyped up the launch of their AWS European Sovereign Cloud.

According to AWS it’s “a new, independent cloud for Europe entirely located within the EU” and has been welcomed by their customers and policymakers.

Basically it’s rolling out new “sovereign” AWS Local Zones in which the data will always remain in the country of origin. That’s data residency, not sovereignty, but it’s a start – if Uncle Sam wanted access to this data they’d have to pressure AWS to hand it over. However, given the US Administration’s willingness to pressure Europe into making Demark hand over Greenland, such pressure isn’t unimaginable.

Stéphane Israël, managing director of the AWS European Sovereign Cloud and digital sovereignty, stated “Customers want the best of both worlds – they want to be able to use AWS’s full portfolio of cloud and AI services while ensuring they can meet their stringent sovereignty requirements. By building a cloud that is European in its infrastructure, operations, and governance, we’re empowering organisations to innovate with confidence while maintaining complete control over their digital assets.”

This is interesting. Governments do impose sovereignty requirements, but that doesn’t mean they get them. It just means a service provider has promised to keep their data within international borders while it’s convenient for them to do so. However, if the company is ultimately based abroad, the government there will be able to pressure it to hand the data over anyway, and the small print in the contract will allow them to comply to all applicable laws (present and future).

I don’t want to make the Americans out to be the bad guys here. For example, late last year a Canadian court ordered French bit barn OVHcloud to hand over customer data as part of a criminal investigation. This could have been handled by a mutual assistance treaty, but the Canadian police decided to go after OVH’s Canadian subsidiary. French law prohibits OVH from complying or they go to jail, but the Canadian’s will take it out on their subsidiary there if they don’t. Place your bets on the outcome.

And this is mild mannered Canada. Imagine what the Chinese or Americans might do, and for less reason? Companies in China are very much under the control of Communist Party, and the US CLOUD Act (Clarifying Lawful Overseas Use of Data Act, 2018) allows US officials to subpoena data from US companies stored on foreign servers with minimal court protections.

Now you might say you don’t really care if the CIA wants to read the innocuous chatter between Wizards on your MUD, but if it’s personal information protected by local law then you might find yourself in trouble whether the option to access it is taken or not.

So what can you do?

You can eschew multinational hyperscalers and use a local cloud provider, of course. Your local government can probably still access the data unless they’re too small to show up on radar, but this may not matter to you or your customers. The risk here, in my experience, is that these companies don’t last forever and may well be subsumed by a multinational. And to be honest, quality has proved even more variable than the hyperscalers. You won’t get the PaaS infrastructure tools and services provided by the likes of AWS, Azure GCP. However, going for the self-managed VPS option doesn’t tie you to a particular provider and give you an escape route if their sovereignty becomes compromised at a later date.

The problem remains – your VPS provider can snoop on you. One particular provider, who shall remain nameless, contacted me to warn I was running a compromised version of RHEL and I really needed to patch it or they’d take my server down. What? It turned out I had a CentOS repository (running on FreeBSD), which was needed to support testing and migrating a legacy project and their scanner had picked up an old (but obviously not running) version of log4j2 amongst the packages. The point is they were scanning the disks, albeit with the best of intentions. And if you think running whole disk encryption on a VM would stop that, think again.

Last week a friend asked what I thought of a particular UK cloud provider as a means to overcome sovereignty problems perceived by his commercial partners, which was the second thing that has prompted me to write this. He followed with “I expect you’ll say I should run my own servers in a data centre”, and he be right in many cases. If you can run your own VPS, you ought to be able to run your own servers, and they’ll be a lot cheaper. If you’re not sure about looking after the hardware, put a consultant on a retainer. And if you think hard disks have become a bit expensive lately, wait until you see what the could providers charge per gigabyte. Having your own servers also means you do have total control of your data. You know where it is at all times, and can whole disk encrypt it if necessary to protect against physical theft. With cheap asymmetric fibre lines you can even keep backups and archives in your company safe, or under a mattress as you prefer.

Posted on

ssh login fun on Ubuntu VM

Won’t use ed25519

I’ve written about setting up ssh logins many times in the past. It’s something I do all the time. But I came across an interesting situation trying to ssh between two adjacent Ubuntu VMs. Ubuntu 22.04.1 LTS, to be exact. Not exactly the latest release, which makes ssh certificate problems more exciting.

So use ssh-keygen -t ed25519 to create a key pair. The snappily named ed25519 is everyone’s favourite cypher right now, and going between two identical Ubuntu installations compatibility is assured, right? Wrong!

I copied the public key into ~/.ssh/authorized_keys using the PuTTY method (cut/paste between windows) and tried an ssh login. It still asked me for my password.

I set the security on the file, although authorized_keys can just as well be world readable – it contains public keys after all. Still no dice.

Here’s a tip – if ssh isn’t playing nice run it in debug mode. Unlike other software, it’s really simple – just use the -v option. Use it multiple times (-vvv) to get a reasonable level of debug output.

$ ssh -vvv user@otherhost
<snip>
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/user1/.ssh/id_rsa RSA SHA256:oOgKcEHVqRgQqZXh5E2++iJUnacbXHDzLsnSHsngNpw
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: /home/user1/.ssh/id_dsa DSA SHA256:7jTWaHnN1cjNAnqD4bOZL9E/3nYMbioPgSimRsgAwuk
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /home/user1/.ssh/id_ecdsa_sk
debug3: no such identity: /home/user1/.ssh/id_ecdsa_sk: No such file or directory
debug1: Trying private key: /home/user1/.ssh/id_ed25519_sk
debug3: no such identity: /home/user1/.ssh/id_ed25519_sk: No such file or directory
debug1: Trying private key: /home/user1/.ssh/id_xmss
debug3: no such identity: /home/user1/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
user@otherhost's password: ???

What’s going on here then? It’s not even trying the key (id_ed25519) – it’s trying one with the _sk suffix instead. Apparently it’ll use a XMSS certificate, nice! Authentication that works after the quantum apocalypse. Unfortunately ssh-keygen didn’t know about it, so no dice. Okay, if ed25519_sk is preferred to plain old ed25519, let’s make one:

$ ssh-keygen -t ed25519_sk
unknown key type ed25519_sk

Hmm. User-hostile software. With a bit of thought it’d list the ones it does know about when this happens, but you can trick it into telling you by typing “ssh-keygen -t” (or some other command syntax error). This will output a somewhat helpful summary that will include valid options for -t type.

$ ssh-keygen -t
option requires an argument -- t
usage: ssh-keygen [-q] [-a rounds] [-b bits] [-C comment] [-f output_keyfile]
                  [-m format] [-N new_passphrase] [-O option]
                  [-t dsa | ecdsa | ecdsa-sk | ed25519 | ed25519-sk | rsa]
                  [-w provider] [-Z cipher]

It turns out it uses a hyphen instead of an underscore, so try it again.

$ ssh-keygen -t ed25519-sk
Generating public/private ed25519-sk key pair.
You may need to touch your authenticator to authorize key generation.
Key enrollment failed: device not found

Eh? Device not found? What is a -sk key anyway? It sounds like a hardware key of some kind going by the error message, which is clearly not appropriate for a VM. Googling didn’t help, so I asked another member of the OS/2 Drinking club that evening and he thought it was indeed a FIDO-related, or at least some kind of discoverable key requiring a hardware authenticator such as YubiKey.
Meanwhile what to do? Trying to edit /etc/ssh/ssh_config to make it offer a standard ed25519 key defeated me. I have a suspicion that Ubuntu drops them from the list if a FIDO hardware is available, and it clearly thought it was. Adding explicitly for all hosts didn’t work:

Host *
        IdentitiesOnly yes
        IdentityFile ~/.ssh/id_rsa
        IdentityFile ~/.ssh/id_dsa
        IdentityFile ~/.ssh/id_ecdsa
        IdentityFile ~/.ssh/id_ecdsa_sk
        IdentityFile ~/.ssh/id_ed25519
        IdentityFile ~/.ssh/id_ed25519_sk

It may be possible to force it using a user’s config file, but using -i to force an identity failed – it just reverted to the keys it wanted to use.

I gave up fighting with it and switched to old trusty RSA. On some systems (like BSD) you’ll need to add it to /etc/ssh/sshd_config as it’s too 1970’s for some people to abide. But Ubuntu was happy with it out of the box, and it’s still the default keytype for ssh-keygen.

PubkeyAcceptedAlgorithms +ssh-rsa
HostKeyAlgorithms +ssh-rsa

There’s a principle on cryptography that a code is either broken or unbroken, and no one has broken RSA yet in spite of theories that it might be possible (and anyway, it’s too old, grandad!) And in this case, Ubuntu seems to have shot itself in the foot forcing you to use RSA anyway. Or DSA, which is actually broken. Which seems appropriate in the circumstances.

Posted on

Blocking script kiddies with PF

OpenBSD’s PF firewall is brilliant. Not only is it easy to configure with a straightforward syntax, but it’s easy to control on-the-fly.

Supposing we had a script that scanned through log files and picked up the IP address of someone trying random passwords to log in. It’s easy enough to write one. Or we noticed someone trying it while logged in. How can we block them quickly and easily without changing /etc/pf.conf? The answer is a pf table.

You will need to edit pf.conf to declare the table, thus:

# Table to hold abusive IPs
table <abuse> persist

“abuse” is the name of the table, and the <> are important! persist tells pf you want to keep the table even if it’s empty. It DOES NOT persist the table through reboots, or even restarts of the pf service. You can dump and reload the table if you want to, but you probably don’t in this use case.

Next you need to add a line to pf.conf to blacklist anything in this table:

# Block traffic from any IP in the abuse table
block in quick from <abuse> to any

Make sure you add this in the appropriate place in the file (near or at the end).

And that’s it.

To add an IP address (example 1.2.3.4) to the abuse table you need the following:

pfctl -t abuse -T add 1.2.3.4

To list the table use:

pfctl -t abuse -T show

To delete entries or the whole table use one of the following (flush deletes all):

pfctl -t abuse -T delete 1.2.3.4
pfctl -t abuse -T flush

Now I prefer to use a clean interface, and on all systems I implement a “blackhole” command, that takes any number of miscreant IP addresses and blocks them using whatever firewall is available. It’s designed to be used by other scripts as well as on the command line, and allows for a whitelist so you don’t accidentally block yourself! It also logs additions.

#!/bin/sh

/sbin/pfctl -sTables | /usr/bin/grep '^abuse$' >/dev/null || { echo "pf.conf must define an abuse table" >&2 ; exit 1 ; }

whitelistip="44.0 88.12 66.6" # Class B networks that shouldn't be blacklisted

for nasty in "$@"
do
        echo "$nasty" | /usr/bin/grep -E '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$' >/dev/null || { echo "$nasty is not valid IPv4 address" >&2 ; continue ; }

        classb=$(echo "$nasty" | cut -d . -f 1-2)

        case " $whitelistip " in
                *" $classb "*)
                echo "Whitelisted Class B $nasty"
                continue
                ;;
        esac

        if /sbin/pfctl -t abuse -T add "$nasty"
        then
                echo Added new entry $nasty
                echo "$(date "+%b %e %H:%M:%S") Added $nasty" >>/var/log/blackhole
        fi
done

That’s all there is two it. Obviously my made-up whitelist should be set to something relevant to you.

So how do you feed this blackhole script automatically? It’s up to you, but here are a few examples:

/usr/bin/grep "checkpass failed" /var/log/maillog | /usr/bin/cut -d [ -f3 | /usr/bin/cut -f1 -d ] | /usr/bin/sort -u

This goes through mail log and produces a list of IP addresses where people have used the wrong password to sendmail

/usr/bin/grep "auth failed" /var/log/maillog | /usr/bin/cut -d , -f 4 | /usr/bin/cut -c 6- | /usr/bin/sort -u

The above does the same for dovecot. Beware, these are brutal! In reality I have an additional grep in the chain that detects invalid usernames, as most of the script kiddies are guessing at these and are sure to hit on an invalid one quickly.

Both of these examples produce a list of IP addresses, one per line. You can pipe this output using xargs like this.

findbadlogins | xargs -r blackhole

The -r simply deals with the case where there’s no output, and will therefore not run blackhole – a slight efficiency saving.

If you don’t have pf, the following also works (replace the /sbin/pfctl in the script with it):

/sbin/route -q add $nasty 127.0.0.1 -blackhole 2>/dev/null

This adds the nasty IP address to the routing table and directs packets from it to somewhere the sun don’t shine. pf is probably more efficient that the routing table, but only if you’re using it. This is a quick and dirty way of blocking a single address out-of-the-box.

Posted on

Bad software and security

Today, LinkedIn decided I might want to see this post from Thomas Barnett:

“Most businesses don’t get hacked because of bad software.
They get hacked because someone clicked.”

(Apologies if the link doesn’t work – it’s a Microsoft site).

He is, of course, quite correct in that phishing and trojans are the most exploitable vulnerability in most organisations, but it hinges on the term “bad software”. If you’re new to the world of computing you won’t remember a time when this wasn’t a problem, but it has become one largely thanks to Microsoft thinking of profits before security, with features to “create a richer user experience”. I’d classify this as “bad software”, and it very much is the cause.

In the early days of the Internet there was assumed security, as any miscreants could be apprehended by the system operator checking which terminal they were on and paying them a visit. Unencrypted data flew back and forth on the network without it being a huge risk, as access to the wires was controlled by bricks and mortar. It took a while to add encryption when the Internet went public, but that’s done. Logins require security certificates. SMTP relays are closed. It should all be good.

Then some fools decided it would be “cool” to embed software in network traffic.

“Let’s allow people to send executable files by email that look like any other file and can be opened by clicking on them.” Bad software.

“Let’s embed JavaScript in web pages so we can run stuff on the user’s machine.” Bad software.

“Let’s embed software in Microsoft Office documents.” Bad Software.

“Let’s use passwords for accessing important data instead of security certificates tied to a host.” Bad Software.

There are other forms of idiocy around, such as downloading software from a package repo, placed there by anyone on the Internet, simply because there are so few actual software engineers around who can configure a server without a Docker image. But using passwords to log into remote systems, encrypted or otherwise, where the user has no way of knowing whether it’s the real login page is nuts. So is embedding software in electronic communications.

A cybersecurity industry has built up trying to place mitigations on this bad software. People like me have been warning about the fundamental dangers of Microsoft’s “Rich user experience” mantra for decades. I remember writing a piece for PCW in November 1996 when Netscape (not Microsoft, for once) proposed adding JavaScript to web browsers. (Previously Java Applets* were theoretically sandboxed).

Before this, when Microsoft added WordBasic in Word for Windows and DOS, people like me who’d been on the forefront of antimalware in the 1980s, were scarcastingly asking “What could possibly go wrong?”

So Mr Barnett is right to say these things are the most effective attack vector. Organisations should be very afraid. But they’re only attack vectors because the software is bad.

*JavaScript and Java share the name but are nothing like the same thing.

Posted on

The real reason AWS failed

Amazon is about to shed 14,000 jobs to “stay nimble”. Would it be rude to tell the Senior Vice President of People Experience and Technology at Amazon that this bird has already flown? Engineers trying to use AWS will tell you that they days of Amazon’s amazing technical prowess are well behind them. And it’s across the board – from their smart devices (has anyone used the new Alexa App) to AWS services that only half work. Only their retail system remains best-in-class.

Amazon blamed the recent DNS outage that took so many of their customers offline a week ago on a “race condition” where the DynamoDB back-end to their unified DNS solution simply failed. They explained it in great detail here:

https://aws.amazon.com/message/101925/

What they didn’t say why it failed; why the race condition existed. Why they screwed up. I’ll hazard a guess that they made the people who really knew how their systems worked redundant by mistake, replacing them with new and cheaper hires to fill the hole in a spreadsheet. There was no one left to spot the flaw until it became manifest. Engineers are interchangable, right? And you just need the right number to fit the project plan.

You don’t need large teams of qualified people to make this stuff work. You need small teams of experienced people who stick with the job and are treated enough respect that they’re empowered to do it. The good ones are not going to stick around to play redundancy roulette every six months, hoping that HR actually understand they’re necessary to keep the show on the road. HR take pride in saying they’re “people people”. Good engineers are not people people; they’re more likely to be neurodiverse. Their managers are unlikely to understand what they do, and HR certainly won’t.

I dare say there is a lot of dead wood in Amazon. They were recruiting like crazy during the pandemic; anyone who looked vaguely like an engineer with a pulse. The trick is identifying who to keep; if, indeed, there is anyone left who they haven’t made redundant already, or who simply got too spooked and left.

Posted on

Malware turns your mouse into spy microphone

Researchers at the University of California in Irvine have discovered that high resolution gaming mice can act as a microphone with a high enough sample rate, using special software installed on the PC. Vibrations in the air, transferred to the mouse mat can be picked up by the sensors, filtered and produce recognisable speech in the right circumstances.

Users of normal mice have little to worry about, but as they point out, anti malware vendors don’t currently treat mouse input as an attack vector.

Read all about it here:

Invisible Ears at Your Fingertips: Acoustic Eavesdropping via Mouse Sensors

https://arxiv.org/html/2509.13581v1

Unix/BSD users are not in danger, as the attack won’t work with a keyboard.

Posted on

Airports “hacked” by ransomware gang

I’m looking at media reporting of the disruption caused to airports by the latest ransomware attack and I’m once again struct by the lack of detail. The victims are, as always, tight-lipped about it and this translates to the media as “we don’t know what happened apart from it was an attack”.

Anyone who knows how this stuff works will have a pretty good idea what went down. So let’s look at the Collins Aerospace system at the heart of it: It’s reported as being MUSE but it’s actually cMUSE

cMUSE stands for common-use Multi-User System Environment, and it allows airlines to share check-in desks. It’s what’s known as a common-use passenger processing system, or CUPPS. When the self-loading cargo presents itself a the check-in it tracks their bags using integration with systems like BagLink, sorts out boarding stuff and so on. It’s main competitor, if you look at it that way, is SITA’s BagManager, but this only handles and tracks luggage.

Now here’s the thing – cMUSE makes a big thing of being cloud based. It runs on AWS. A SaaS product. It is possible to run it on your own infrastructure, but they sell the benefits of not needing your own servers and expensive IT people to manage it – just let them do it for everyone on AWS.

So what went wrong? They haven’t said, but a penny to a pound it’s the AWS version that got hit. This is why so many airlines got their check-in hijacked in one go. A nice juicy target for the ransomware gangs.
At Heathrow, I believe it’s deployed on over 1,500 terminals on behalf of more than 80 airlines. It’s used in over 100 airports worldwide, which isn’t a huge share of the total number (there are over 2000 big ones according to the ACI), but it’s been sold extensively to the big european ones – high-traffic multi-carrier hubs. The ones that matter. Heathrow renewed for another six-year contract this April.

Collins claims it will save $100K per airport going to AWS, but that must seem like a false economy right now. Its predecessor, vMUSE, dates before cloud-mania and users of the legacy system must be feeling quite smug. Many airports have a hybrid of cMUSE and vMUSE and it’s hard to know the mix.

Ottawa International went cloud with a fanfare in 2017, and Shannon Airport chugged down the kool-aid, renewing for cloud-only in 2025. Heathrow is likely mostly cloud. Cincinnati/Northern Kentucky, Indira Gandhi International (Delhi) are publicly know to be cloud users. What bet Brussel and Berlin Brandenburg are on the list? Lesser problems at Dublin and Cork, which use the system, suggest they’re hybrid or still on vMUSE.

Subscribing to a cloud service for anything important is such a bad idea. You’re only as safe as your cloud provider. There’s no such thing as a virtual air-gap and large-scale attacks are only possible because everyone’s using the same service. If airports save $100K by switching, they’d be much better off having servers on-site and paying someone to look after them – part-time if it’s such a small amount in question.

If you want a games server in the cloud go ahead. If my business depended on it, I’d want to know where my data was and who could get at it.

Posted on

FCA and Government want to scrap 2FA on contactless payments

You couldn’t make this up. The government has ordered the Financial Conduct Authority to come up with bright ideas to stimulate the economy (after other government policy has sent it in the opposite direction). Their latest wheeze, published today, is to make contactless card payments unlimited in value. I kid you not.
https://www.fca.org.uk/news/press-releases/proposed-contactless-changes-could-increase-convenience-consumers

This means that if someone steals a contactless card they can spend as much as they like from your bank account. But don’t worry, the FCA says the banks will have to refund you if it happens.

Part of their justification is that digital wallets (Apple Pay and Google Pay) allow for much higher contactless transactions than the current £100 limit. For anything over £100 and the card system asks for a PIN to prove it’s really you. Even with that safeguard, criminals make a series of transactions of around £90 before the banks fraud system detects something suspicious.

You might remember that the contactless limit was £30 from 2015 to pandemic, after which it was raised to £45 and then £100 in 2021 to reduce the amount of contaminated cash in circulation. It was never reduced, which some say was a mistake.

The difference between physical cards and Apple Pay/Google Wallet is that they require you to unlock the phone first, which is arguably more secure than a four-digit PIN. Claiming that because these are unlimited that the PIN security should be stripped from physical cards is the craziest thing I’ve heard in years. And the FCA is going out of its way to blame the government.

Posted on

LetsEncrypt, acme.sh and Apachectl reloads

This morning I woke up to an expired TLS certificate on this blog. This is odd, as it’s automatically renewed from LetsEncrypt using acme.sh, kicked off by a cron job. So what went wrong?

I don’t write about LetsEncrypt or ACME much as I don’t understand everything about it, and it keeps surprising me. But I had discovered a problem with FreeBSD running the latest Apache 2.4 in a jail. As I run my web servers in jails, this applies to me.

I like acme.sh. It’s a shell script. Very clever. No dependencies. Dependencies are against my religion. Why anyone would use a more complex system when there’s something simple that works?

For convenience reasons the certificates are renewed outside of a jail, and the sites are created using a script that sets it all up for me. One source of certificates for multiple jails; it’s easier to manage. It manages sites on other hosts using a simple NFS mount.

When you use acme.sh to renew a certificate for Apache you need to be able to plonk something on the web site. This is easy enough – the certificate host (above the jails) can either get direct access through the filing system, or via NFS. It then gets the new certificate and copies it into the right place. When you first issue yourself a certificate you specify the path you want the certificate to go, and the path to the web site. You also specify the command needed to get your web server to reload. It magically remembers this stuff so the cron job just goes along and does them all. But that’s where the fun starts.

I rehosted the blog on a new instance of Apache, and created a new temporary website to make sure SSL worked – getting acme.sh to issue it a certificate in the process. All good, except I noticed that inside a jail, the new version of Apache stops but doesn’t restart after an “apachectl graceful”. The same with “apachectl reload”. Not great, but I tried using “service -j whatever apache24 restart”. A bit drastic but it worked, and I’ve yet to figure out why other methods like “jexec whatever apachectl graceful” stall.

So what happened this morning at 6am? There were some certificates to renew and acme.sh –cron accidentally KOed Apache. It’s the first time any had expired.

Running acme.sh manually between restarting Apache manually worked, but it’s hardly the dream of automation promised by Unix. Debugging the script I found it was issuing a graceful restart command, and I thought I’d specified something more emphatic. So I started grepping for the line in was using, assuming it must be in a config file somewhere. Nothing.

Long story short, I eventually found where it had hidden the command: in .acme.sh/domain.name/domain.name.conf , in spite of having looked there already. It turns out that it’s the line “Le_ReloadCmd=”, and its unique for each domain (sensible idea), but it’s base64 encoded instead of being plain text! And it’s wrapped between “_ACME_BASE64__START_” and “_ACME_BASE64__END_”. I assume this is done to avoid difficulties with certain characters in shell scripts but it’s a bit of a pain to edit it. You can create a new command by piping it through base64 and editing very carefully, but readable it ain’t.

There is an another way – just recopy the certificate. Unfortunately you need to know, and use, the same options as when you originally created it – you can’t just issue a different –reloadcmd. You can check these by looking at the domain.name.conf file, where fortunately these are stored in plain text. Assuming they’re all the same, this little script will do them all for you at once. Adjust as required.

#!/bin/sh

# Make sure you're in the right directory
cd ~/.acme.sh

# Jail containing web site, assumed all the same.
WJAIL=web

for DOM in $(find . -type d -depth 1 | sed "s|^\./||")
do

echo acme.sh $TEST -d $DOM  --install-cert \
        --cert-file /jail/$WJAIL/data/certs/$DOM/cert.pem \
        --key-file /jail/$WJAIL/data/certs/$DOM/cert.key \
        --fullchain-file /jail/$WJAIL/data/certs/$DOM/Fullchain.pem \
        --reloadcmd "service -j $WJAIL apache24 restart"
done

You will notice that this only echos the command needed, so if anyone’s crazy enough to copy/paste it then it won’t do any damage. Remove the “echo” when you’re satisfied it’s doing the right thing for you.

Or you could just edit all the conf files and replace the Le_ReloadCmd= line – you only have to generate it once, after all.