Frank Leonhardt's Blog – Life, with Technology

Simple guide to the vi editor

vi is the standard text editor that’s been around on Unix and Unix-like systems since 1976. It was written by Bill Joy in one weekend as an enhancement of the original ex editor, which lacked “full screen” mode because, to put it bluntly, full screen terminals hadn’t been invented.

It’s part of the POSIX standard and you’ll never find a Unix that hasn’t got it. Love it or hate it, vi is standard; so you’d better make friends with it because you’re going to need it, even if it’s only briefly while you compile something more modern as a replacement.

There are numerous tutorials and “cheat sheets” for vi, but they’re over-complicated to my mind. Yes, you can do a lot with ex and vi if you remember and type various key sequences correctly, but most people just want to edit a text file, quickly. There are much better editors out there for big jobs, although they weren’t written in a weekend.

So how do you use it, if you don’t want to remember more complex keystrokes than necessary?

The modes

The first thing you need to understand about vi is that it has “modes”. With any normal editor, you move the cursor to wherever you want to type something, and type away. Not so with vi, but it’s not that bad when you understand it. When you first start vi it’s in Command Mode, which would be better described as “move the cursor mode”. It’s waiting for you to move the cursor somewhere. You can enter a command by prefixing it with a colon (“:”), when it will jump the cursor to the bottom of the screen so you can enter the command. Other modes let you enter text or search for things. To return to the Command Mode (i.e. move the cursor mode) you generally press the [Esc] key. Remember that if you’re stuck.

In Command Mode most of the keys do things, usually moving the cursor, often in expected ways. Be careful what you press.

In this text I’m representing the Enter key on the keyboard with [Enter], and Escape with [Esc]. Other characters in double quotes to make them stand out, but are typed without the double quotes. So let’s get started…

Loading a file to edit

If you’re new to vi you might want to make a backup of the file you’re about to mess with. There are mechanisms to have vi do this for you, but for safety just make a copy yourself:

cp filename filename.safe

Then on to edit the file. Type “vi filename”

This will load the file named filename into the editor. To make the editor a little more friendly you can type:

:set verbose showmode

Followed by Enter.

Quitting or saving a file.

This uses a colon command (see above if you skipped the introduction). To quit type :q followed by the Enter key. If you’ve made changes to the file it will say “File modified since last complete write” or similar, and won’t let you. You can override this by adding a ! to the command to show you really mean it

:q! [Enter]

To write changes to disk use :w <enter> This is the equivalent to “Save” on a modern editor. To write the current file to another file (i.e. Save As…) use :w newfilename [Enter]

A quick shortcut if you want to save and exit the editor in one go is :x [Enter]

Moving around the file

When the file is loaded you’re in Command Mode, which might more usefully be thought of as Movement Mode. You can’t type anything but you can move the cursor around. On anything reasonably modern you can do this using the arrow keys on the keyboard and [PgUp] and [PgDn]. [Home] and [End] probably work too. There are other ways of doing this using original keyboards that lacked cursor keys, which I’ll cover later if needed.

Changing stuff

Let’s assume you’ve got the cursor to the place in the file where you want to make the change. Most of the time you’ll want to either delete stuff or insert stuff. To delete what’s under the cursor type the “x” key. If you want to delete a lot, “D” deletes everything to the end of the line and “dd” deletes the whole line, but if you can’t remember just stick to “x”. Note that it’s case sensitive – “D” and “d” are not the same.

To insert something, type the “i” key to get into Insert Mode. Other modes are available, but Insert Mode is what most people are used to.

When you’re in Insert Mode, everything you type will be inserted, including new lines if you press the return key. To get out of insert mode press the [Esc] key. And this is a general rule, if you’re in a mess with vi keep hitting the [Esc] key until you get back into Command Mode.

You might, of course, have made a mess of the edit. A single “u” will undo the last change you made in Insert Mode. If you press “u” a second time it will toggle the undone changes back.

If you make a real mess of it just quit out using “:q!” and start again. It sometimes pays to do a “:w” while editing to save good changes.

Cut+paste with a mouse

vi has all sorts of ways of moving text around, but to keep it simple I’m going to assume you’re using a virtual terminal (something like PuTTY) and have mouse. Just select the text you want to copy, move to where you want it to go and put vi in insert mode with “i”. If you can’t remember which mode you’re in type [Esc] “i” to be sure. Then right-click the mouse and it will paste.

Search and replace

To search for something in the file hit [Esc] to make sure you’re in Command Mode and type “/” followed by whatever you’re looking for, followed by Enter. If you want to search backwards use “?” instead of “/”. The cursor will jump to the first occurrence it finds.

If you want to search for the next occurrence use the “n” key, if you want to go backwards use the capital “N”.

To do a Search and Replace you’ll just have to go with me on this. Again, make sure you’re in Command Mode and hit “:” for a colon command. To replace “old” with “new” in the entire file the colon command looks like this:

:%s/old/new/g

Basically the %s means search every line and the g means change every occurrence in a line. The “/” marks the old and new fields. If you want to replace something with a “/” character in it use a different separator character like “.” or “|” – it’s just looking for a punctuation mark and it will carry on using whatever punctuation mark it finds first.

That’s it?

You want it to be more complicated? I’ve seen many tutorials and cheat sheets explain a lot of stuff you don’t need for simple editing. Commands to move the cursor quickly, repeated commands and so on. Yes, you can scroll ten lines down using 10j but who cares? Just hit the down arrow ten times – assuming you even know that your destination is that far below. I might add a batch of secondary very useful commands later but they’re not essential. However, read on if you don’t have cursor keys or can’t use a mouse for cut and paste, or care about line numbers.

Line numbers

If you care which line you’re on, possibly because you’re getting a message line “error in line 123” of your config file and need to fix it, there are a few extras that might help. If you want to jump to line 123 use “:123”. As you can imagine, to get to the top of the file quickly you can also use “:1”, and use “:$” to go to the bottom.

You can turn on a display of the current line number and cursor column with “:set ruler” and display line numbers with “:set number”. To turn these things off prefix them with “no” – e.g. “:set nonumber”

Cut+paste with keyboard

If you’re using a real hardware terminal instead of a virtual software one (with a mouse) you can still cut and paste, but I’ll need to explain something about buffers and deleting first. When you delete anything in vi it goes into a buffer. The basic delete command is “d” followed by any movement command. Anything between the old and new cursor positions gets deleted and put in the buffer. There are a lot of movement commands, but the one we’re interested in is “jump to marker”.

To place a marker in the file at the cursor position you must be in Command Mode (hit [Esc] if you’re not sure) and then type “ma”. This sets marker “a”. You can also set marker b, c, d and so on, which is useful if you want to bookmark different places in a file.

To jump to a marker use a “ ` ” (single back quote, normally to the left of the “1” on the keyboard) followed by the marker letter – i.e. `a

So to cut some text, set the marker at the start, move to the end and type d`a and the text will disappear into the paste buffer. If you wanted to Copy instead of Cut use y`a instead – y is Copy.

In order to paste the contents of the buffer, go to where you need it and type capital “P”. This will insert at the current cursor position, which is what most people want.

A variation in this is to use a ‘ instead of a ` (a single quote instead of a backquote), which will cut/paste whole lines between the cursor and the marker. Beware, a lower case “p” instead of a capital for paste puts the buffer after the current cursor position by either one character or one line depending on whether you used a ` or a ‘ originally. Stick with P to avoid confusion.

No cursor keys?

If your keyboard doesn’t have cursor keys, or they’re not working, you can use the following to move around:

left=h, right=l, k=up, j=down. Ctrl-B and Ctrl-F give you Page Up and Page Down.

A note on VIM

vim is a “Vi Improved” editor found in GNU/Linux. It claims to be mostly compatible with VI and most of the above should work. One of its improvements is a multi-buffer undo facility, presumably because Linux users make more mistakes.

It’s curious that anyone would want to improve in vi, and I say this having used it for over 45 years now. It’s not an editor that I’d choose as a starting point to write a better one. Improved editors are out there, including the straightforward nano editor that is bundled with most GNU/Linux distros; you may find that more friendly if it’s available.

After all this time using vi I certainly know a few tricks and can use it very quickly and efficiently, but the main reason for learning it now is that it will be there when you need to edit something, on every Unix-like system.

6-May-26

Singapore Airlines appears to have data breach

How do I know this? My alarm systems have just triggered on an email address known only to Singapore Airlines, that’s how. For decades I’ve seeded special email addresses with various organisations and waited to see if criminals got hold of them.

But hang on – they suffered a supply chain attack in 2021. Except this one was seeded on 9th April 2025, so my guess is that it’s happened again.

Watch this space!

27-April-2628-April-26

BBC Uncanny – the White Mountain UFO

Anyone interested in supernatural folklore is likely to have heard of the Danny Robins series “Uncanny” on Radio 4. Episodes are written and presented by Danny Robins, with guest various experts – notably Ciarán O’Keeffe for the septics and Evelyn Hollow for the believers. The latest venture is “Uncanny Cold Cases”, currently running, presenting older stories instead witness testimony.

This is worth a listen if you’re interested in what leads people to believe what they do, and an awful lot of people do think they’ve seen a ghost. But for some reason, Uncanny can’t keep away from UFO stories. With ghosts, people say they’ve seen something strange and you either believe them or don’t, but UFOs are more objective.

The latest episode is “The White Mountain UFO”, which is referring to the alleged abduction of Barney and Betty Hill in 1961. The couple were driving back from a holiday in Montreal to their home in Lancaster, New Hampshire, USA. Seeing a light in the sky they stopped to have a better look. Having seen a flying saucer and aliens they fell into a daze and woke up in their car, close to home, with no recollection of how they got there. The story has become embellished over time.

A mundane explanation is that they were over-tired, hallucinating and it’s lucky they didn’t fall asleep and drive into a tree. However, under hypnosis they both told a similar story about being abducted by aliens in the missing hours.

In an era of cold war paranoia, and in a country gripped in the 1960s flying saucer hysteria, the Hills were interviewed by the USAF and NICAP, the (American) National Investigations Committee On Aerial Phenomena.

So far so meh. Where Uncanny suddenly became interesting was with some physical evidence. Apparently Mrs Hill’s dress had some unexplained pink stains that were investigated later and found to be contaminated in a way that was impossible by natural means. They found Rhodium and Tellurium on the dress, which are present at the earth’s surface at 0.0002 parts per million – but possibly more common in space. These are the “facts” presented.

But is this the smoking gun Uncanny is suggesting? Some physical indisputable evidence? I hadn’t heard of the dress being tested before before, so I did some digging. You’ll never guess what I found.

The tests were actually done by analytical chemist Phyllis Budinger, who had become interested in the case. She believes in UFOs. She contacted Kathleen Marden, Betty Hill’s niece, and obtained five fabric samples in 2001; the lining, three from the discoloured areas and a fourth from a clean patch as a control. Over the next couple of years she conducted tests – infrared spectra on the surface and of solvent extractions, and good old microscope analysis. She also got some X-ray fluorescence tests done by the University of Pittsburgh. The results appeared in a book, “Captured!”, written none other than Ms Marden and Stanton Friedman, where it’s stated that this new analysis found rare and unusual heavy metals. And this appears to be the source used in Uncanny.

However, this is a very selective interpretation. These heavy metals are indeed seriously rare. However the figure giving their prevalence is an average over the earth’s surface but this says nothing about concentrations. Being rare on average is not the same as being rare on early 1960s dresses.

Those doing the XRF were somewhat more circumspect. They noted it was unusual, and suggested the graduate chemistry department conduct more extensive scans to corroborate, describing their own tests as “cursory”. As far as I can tell, no such further testing was carried out.

Budinger herself suggests getting a biologist involved and looking at the DNA, but concluded the pink was a result of mould caused by whatever was contaminating the fabric in some areas.

The results can be found as Technical Service Response No UT025 from Frontier Analysis Ltd.

So that’s that for the physical evidence. What of the hypnosis? Both of the hills recounted a story with details very similar to episodes of a TV programme “The Outer Limits” that aired in the weeks before the hypnotic regressions. The Hills may have been accessing repressed memories of their abduction three years earlier, or they may have been remembering a TV programme they watched three weeks earlier. You decide.

The sessions were conducted by psychiatrist Dr. Benjamin Simon, who concluded that the abduction experience as recalled under hypnosis “did not take place but was a reproduction of Betty’s dream which occurred right after the sighting.” The story revealed under hypnosis is often presented as true, but this is a misrepresentation. Dr Simon’s letters to Philip J. Klass make his scepticism clear. Meanwhile other people, including the aforementioned Ms Marden, need it to be true to sell books. This may affect their credulity, again you decide.

Further corroborating evidence of the Hills account was presented in the episode – specifically radar returns (contacts) of an unknown object at the time of their disappearance. These may have been taken from various books on the subject – quite likely Jacques Vallée’s “Dimensions”, which describes a return signal monitored by Pease Air Force Base. This was on the approach radar, and the extract reads:

observed unidentified aircraft come on precision approach radar 4 miles out, aircraft made approach and pulled up at half a mile, shortly after observed weak target on downwind, then when it made low approach, tower unable to see any aircraft at any time.

This was at 2:14am local time, after the Hill’s sighting, possibly the UFO flying away after being undetected for the last four hours? Or perhaps it was an unidentified aircraft on an aborted approach, which isn’t exactly unusual.

Another return comes from the North Concord Air Force in Vermont, which tracked an object for 18 minutes – fitting the podcast. It was recorded by Project Blue Book. However, it’s not corroboration either:

…one object detected on height-finder radar at 62,000 feet, appearing at 196 degrees at 84 miles, lost at 199 degrees at 80 miles, moving north then south, observed for 18 minutes.

It was high in the air, moving slowly and erratically, exactly like a weather balloon – which was their actual assessment. It was also seen at 5:22pm Eastern time – long before the Hills’ sighting.

Mr Hill died before his time in 1969 aged 43, but his wife survived until 2004 and was well known in UFO circles, although not universally believed according to many accounts.

So where does this leave the BBC’s Uncanny podcast? The evidence that this story doesn’t amount to a white hill of beans is out there, so why didn’t Dr O’Keeffe blow it out of the water? I guess that’s entertainment!

17-April-26

FreeBSD error: leapsecond file expired

You might see something odd in the console log about a leapsecond file:

ntpd[905]: leapsecond file ('/var/db/ntpd.leap-seconds.list'): expired 841 days ago

It just means that the leap second file is out of date, and couldn’t be updated automatically. It’s not a big deal as this data rarely changes. In fact the last one was in 2017. You can force an update with the following line:

service ntpd fetch

However, this might well give you another error like “fetch: https://www.ietf.org/timezones/data/leap-seconds.list: Not Found” The reason is that the IETF is not longer hosting the leap-seconds file although it’s baked into the FreeBSD (and likely other) configs.

On FreeBSD you can easily point it to another host at IANA by adding the following to /etc/rc.conf:

ntp_leapfile_sources="https://data.iana.org/time-zones/data/leap-seconds.list"

After you’ve done this you can trigger it manually using the fetch line above. The old source is actually configured originally in defaults/rc.conf, but it’s best to override it in the local rc.conf – and newer releases of FreeBSD have updated it anyway.

4-March-2610-March-26

The Crazy Politics of Age Verification

The UK government required age verification for pornographic website to protect children. Australia was the first country in the world to ban children from social media. Children, of course, circumvented all such restrictions immediately. And now the UK government and others around the world are thinking “This plays well with the public” and is trying to follow suit, in spite of the fact the teenagers are laughing at the restrictions.

Rather than getting an informed opinion on the practicality of such measures from technology experts, the politicians, as usual, get their technical understanding from some random teenagers on social media.

This lack of tech savvy would be hilarious if it wasn’t such a serious matter. It’s nothing new. And these proposed technological solutions to problems in society are doing nothing to fix the very real issues they’re trying to address. You cannot fix social problems using technology. Leave a comment if you can think of any example where this has been the case.

Next Monday there’s a vote in the commons on just this, and I fear grandstanding arts-graduate politicians will let it through, regardless of the technical impossibility of complying. And there’s no end to this nonsense.

There’s crazy, then there’s California

And just when you think it can’t get any worse, the Californians have taken it to the next level with a new bill called Age Verification signals: software applications and online services.

Here’s an extract:

1798.501. (a) An operating system provider shall do all of the following:

(1) Provide an accessible interface at account setup that requires an account holder to indicate the birth date, age, or both, of the user of that device for the purpose of providing a signal regarding the user’s age bracket to applications available in a covered application store.

(2) Provide a developer who has requested a signal with respect to a particular user with a digital signal via a reasonably consistent real-time application programming interface that identifies, at a minimum, which of the following categories pertains to the user:

(A) Under 13 years of age.
(B) At least 13 years of age and under 16 years of age.
(C) At least 16 years of age and under 18 years of age.
(D) At least 18 years of age.

Having an handy OS function to determine the age of the user seems like a good idea, doesn’t it? Naughty software can refuse to run if the user is under-age. But it’s technical nonsense; not thought through at all. For example, it’s referring to a “real-time application programming interface” – so this only applies to a RTOS? Obviously they saw the phrase, thought it sounded good and included it.

But the concept has some merit – rather than everything doing age verification, why not dump the problem on Microsoft, Apple and Google? They’ve got the resources, and with their penchant for knowing everything about their customers, their OS could easily provide age verification information along with their shoe size and taste in pot plants.

Except… you need to consider the chilling effect this will have on anything called an Operating System. The lawmakers probably wouldn’t recognise an OS if it jumped out at them waving a flag, but this is important here. For starters, every embedded system has software that fits the definition of Operating System. Do they have to provide age verification?

And what about, say, servers? They don’t have a user as such, but they have people using them (administrators). How can you tell the age of the administrator at any given time? The account may well be shared by a number of people.

But the most damage from this nonsense will be to FOSS operating systems like Linux. They don’t have a KYC culture like Microsoft/Apple/Google to make it easy and don’t have the time, money or resources to solve the problem. An open source operating system is community owned, and the community has no incentive to track the personal details of who might be using it.

Although Linux is free, it’s still licensed using the GPL and has a vendor – even if no money changes hands. If this bill ever passes, the State of California may well go after the vendor for non-compliance and try to force them into it by fines and suchlike.

The logical solution to the problem for Linux distributions (and FreeBSD and whatever) is to change the terms of their license saying “You may not download or use this software in the State of California”. Microsoft, Apple and Google will love that.

28-February-26

Process states in “top”

This applies to FreeBSD, but is similar on Linux.

Both the top and ps utilities will tell you what a given process is doing, which is generally running on a CPU or waiting for something. However, the documentation doesn’t really tell you what these states mean. The man page for the ps utility suggests reading the system source code. (sys/proc.h).

In this post I’ll deal with the common process states in top, the STATE column in the screenshot below.

Other columns are:

PID is the process-ID
USERNAME the user that the process is running under.
THR isn’t documented but I’m very sure it’s the thread count – i.e. the number of threads used by a multi-threaded process.
PRI is the current process priority, and NICE is the nice value – an often misunderstood weighting used by the scheduler when determining the current priority. It’s outside the scope of this post.
SIZE and RES are the total size of the process and the amount of real RAM currently being used, given it may have allocated memory that hasn’t been used yet or may be paged out.
C is the CPU number to which the process is currently assigned
TIME is the amount of CPU time (in seconds) the process has used since it was started.
WCPU is the percentage CPU time currently being used by the process. Note that if you have four CPUs you can have 400% utilisation, as this applies to a single CPU.

And then, of course, there’s STATUS.

Officially, the status is one of one of “START”, “RUN”, “SLEEP”, “STOP”, “ZOMB”, “WAIT”, “LOCK” or the event being waited for. Run means it’s the currently running process, but SMP systems, RUN will be replaced by CPUn, where n is the CPU number doing the running. You’re unlikely to actually see the others as if a process isn’t running it’s going to be waiting for an event. But this is what they mean:

START. A very short-lived state when the process is in the process of being created.
SLEEP. The process can’t run as it’s waiting for an event (a character to be typed, a disk operation to complete and suchlike). In top you normally see the event being waited for, and these will be listed later.
WAIT. A parent process is waiting for a child process to finish, or more accurately, change state. This means the parent process has called wait(), waitpid(), wait4() or similar (see man 2 wait for a full list).
LOCK. The process is waiting until the kernel grants it a lock of some kind. You normally see the lock its waiting for prefixed with a ‘*’ rather than just plain “LOCK”.
CPUn. The process is currently running on CPU n on an SMP system.
RUN. The process is currently running on the single CPU.
STOP. The process has been stopped (suspended) by sending it a SIGSTOP (e.g. by typing Ctrl-Z). It may be restarted using SIGSTART (or running fg/bg).
ZOMB. A process has stopped but remains in memory as the parent hasn’t collected its exit status yet. This state doesn’t normally last long unless something’s wrong with the parent. You can’t kill a zombie process (the clue is in the name) so if you have one hanging around it will need a reboot to clear it – but don’t worry too much as it won’t be using much memory or other resources.

As I’ve said, you probably won’t see many of these as a process spends most of its time waiting for an event to happen, and in such cases, it shows the event in question. Common events are:

STATE	Meaning	Reason or system call(s) involved
kqread	Waiting for an event to be posted to a kqueue descriptor	kevent() extremely common in modern servers (e.g., nginx, OpenZFS-related daemons, libevent-based apps)
sigwai	Waiting for a signal	sigwait(), sigwaitinfo(), sigtimedwait(); used by POSIX signal-handling threads
select	Waiting to read/write file.	Legacy select() or pselect() calls, still common but being replaced with kqueue/poll.
nanslp	Sleeping with nanosecond precision	nanosleep() or clock_nanosleep() used for timers, short sleeps, Rate limiting.
lockf	Blocked waiting on an advisory file record lock (byte-range lock)	Database or similar waiting to lock part of a shared file. fcntl(…, F_SETLKW, …)
accept	Waiting for incoming TCP connection	Classic blocking accept loop; seen in prefork servers, simple daemons calling accept()
pause	Suspended waiting for any signal	Used by older software (including the shell!) calling pause()
wait	Waiting for a child process to change state or end.	wait(), waitpid() etc. Very common for parent processes (shells, init-like processes, daemons that fork children)
CPUn	Actively running on CPU number ‘n’	It may mean that the process is in a state that it can be given to a CPU, or it may actually be running.
sbwait	Waiting for socket buffer space (send) or data arrival (receive)	Socket I/O wait (e.g., TCP send buffer full or recv waiting)
biord biow	Blocked on block I/O read / write (disk/network filesystem operations)	Waiting for disk I/O completion
piperd pipewr	Blocked reading or writing to a pipe	Pipe I/O wait. Given pipes are now sockets you don’t see this on BSD any more (or at least, I don’t).
uwait	Userland wait	Often related to threading / synchronization primitives like pthread_cond_wait() , sem_wait()

10-February-2617-February-26

Installing K8S on RHEL or Oracle Linux 9

Installing Kubernetes is like nailing jelly to the ceiling. Here’s a script that actually does it, with comments so you know what’s going on.

Realistically, there’s no point in running Kubernetes on a single host – its for clustering. But just to prove it’s possible, this will do it. You can’t normally run a pod on a control node, but with one node you can remove the taint and do it anyway.

I had so many goes at doing this that I wrote this script so I could automate it until I got it right. You can run this script, or do it a command at a time (probably better) as this is only known to work on one particular configuration.

Because its very picky about which versions of various things you have, for the important stuff like containerd and the Kubernetes utilities itself I’ve ended up downloading them from github directly, and installing the various config files for systemd manually. At the start of the script I’m defining these version numbers so they are easy to tweak. One day I might rewrite using dnf based on information gleaned by using the direct approach.

To understand what’s going on, read the comments.

And as a bonus there’s a “hello” pod installed, running Nginx. You can pull its “welcome” page using curl.

#!/bin/bash

# This is a trick that will cause the script to exit immediately if
# any command returns a failure. Do not use this if you are running
# the commands by hand!

set -euo pipefail

# Set up the versions we're using. Trial and error has proved
# that these versions of various things play nice together
# on Oracle or RHEL 9.7

K8S_VERSION="v1.29.15"
CONTAINERD_VERSION="1.7.5"
APISERVER_IP=$(hostname -I | awk '{print $1}')
# Choose a published version (e.g., v1.32.0) as 1.29 isn't.
CRICTL_VERSION="v1.32.0"
CNI_VER="v1.1.1"   # stable version

# Prior to K8S 1.21 alpha, having swapping enabled was
# a disaster. Now it's supposed to work, but having a swap device
# on a VM is a bit crazy, so we'll disable it anyway.

echo "=== Disable swap ==="
swapoff -a
sed -i '/swap/d' /etc/fstab

# These packages are going to be needed and may not be installed
# already.

echo "=== Install dependencies ==="
dnf install -y curl tar wget socat conntrack iptables iproute-tc git

# Assuming firewalld is running it's going to stop us communicating.
# It's probably best to disable it completely with
#
# systemctl disable --now firewalld
#
# Getting K8S running can be tricky enough without a firewall getting
# in the way. However, we can open the ports we know about and hope
# for the best.

echo "=== Open firewall ports ==="
firewall-cmd --permanent --add-port=6443/tcp
firewall-cmd --permanent --add-port=10250/tcp
firewall-cmd --reload

# So now we're set up for installing Kubernetes. We'll start with
# our container managing choice, containerd. Note that it no longer
# requires Docker (or Podman). It handles the contaiers directly.
#
# To avoid problems with repos and "latest versions" I'm just
# downloading the versions I want from the github repos.

echo "=== Install containerd from GitHub ==="
curl -LO https://github.com/containerd/containerd/releases/download/v${CONTAINERD_VERSION}/\
containerd-${CONTAINERD_VERSION}-linux-amd64.tar.gz

tar Cxzvf /usr/local containerd-${CONTAINERD_VERSION}-linux-amd64.tar.gz
rm -f containerd-${CONTAINERD_VERSION}-linux-amd64.tar.gz

# This is the systemd service file, which we need to install manually.

cat <<EOF > /etc/systemd/system/containerd.service
[Unit]
Description=containerd container runtime
After=network.target

[Service]
ExecStart=/usr/local/bin/containerd
Restart=always
RestartSec=5
Delegate=yes
KillMode=process
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
EOF

# We need to create a config file for containerd manually too.
# This is done by getting it to dump its default config.

mkdir -p /etc/containerd
/usr/local/bin/containerd config default > /etc/containerd/config.toml

# We need to enable the systemd cgroup driver in containerd
# as it's disabled by default. Edit /etc/containerd/config.toml
# and find the line SystemdCgroup = false, and change it to true.
# (Scripted here using sed)

sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

# The next three commands will tell systemd something
# has changed so it will reload our new files. we
# can also kick containerd off now. I think the third
# systemctl is unnecessary as the --now when it is enabled
# should be enough.

systemctl daemon-reload
systemctl enable --now containerd
systemctl restart containerd

# So much for systemd, now wen need to do the same for Kubernetes.
# Installing it manually is very easy - just download the files into
# /usr/local/bin/ and make them executable.

echo "=== Install kubeadm/kubelet/kubectl from dl.k8s.io ==="
mkdir -p /usr/local/bin
cd /usr/local/bin
# Download kubeadm kubelet and kubectl
curl -LO https://dl.k8s.io/release/${K8S_VERSION}/bin/linux/amd64/kubeadm
curl -LO https://dl.k8s.io/release/${K8S_VERSION}/bin/linux/amd64/kubelet
curl -LO https://dl.k8s.io/release/${K8S_VERSION}/bin/linux/amd64/kubectl
chmod +x kubeadm kubelet kubectl

# Now we have to set up kubelet systemd service

cat <<EOF > /etc/systemd/system/kubelet.service

[Unit]
Description=Kubernetes Kubelet
After=network.target containerd.service
Requires=containerd.service

[Service]
ExecStart=/usr/local/bin/kubelet \
  --kubeconfig=/etc/kubernetes/kubelet.conf \
  --config=/var/lib/kubelet/config.yaml \
  --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
  --cgroup-driver=systemd \
  --v=2
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

# As before, we'll tell systemd something has changed and kick
# off the kubelet service.

systemctl daemon-reload
systemctl enable --now kubelet
systemctl restart kubelet


# crictl is a lightweight command line utility for managing containers
# and suchlike, used by Kubernetes in preference to Docker or Podman.
# Again, we're going to download a specific version direct from github.
# Download and extract crictl as a tarball, unpack it in to /usr/local/bin
# and clean up afterwards.

curl -LO https://github.com/kubernetes-sigs/cri-tools/releases/download/\
${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-amd64.tar.gz

tar zxvf crictl-${CRICTL_VERSION}-linux-amd64.tar.gz -C /usr/local/bin
chmod +x /usr/local/bin/crictl
rm -f crictl-${CRICTL_VERSION}-linux-amd64.tar.gz

# Next we need to attend to the networking.
# We need to bridge two network interfaces
# using the br_netfilter kernel module and then
# enable port forwarding.

# This sets it up now, live.

modprobe br_netfilter
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/ipv4/ip_forward

# This creates a sysctl file so it will be set on boot.
# I've added some support for IPv6. We can make it reload
# to make it live immediately using sysctl.

cat <<EOF > /etc/sysctl.d/99-kubernetes.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sysctl --system



# Some installs of RHEL don't have /usr/local/bin in the path all
# of the time, especially if you're switching user. This bit of
# script checks and adds it if necessary.
echo $PATH | grep /usr/local/bin || PATH=$PATH:/usr/local/bin

# Static pods like kube-apiserver don't need CNI, but kubelet
# requires the pause image to start sandbox pods.

echo "=== Get the pause image ==="

ctr images pull k8s.gcr.io/pause:3.10
# Check it's worked using grep. Note this script stops it it doesn't find it.
ctr images ls | grep pause

# kubeadm will pull the required images during init
# but we're going to pull them ahead of time. They can take
# a while and we want to make sure they're available. 

echo "=== Pull the k8s images ==="
kubeadm config images pull

# Finally, we initialise the Kubernetes cluster. Note we
# finessed our main IP address at the start. The pod network 
# is your choice. This step can take a while.

echo "=== Initialize Kubernetes cluster ==="
kubeadm init \
  --pod-network-cidr=10.244.0.0/16 \
  --cri-socket /run/containerd/containerd.sock \
  --apiserver-advertise-address=${APISERVER_IP}

# Assuming it initialised, you're probably good!
#
# If we're running K8S as the root user, which isn't always
# a good idea but for testing it's fine, we need to create about
# .kube directory for root's configuration files.
#
# We could instead export KUBECONFIG the one in /etc with:
#
# export KUBECONFIG=/etc/kubernetes/admin.conf
#

echo "=== Configure kubectl for root user ==="
mkdir -p /root/.kube
cp -f /etc/kubernetes/admin.conf /root/.kube/config
chmod 600 /root/.kube/config
export KUBECONFIG=/root/.kube/config

# Flannel is the last important thing we need configured. It's a CNI
# plugin for Kubernetes that provires the layer 3 (IP) networking.
# Here we're telling kubectl to download it direct from github as it's
# a bit long to embed in this script.

echo "=== Install Flannel CNI ==="
kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

# You may start off with the CNI plugins in /opt/cni/bin but they're
# probably gone by now. To be sure we'll download them from github.
# However, before you do this you can check /opt/cni/bin and it ought to container
# things like localhost as well as flannel.

mkdir -p /opt/cni/bin
curl -L https://github.com/containernetworking/plugins/releases/\
download/$CNI_VER/cni-plugins-linux-amd64-$CNI_VER.tgz | tar -xz -C /opt/cni/bin

# Right now we're running Kubernetes on a single node, which defeats
# the whole point but we're only testing at present. The snag is that
# Kubernetes needs a control node and worker nodes to run pods on, and you
# can't have a pod on the contoller. It's a bad idea. But we can force
# it to allow us anyway by removing the taint from it. A taint is a node
# property that tells the Kubernetes scheduler to "keep away", and is
# normally used to reserve a node for specific workloads. This include
# the control node, so if we remove the taint it will drop pods on
# itself anyway.

echo "=== Remove control-plane taint for single-node ==="
kubectl taint nodes --all node-role.kubernetes.io/control-plane-

# And to prove it's all working, we'll drop the nginx web server
# on a hello pod and expose http port 80 so we can talk to it.

echo "=== Deploy Hello World NodePort ==="
kubectl create deployment hello --image=nginx --replicas=1
kubectl expose deployment hello --type=NodePort --port=80

# This will verify that everything is running, although it may take a moment or
# three to start.

kubectl get svc

# This bit of shell scriptery extracts the node port visible
# when you run kubectl get svc and prints out the line necessary
# to allow you to access the http server using curl.

NODEPORT=$(kubectl get svc | grep hello | awk '{print $5}' | cut -d: -f2 | cut -d/ -f1)

if test -z "$NODEPORT"
then
    echo "It doesn't look like the hello pod is running."
else
    echo "To get the nginx hello page use curl $APISERVER_IP:$NODEPORT"
    echo "once it's had time to start"
fi

As a bonus, here’s a script to “clean up” after a bad attempt at kubeadm init. kumeadm reset doesn’t do enough!

#!/bin/sh

# Nuclear reset
# This code cleans up after a bad attempt at configuration (kubeadm init)

systemctl stop kubelet
systemctl stop containerd
systemctl disable kubelet
systemctl disable containerd

kubeadm reset -f
rm -rf /etc/kubernetes
rm -rf /var/lib/kubelet/*
rm -rf /var/lib/etcd
systemctl stop containerd
rm -rf /var/lib/containerd/*
echo Checking ports
ss -lntp | grep -E "6443|10250|10251|10252|10257|10258|10259"
# Line to automate kill, but leave it manual "| cut -d = -f 2 | cut -d , -f 1"
echo Anything come up? Please kill -9 the PID

20-January-2621-January-26

Linux swap files

Last year I wrote a piece about swap file strategies in general (Unix/Linux), but on further investigation I have discovered Linux has a twist in the way it handles swap devices. On Unix there has never been any metadata written to a swap device – it’s just a sequence of blocks, and that’s what I’ve always told people. It turns out that with modern Linux, there is.

Before Linux will use a file or block device for swapping you need to use the mkswap command on it (there’s no Unix equivalent). The man page doesn’t exactly explain what it does as the low level, and I was curious. Unix hasn’t needed metadata on swap (or page) files from 1971 to the present day, so what is Linux doing differently?

Dumping a zeroed swap device before and after running mkswap on it revealed the following had been added:

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400  01 00 00 00 ff ff 03 00  00 00 00 00 ad 40 e5 4b  |.............@.K|
00000410  87 1a 46 d2 b0 36 ee 60  6d 40 08 d1 00 00 00 00  |..F..6.`m@......|
00000420  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000ff0  00 00 00 00 00 00 53 57  41 50 53 50 41 43 45 32  |......SWAPSPACE2|
00001000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
40000000

So it’s definitely doing something. As you can see the first 1K remains unchanged, which is presumably to avoid messing up partition tables and suchlike. There’s some binary data at 0x400 (1K if you don’t speak hex) and then right on the 4K boundary there’s what appears to be a signature, “SWAPSPACE2”.

Time to go digging into the source code (not so easy with a Linux, as it’s not normally present). It turns out that this is mapped by a union of two structures, and confirmed my suspisions:

union swap_header {
    struct {
        char reserved[PAGE_SIZE - 10];
        char magic[10];
    } magic;
    struct {
        char bootbits[1024];
        __u32 version;
        __u32 last_page;
        __u32 nr_badpages;
        __u32 padding[117];
        __u32 badpages[1];
    } info;
};

If you’re not clear with ‘C’ structures, this is basically saying the header is PAGE_SIZE long, with the last ten bytes being used as a signature (called “magic” – the SWAPSPACE2 in the dump). PAGE_SIZE is, logically, the size of the page the kernel is using for virtual memory, so the header occupies exactly one slot. It’s found in arch/x86/include/asm/page.h, and is the result of a shift calculation based on another tunable, PAGE_SHIFT and results in 4K on most systems but might be adjusted to suit the MMU on non-Intel platforms.

The second structure in the swap_header, which is overlaid on the first because it’s a union, skips the first 1K, calling it “bootbits”, which is fair enough. There might be a bootstrap loader or partition table there. In fact there may well be if the whole drive is being used for swap. It’s not necessary to put a header on a raw block device but I suspect some Linux tools like to see one.

Then comes the version number, which appears to always be ‘1’, and the address of the last page in the device in case the kernel doesn’t know. Finally there’s the size of bad page table. After padding it up to 0x600 there’s a bad page table, four bytes per entry, listing blocks on the page device that can’t be used (the apparent one entry array is a ‘C’ pointer trick). Since the introduction of IDE/SCSI drives around 1990, the disk drives themselves handle bad block mapping so this is only really relevant to ST506 type interfaces, which indicates it must have been in the design of Linux since the very beginning – and even then it would have been there for legacy drives.

There’s another twist I found in the code – the first sixteen bytes of the padding (0x40C to 0x41B) may contain a UUID (in binary, not ASCII) and 0x41C to 0x42B might contain a volume label if you set one using the -L option on mkswap (which is the source I looked at to find out what was going on).

If you try to run swapon on a device or file that doesn’t have this header it will bork. I had a look a the kernel, but I don’t think it cares – however I dare say some userland utilities do (including the aforementioned swapon).

Does this matter? Well it means you’re being short-changed by 4K of swap space on each device, but that’s hardly a big deal. The advantage is that utilities can detect and refuse to swap on a drive or partition you didn’t intend for swapping. The important thing is to remember to put the header on if you’re using Linux instead of Unix.

19-January-2619-January-26

Does the new AWS Euro Cloud solve the Data Sovereignty problem?

No, obviously not. It’s just fine words, along with every other promise made by cloud computing vendors. I’m using AWS as an exemplar, as they’ve just hyped up the launch of their AW S European Sovereign Cloud.

According to AWS it’s “a new, independent cloud for Europe entirely located within the EU” and has been welcomed by their customers and policymakers.

Basically it’s rolling out new “sovereign” AWS Local Zones in which the data will always remain in the country of origin. That’s data residency, not sovereignty, but it’s a start – if Uncle Sam wanted access to this data they’d have to pressure AWS to hand it over. However, given the US Administration’s willingness to pressure Europe into making Demark hand over Greenland, such pressure isn’t unimaginable.

Stéphane Israël, managing director of the AWS European Sovereign Cloud and digital sovereignty, stated “Customers want the best of both worlds – they want to be able to use AWS’s full portfolio of cloud and AI services while ensuring they can meet their stringent sovereignty requirements. By building a cloud that is European in its infrastructure, operations, and governance, we’re empowering organisations to innovate with confidence while maintaining complete control over their digital assets.”

This is interesting. Governments do impose sovereignty requirements, but that doesn’t mean they get them. It just means a service provider has promised to keep their data within international borders while it’s convenient for them to do so. However, if the company is ultimately based abroad, the government there will be able to pressure it to hand the data over anyway, and the small print in the contract will allow them to comply to all applicable laws (present and future).

I don’t want to make the Americans out to be the bad guys here. For example, late last year a Canadian court ordered French bit barn OVHcloud to hand over customer data as part of a criminal investigation. This could have been handled by a mutual assistance treaty, but the Canadian police decided to go after OVH’s Canadian subsidiary. French law prohibits OVH from complying or they go to jail, but the Canadian’s will take it out on their subsidiary there if they don’t. Place your bets on the outcome.

And this is mild mannered Canada. Imagine what the Chinese or Americans might do, and for less reason? Companies in China are very much under the control of Communist Party, and the US CLOUD Act (Clarifying Lawful Overseas Use of Data Act, 2018) allows US officials to subpoena data from US companies stored on foreign servers with minimal court protections.

Now you might say you don’t really care if the CIA wants to read the innocuous chatter between Wizards on your MUD, but if it’s personal information protected by local law then you might find yourself in trouble whether the option to access it is taken or not.

So what can you do?

You can eschew multinational hyperscalers and use a local cloud provider, of course. Your local government can probably still access the data unless they’re too small to show up on radar, but this may not matter to you or your customers. The risk here, in my experience, is that these companies don’t last forever and may well be subsumed by a multinational. And to be honest, quality has proved even more variable than the hyperscalers. You won’t get the PaaS infrastructure tools and services provided by the likes of AWS, Azure GCP. However, going for the self-managed VPS option doesn’t tie you to a particular provider and give you an escape route if their sovereignty becomes compromised at a later date.

The problem remains – your VPS provider can snoop on you. One particular provider, who shall remain nameless, contacted me to warn I was running a compromised version of RHEL and I really needed to patch it or they’d take my server down. What? It turned out I had a CentOS repository (running on FreeBSD), which was needed to support testing and migrating a legacy project and their scanner had picked up an old (but obviously not running) version of log4j2 amongst the packages. The point is they were scanning the disks, albeit with the best of intentions. And if you think running whole disk encryption on a VM would stop that, think again.

Last week a friend asked what I thought of a particular UK cloud provider as a means to overcome sovereignty problems perceived by his commercial partners, which was the second thing that has prompted me to write this. He followed with “I expect you’ll say I should run my own servers in a data centre”, and he be right in many cases. If you can run your own VPS, you ought to be able to run your own servers, and they’ll be a lot cheaper. If you’re not sure about looking after the hardware, put a consultant on a retainer. And if you think hard disks have become a bit expensive lately, wait until you see what the could providers charge per gigabyte. Having your own servers also means you do have total control of your data. You know where it is at all times, and can whole disk encrypt it if necessary to protect against physical theft. With cheap asymmetric fibre lines you can even keep backups and archives in your company safe, or under a mattress as you prefer.

17-January-2629-January-26

Linux Network Cheat Sheet

This is an rewrite of a much older post.

Some Linux distributions use a system called iproute2 to configure network interfaces, which is a more limited version of the standard Unix system based around ifconfig. The original Linux rewrite of ifconfig was less capable, so rather than bringing it up to Unix standards it was thrown out and a new one written. This lists common tasks that are possible of both, and their equivalents. NB. “eth0” is any network interface in the format your system uses.

Task	Unix standard	Linux iproute2
List all interfaces	`ifconfig`	ip addr
List specific interface	ifc`onfig eth0`	`ip addr show dev eth0`
Bring up/down interface	`ifconfig eht0 up (or down)`	`ip link set eth0 up (or down)`
Set IP address	`ifconfig eth0 192.168.1.10/24`	`ip addr add 192.168.1.10/24 dev eth0`
Configure using DHCP	`dhclient eth0`	`dhclient eth0`
Set IPv6 address (syntax works with add/delete/alias etc)	`ifconfig eth0 inet6 2001:db8::1 prefixlen 64` or`ifconfig eth0 inet6 2001:db8::1/64`	`ip -6 addr add 2001:db8::1/64 dev eth0`
Add appletalk address	`ifconfig eth0 atalk 45.156`	`ip --appletalk addr add 45.156 dev eth0`
Set IP + netmask + broadcast (where defaults not suitable)	`ifconfig eth0 192.168.1.10 netmask 255.255.255.0 broadcast 192.168.1.255`	`ip addr add 192.168.1.10/24 broadcast 192.168.1.255 dev eth0`
Add alias (not explicit on Linux)	`ifconfig eth0 alias 192.168.1.2 netmask 255.255.255.0`	i`p addr add 192.168.1.2/24 dev eth0`
Delete specific IP (not possible on old Linux)	`ifconfig eth0 delete 192.168.1.2`	`ip addr del 192.168.1.2/24 dev eth0`
Rename interface	`ifconfig eht0 name wan1`	`ip link set eth0 down ip link set eth0 name wan1 ip link set wan1 up`
Show routing table	`netstat -rn`	`ip route`
Add default route	`route add default 192.168.1.1`	`ip route add default via 192.168.1.1`
Delete default route	`route delete default 192.168.1.1`	`ip route del default via 192.168.1.1`
Show DNS servers	`cat /etc/resolv.conf`	`resolvectl status`
Add DNS server	Edit `/etc/resolv.conf: nameserver 8.8.8.8`	`resolvectl dns eth0 8.8.8.8`
Delete all DNS servers	Edit `/etc/resolv.conf`	`resolvectl dns eth0 ""`
Set domain search order	Edit `/etc/resolv.conf`: `search example.com local.example.com`	`resolvectl domain eth0 example.com local.example.com`
Show listening sockets	netstat -an	`ss -tulnp`
Show interface status	`netstat -i`	`ip -s link`
Set mtu	`ifconfig eth0 mtu 9000`	`ip link set dev eth0 mtu 9000`
View ARP cache	`arp -a`	`ip neigh show`
Delete ARP entry	`arp -d 192.168.1.65`	`ip neigh del 192.168.1.65 dev eth0`
Delete ARP cache	`arp -d -a`	`ip neigh flush all`

Notes

Old Linux ifconfig couldn’t remove a specific IP address but could remove them all using ifconfig eth0 0.0.0.0. You can get the same effect on iproute2 using “ip addr flush dev eth0”. Unix doesn’t have a command that’s quite so destructive.

If you add an alias address on the same subnet as an existing IP address, give it a netmask of /32.

Old Linux produced a routing table using route -n (not -rn) and you’d need to use “gw” when adding or deleting one (e.g. route add default gw 192.168.1.1)

On Solaris you need to add a -p between “route” and “add” – e.g. route -p add default 192.168.1.1

Most versions of ifconfig on Unix systems accept CIDR notation as an alternative to specifying “netmask 255.255.255.255” – for example “192.168.1.1/24“

The ip command will usually infer which are IPv4 and IPv6 addresses but it can be made explicit using “ip -4” or “ip -6”. Likewise ifconfig normally doesn’t require “inet” or “inet6” to figure out which kind of IP address you’re giving it. -6 and inet6 have been used above to be 100% explicit. Linux supports fewer interface types than Unix.

It is not possible to delete or edit a single DNS server using the linux ip system, but you can delete them all and add back the ones you want.

On Linux, ip can only configure interfaces for IPv4 and IPv6 (i.e. it won’t support AppleTalk, ATM, Token Ring, or IPX etc.).

Making changes persist through boot

To configure an interface on boot on BSD use sysrc ifconfig_eth0="DHCP", for a DHCP address or edit rc.conf similarly. For a static address use ifconfig_bge0="192.168.1.123/24” and defaultrouter=”192.168.1.1” and add the nameservers to /etc/resolv.conf with a line like “nameserver 192.168.1.2” For Linux it’s much more complicated using other files or databases.

Debian/Ubuntu older versions

edit /etc/network/interfaces:

auto eth0
iface eth0 inet dhcp
For static address use:
auto eth0
iface eth0 inet static
address 192.168.1.123/24
gateway 192.168.1.1
dns-nameservers 192.168.1.2 192.168.1.3

Ubuntu with Netplan

edit /etc/netplan/01-netcfg.yaml (or similar):

network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: true

For static address use:

network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: false
      addresses: [192.168.1.123/24]
      gateway4: 192.168.1.1
      nameservers:
        addresses: [192.168.1.2, 192.168.1.3]

Red Hat/CentOS/Fedora (using network-scripts, older versions):

Edit /etc/sysconfig/network-scripts/ifcfg-eth0:

DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes

For static address use:

DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.1.123
PREFIX=24
GATEWAY=192.168.1.1
DNS1=192.168.1.2
DNS2=192.168.1.3

Systems with NetworkManager

(e.g., Fedora, Ubuntu desktop), try ip first but if you have to, use nmcli:

sudo nmcli con mod "Wired connection 1" ipv4.method auto
sudo nmcli con up "Wired connection 1"

For static address use:

sudo nmcli con mod "Wired connection 1" ipv4.method manual ipv4.addresses 192.168.1.123/24 ipv4.gateway 192.168.1.1 ipv4.dns "192.168.1.2 192.168.1.3"
sudo nmcli con up "Wired connection 1"

Replace “Wired connection 1” with your connection name (list with “nmcli con show”, get lots of stuff to tweak with “nmcli con show eth0”. You may also use the name of the ethernet interface when you find it.

Systems with systemd-networkd:

Create or edit /etc/systemd/network/20-wired.network:

[Match]
Name=eth0

[Network]
DHCP=yes

For static address use:

[Match]

Name=eth0

[Network]
Address=192.168.1.123/24
Gateway=192.168.1.1
DNS=192.168.1.2
DNS=192.168.1.3