Posted on

FreeBSD hr utility – human readable number filter (man page)

Several years ago I wrote a utility to convert numeric output into human readable format – you know the kind of thing – 12345678 becomes 12M and so on. Although it was very clever in the way it dealt with really big numbers (Zetabytes), and in spite of ZFS having really big numbers as a possibility, no really big numbers have actually come my way.

It was always a dilemma as to whether I should use the same humanize_number() function as most of the FreeBSD utilities, which is limited to 64-bit numbers as its input, or stick with my own rolling conversion. In this release, actually written a couple of years ago, I’ve decided to go for standardisation.

You can download it from  here. I’ve moved it (24-10-2021) and it’s not on a prettified page yet, but the file you’re looking for is “hr.tar”.

This should work on most current BSD releases, and quite a few Linux distributions. If you want binaries, leave a note in comments and I’ll see what I can do. Otherwise just download, extract and run make && make install

Extracted from the man page:

NAME

hr — Format numbers in human-readable form

SYNOPSIS

hr [-b] [-p] [-ffield] [-sbits] [-wwidth] [file ...]

DESCRIPTION
The hr utility formats numbers taken from the input stream and sends them
to stdout in a format that’s human readable. Specifically, it scales the
number and adds an appropriate suffix (e.g. 1073741824 becomes 1.0M)

The options are as follows:

-b      Put a ‘B’ suffix on a number that hasn’t been scaled (for Bytes).

-p     Attempt to deal with input fields that have been padded with spaces for formatting purposes.

-wwidth      Set the field width to field characters. The default is four
(three digits and a suffix). Widths less than four are not normally useful.

-sbits  Shift the number being processed right by bits bits. i.e. multi-
ply by 2^bits. This is useful if the number has already been scaled in to units. For example, if the number is in 512-byte
blocks then -s9 will multiply the output number by 512 before scaling it. If the number was already in Kb use -s10 and so on.
In addition to specifying the number of bits to shift as a number you may also use one of the SI suffixes B, K, M, G, T, P, E
(upper or lower case).

k-ffield      Process the number in the numbered field , with fields being numbered from 0 upwards and separated by whitespace.

The hr utility currently uses the humanize() function in System Utilities Library (libutil, -lutil) to format the numbers.  This will repeatedly divide the input number by 1024 until it fits in to a width of three digits (plus suffix), unless the width is modified by the -w option. Depending on the number of divisions required it will append a k, M, G, T, P or E suffix as appropriate. If the -b option is specified it will append a ‘B’ if no division is required.

If no file names are specified, hr will get its input from stdin. If ‘-‘ is specified as one of the file names hr will read from stdin at this point.

If you wish to convert more than one field, simply pipe the output from one hr command into another.

By default the first field (i.e. field 0) is converted, if possible, and the output will be four characters wide including the suffix.

If the field being converted contains non-numeral characters they will be passed through unchanged.

Command line options may appear at any point in the line, and will only take effect from that point onwards. This allows different options to apply to different input files. You may cancel an option by prepending it with a ‘-‘. For consistency, you can also set an option explicitly with a ‘+’.  Options may also be combined in a string. For example:

hr -b file1 -b- file2

Will add a ‘B’ suffix when processing file1 but cancel it for file2.

hr -bw5f4p file1

Will set the B suffix option, set the output width to 5 characters, process field 4 and remove excess padding from in front of the original  digits.

EXAMPLES
To format the output of an ls -l command’s file size use:

ls -l | hr -p -b -f4

This output will be very similar to the output of “ls -lh” using these options. However the -h option isn’t available with the -ls option on the “find” command. You can use this to achieve it:

find. -ls | hr -p -f6

Finally, if you wish to produce a sorted list of directories by size in human format, try:

du -d1 | sort -n | hr -s10

This assumes that the output of du is the disk usage in kilobytes, hence the need for the -s10

DIAGNOSTICS
The hr utility exits 0 on success, and >0 if an error occurs.

Posted on

How to hack UNIX and Linux using wildcards

Leon Juranic from Croatian security research company Defensecode has written a rather good summary of some of the nasty tricks you can play on UNIX sysadmins by the careful choice of file names and the shell’s glob functionality.

The shell is the UNIX/Linux command line, and globbing is the shell’s wildcard argument expansion. Basically, when you type in a command with a wildcard character in the argument, the shell will expand it into any number of discrete arguments. For example, if you have a directory containing the files test, junk and foo, specifying cp * /somewhere-else will expand to cp test junk foo /somewhere else when it’s run. Go and read a shell tutorial if this is new to you.

Anyway, I’d thought most people knew about this kind of thing but I was probably naïve. Leon Juranic’s straw poll suggests that only 20% of Linux administrators are savvy.

The next alarming thing he points out is as follows:
Another interesting attack vector similar to previously described 'chown'
attack is 'chmod'.
Chmod also has --reference option that can be abused to specify arbitrary permissions on files selected with asterisk wildcard.

Chmod manual page (man chmod):
--reference=RFILE
use RFILE's mode instead of MODE values

 

Oh, er! Imagine what would happen if you created a file named “–reference=myfile”. When the root user ran “chmod 700 *” it’d end up setting the access permissions on everything to match those of “myfile”. chown has the same option, allowing you to take ownership of all the files as well.

It’s funny, but I didn’t remember seeing those options to chmod and chown. So I checked. They don’t actually exist on any UNIX system I’m aware of (including FreeBSD). On closer examination it’s an enhancement of the Linux bash shell, where many a good idea turns out to be a new vulnerability. That said, I know of quite a few people using bash on UNIX.

This doesn’t detract from his main point – people should take care over the consequences of wildcard expansion. The fact that those cool Linux guys didn’t see this one coming proves it.

This kind of stuff is (as he acknowledges) nothing new. One of the UNIX administrators I work with insists on putting a file called “-i” in every directory to stop wild-card file deletes (-i as an argument to rm forces an “Are you sure?” prompt on every file. And then there’s the old chestnut of how to remove a file with a name beginning with a ‘-‘. You can easily create one with:
echo test >-example
Come back tomorrow and I’ll tell you how to get rid of it!

Update 2nd July:

Try this:
rm ./-example

Posted on

Mark Shuttleworth’s Ubuntu Edge Dream

Mark Shuttleworth’s software company, Canonical Ltd, trying to raise $32M to build the first 40,000 units of a smart-phone type device that can run Ubuntu Linux. I predict he’s raise the money, and make the handsets. But the idea will tank anyway. Here’s why.

The concept of a ‘phone capable of running a desktop OS is easy to understand. When you want to use the desktop Ubuntu side you plug it in to a real monitor and keyboard – say one at home and one in the office. When you’re on the move it will run Android Linux (for Android is simply Linux with an Android graphical shell). You carry your environment with you, and carry on working wherever you are, assuming you have a monitor and keyboard available. If you run the Ubuntu graphical environment on the move, using the handset’s touch-screen it’s going to be pretty painful.

People investing about £600 will get a ‘phone, if they’re ever made. Is this an investment, or a pre-ordering deal? I think it’s up to you whether you invest enough to get a ‘phone, or buy even more equity as an investment in the future of the device, but I suspect a lot of people will simply be after the latest gadget. Whether £600 is too much for the Penguinistas, remains to be seen.

I think they stand a good chance of raising the money because they’re selling a dream that’s been around in various forms since the dawn of personal computing. One of the early incarnations would be the Apple IIc, which looked a bit like a portable typewriter when cut free from its monitor. With it you could carry your computer back from the office, but it didn’t catch on. Then, came the Tandon Data Pac, a hard disk cartridge. With a cartridge slot in PCs at the locations you needed to work, you could carry the important part of your environment with you. In those days, Microsoft didn’t do anything prevent hard disk transplants, so this was a realistic idea. But it didn’t catch on. Whether there are 40,000 people in the world who still have this dream is a good question.

Now we have laptop/notebook/netbook PCs, which are easy enough to carry in a briefcase if you get the right kind. I have always had the right kind, starting with the Cambridge Z88, moving on to the Sony Vaio and currently the Lenovo S10-3. At around 1Kg, they’re truly portable but although the Lenovo is modern, it was only on the market for a year or two as the 10″ screens format isn’t for well received by the masses. They demands big and fast, and they aren’t really worried about the battery life as long as they look cool. People often ask me “where can I get one of those”, and I tell them. (Currently only Asus and Acer producing a highly portable laptop/netbook). The snag is that when they get one they then “must” run Office 365, or some similar bloatware that a small CPU can’t handle fast.

If you don’t need battery life and the ability to work on the move, but simply want to carry your PC to and from the office, there are small form factor machines also from ASUS and Acer. If you want really small there’s the Fit-PC2 which can actually fit in a pocket. I must admit, I bought one because I thought it was a neat design. These are all Intel based and can run unmodified Windows, and yet they haven’t really caught on either. The Ubuntu Edge will not run Windows; it runs Linux. This means it won’t run Microsoft Office, ever. My experience has shown this is a big problem for a lot of people. There’s nothing wrong with OpenOffice; it’ll work with Microsoft Office files and vice versa. It’s free, whereas Microsoft Office costs and small fortune. Yet in nearly every case, people who I’ve set up with OpenOffice for cost reasons have hankered after the Microsoft version, and most have gone out and bought it (or otherwise acquired it) within a year.

The CPU for the Ubuntu Edge has yet to be announced, but based on size, battery life and heat dissipation it’s very unlikely to be Intel, or even Intel compatible. The only thing that will fit will be RISC, and given the binary nature of Linux distributions it’ll be the second-best choice of ARM. Or will its users be expected to compile everything from source? No. It’ll be an ARM and the models that are capable of running Linux with a GUI at nearly the right speed will still rip through the battery at an alarming rate.

The final nail in its coffin will be the way people currently commute with their computing environment. This comes down to cheap and cheerful thumb drive, if you can find a ubiquitous Windose PC at both ends, or on-line applications such as Google Docs if you’re really serious about it; all your data and applications on every web browser, and impossible to lose at that. If you can find a keyboard and monitor at both ends, you’re probably going to find a web browser anyway so why bother to carrying your stuff on a mobile ‘phone instead? It’s a solution to a problem that has been a “difficult sell” for 30 years, and which has now been solved by the Internet. Okay, this allows you to use an Android ‘phone between PCs, but you could just get an Android ‘phone to plug that gap in your life.