Frank Leonhardt's Blog

I’ve had a problem with mysql failing to start with error 35 in the log file:

InnoDB: Unable to lock ./ibdata1, error: 35
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files. 

What to do? Google and you get a lot of Linux people saying that the answer is to reboot the box. Hmm. Well you don’t have to.

What causes the error is mysqld crashing, usually when system resources are exhausted. Rebooting will, indeed, unlock ibdata1 but so will killing the process that locks it. Yet the server isn’t running, so how can this be? Well actually part of it is – just not the part the service manager sees.

Run “ps -auxww | grep mysql” and you’ll find a few more. Send them a kill, wait for it to work and then restart. Obviously you can only do this and expect it to work if you’ve sorted out the resource problem.

If you’re in the data recovery, forensics or just storage maintenance business (including as an amateur) you probably already know about ddrescue. Released about twenty years ago by Antonio Diaz Diaz, it was a big improvement over the original concept dd_rescue from Kurt Garloff in 1999. They copy disk images (which are just files in Unix) trying to get as much data extracted when the drive itself has faults.

If you’re using Windows rather than Unix/Linux then you probably want to get someone else to recover your data. This article assumes FreeBSD.

The advantage of using either of these over dd or cp is that they expect to find bad blocks in a device and can retry or skip over them. File copy utilities like dd ignore errors and continue, and cp will just stop. ddrescue is particularly good at retrying failed blocks, and reducing the block size to recover every last readable scrap – and it treats mechanical drives that are on their last legs as gently as possible.

If you’re new to it, the manual for ddrescue can be found here. https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html

However, for most use cases the command is simple. Assuming the device you want to copy is /dev/da1 and you’re calling it thumbdrive the command would be:

ddrescue /dev/da1
thumbdrive.img thumbdrive.map

The device data would be stored in thumbdrive.img, with ongoing state information stored in thumbdrive.map. This state information is important, as it allows ddrescue to pick up where it left off.

However, ddrescue was written before USB flash drives (pen drives, thumb drives or whatever). That’s not to say it doesn’t work, but they have a few foibles of their own. It’s still good enough that I haven’t modified ddrescue base code to cope, but by using a bit of a shell script to do the necessary.

USB flash drives seem to fail in a different way to Winchester disks. If a block of Flash EPROM can’t be read it’s going to produce a read error – fair enough. But they have complex management software running on them that attempts to make Flash EPROM look like a disk drive, and this isn’t always that great in failure mode. In fact I’ve found plenty of examples where they come across a fault and crash rather than returning an error, meaning you have to turn them off and on to get anything going again (i.e. unplug them and put them back in).

So it doesn’t matter how clever ddrescue is – if it hits a bad block and the USB drive controller crashes the it’s going to be waiting forever for a response and you’ll just have come reset everything manually and resume. One of the great features of ddrescue is that it can be stopped and restarted at any time, so continuing after this happens is “built in”.

In reality you’re going to end up unplugging your USB flash drive many times during recovery. But fortunately, it is possible to turn a USB device off and on again without unplugging it using software. Most USB hardware has software control over its power output, and it’s particularly easy on operating systems like FreeBSD to do this from within a shell script. But first you have to figure out what’s where in the device map – specifically which device represents your USB drive in /dev and which USB device it is on the system. Unfortunately I can’t find a way of determining it automatically, even on FreeBSD. Here’s how you do it manually; if you’re using a version of Linux it’ll be similar.

When you plug a USB storage device into the system it will appear as /dev/da0 for the first one; /dev/da1 for the second and so on. You can read/write to this device like a file. Normally you’d mount it so you can read the files stored on it, but for data recovery this isn’t necessary.

So how do you know which /dev/da## is your media? This easy way to tell is that it’ll appear on the console when you first plug it in. If you don’t have access to the console it’ll be in /var/log/messages. You’ll see something like this.

Jun 10 17:54:24 datarec kernel: umass0 on uhub5
kernel: umass0: <vendor 0x13fe USB DISK 3.0, class 0/0, rev 2.10/1.00, addr 2> on usbus1
kernel:  umass0 on uhub5
kernel:  umass0:  on usbus1
kernel:  umass0:  SCSI over Bulk-Only; quirks = 0x8100
kernel:  umass0:7:0: Attached to scbus7
kernel:  da0 at umass-sim0 bus 0 scbus7 target 0 lun 0
< USB DISK 3.0 PMAP> Removable Direct Access SPC-4 SCSI device
kernel:  da0: Serial Number 070B7126D1170F34
kernel:  da0: 40.000MB/s transfers
kernel:  da0: 59088MB (121012224 512 byte sectors)
kernel: da0: quirks=0x3
kernel: da0: Write Protected

So this is telling us that it’s da0 (i.e /dev/da0)

The hardware identification is “<vendor 0x13fe USB DISK 3.0, class 0/0, rev 2.10/1.00, addr 2> on usbus1” which means it’s on USB bus 1, address 2.

You can confirm this using the usbconfig utility with no arguments:

ugen5.1:  at usbus5, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (0mA)
 ...snip...
ugen1.1:  at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (0mA)
ugen1.2:  at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (300mA)

There it is again, last line.

usbconfig has lots of useful commands, but the ones we’re interested are power_off and power_on. No prizes for guessing what they do. However, unless you specify a target then it’ll switch off every USB device on the system – including your keyboard, probably.

There are two ways of specifying the target, but I’m using the -d method. We’re after device 1.2 so the target is -d 1.2

Try it and make sure you can turn your USB device off and on again. You’ll have to wait for it to come back online, of course.

There are ways of doing this on Linux by installing extra utilities such as hub-ctrl. You may also be able to do it by writing stuff to /sys/bus/usb/devices/usb#/power/level” – see the manual that came with your favourite Linux distro.

The next thing we need to do is provide an option for ddrescue so that it actually times out if the memory stick crashes. The default is to wait forever. The –timeout=25 or -T 25 option (depending on your optional taste) sees to that, making it exit if it hasn’t been able to read anything for 25 seconds. This isn’t entirely what we’re after, as a failed read would also indicate that the drive hadn’t crashed. Unfortunately there’s no such tweak for ddrescue, but failed reads tend to be quick so you’d expect a good read within a reasonable time anyway.

So as an example of putting it all into action, here’s a script for recovering a memory stick called duracell (because it’s made by Duracell) on USB bus 1 address 2.

#!/bin/sh
while ! ddrescue -T 25 -u /dev/da0 duracell.img duracell.map
 do
     echo ddrescue returned $?
     usbconfig -d 1.2 power_off
     sleep 5
     usbconfig -d 1.2 power_on
     sleep 15
     echo Restarting
 done

A few notes on the above. Firstly, ddrescue’s return code isn’t defined. However, it appears to do what one might expect so the above loop will drop out if it ever completes. I’ve set the timeout for time since last good read to 25 seconds, which seems about right. Turning off the power for 5 seconds and then waiting for 15 seconds for the system to recognise it may be a bit long – tune as required. I’m also using the -u option to tell ddrescue to only go forward through the drive as it’s easier to read the status when it’s always incrementing. Going backwards and forwards makes sense with mechanical drives, but not flash memory.

Aficionados of ddrescue might want to consider disabling scraping and/or trimming (probably trimming) but I’ve seen it recover data with both enabled. Data recovery is an art, so tweak away as you see fit – I wanted to keep this example simple.

Now this system isn’t prefect. I’m repurposing ddrescue, which does a fine job on mechanical drives, to recover data from a very different animal. I may well write a special version for USB Flash drives but this method does actually work quite well. Let me know how you get on.

I’ve seen all sorts of stuff on forums about how to process command line argument in C or C++. What a load of fuss and bother. There’s a standard getopt() function in the ‘C’ library, similar to the shell programming command, but it’s not great.

The main problem with getopt() is that it produces its own error message. Although this saves you the trouble, it can be a bit user unfriendly for the user, especially when things get complex. For example, you might have mutually exclusive arguments and want to print out a suitable message. That said, it’ll work most of the time.

Here’s a page showing a good explanation for the GCC version, whcih is pretty standard.

Example of Getopt (The GNU C Library)

But rolling your own is not hard. Here’s a skeleton I use. It’s pretty self-explanatory. My rules allow single options (e.g. -a -b) or combined options (e.g. -ab), or any mixture. “–” ends options, meaning subsequent arguments are going to be a actual arguments.

If you want to pass something as an option, such as a filename, you can. -fmyfile or -f myfile are both handled in the example.

You can add code to detect a long option by adding “if (!strcmp(p,”longname”)) … just after the char c. But I don’t like long options.

#include <stdio.h>

void process(char *s)
{
    printf("Processing %s\n",s);
}

int main (int cnt, char **a)
{
    int i;
    for (i=1; i<cnt && a[i][0] == '-'; i++)
    {
        char *p = &a[i][1];
        char c;
        if (*p == '-')
        {
            i++;
            break;
        }
        while (c = *p)
            switch (*p++)
            {
            case 'a':
                printf("Option a\n");
                break;
                
            case 'b':
                printf("Option b\n");
                break;

            case 'f':
                if (!*p && i+1 < cnt)
                    printf("Value for f=%s\n", a[++i]);
                else
                {
                    printf("Value for f=%s\n", p);
                    while (*p)
                        p++;
                }
                break;
                
            default:
                printf("Bad switch %c\n",c);

            }
    }
    for (;i<cnt;i++)
        process(a[i]);
}

The above code assumes that options precede arguments. If you want to mix them the following code allows for complete anarchy – but you can end it using the “–” option, which will take any following flags as arguments. As a bonus it shows how to add a long argument.

#include <stdio.h>
#include <string.h>

void process(char *s)
{
    printf("Processing %s\n",s);
}

int main (int cnt, char **a)
{
    int i;
    int moreargs=1;

    for (i=1; i<cnt; i++)
    {
            if (moreargs && a[i][0] == '-')
            {
            char *p = &a[i][1];
            char c;
                if (*p == '-')
                {
                    moreargs = 0;
                    continue;
                }
                if (!strcmp(p,"long"))
                {
                    printf("Long argument\n");
                    i++;
                    continue;
                }

                while (c = *p)
                    switch (*p++)
                    {
                    case 'a':
                        printf("Option a\n");
                        break;

                    case 'b':
                        printf("Option b\n");
                        break;

                    case 'f':
                        if (!*p && i+1 < cnt)
                            printf("Value for f=%s\n", a[++i]);
                        else
                        {
                            printf("Value for f=%s\n", p);
                            while (*p)
                                p++;
                        }
                        break;

                        default:
                            printf("Bad switch %c\n",c);

            }
        }
    else
        process(a[i]);
    }
}