Data Recovery: SD, ddrescue and Android

All things come to an end at some point, so do uSD cards. And they tend to do that just about the time you normally LEAST expect them to.
Anyways, at my country house, away from the noise of the big city I had a cheap cellphone tethering internets over an OpenVPN connection. The operator does not offer proper external IP service, so I have to run an OpenVPN connection to have access to surveillance.
I have a bunch of cams here and there, mostly watching after these guys:

The cellphone itself runs a rooted android and a debian chroot with OpenVPN off an SD card. The SD card died this weekend and at some point I realised that I don’t have a recent backup. It was no big deal, just a debian rootfs + a bunch of config files for OpenVPN, but since I spent a while then and now perfecting configs and tuning OpenVPN for performance over the celluar network those weren’t backed up. Ooops.

Anyways, this note talks about data recovery from such an SD card and the common pitfalls.

How SD cards die

SD cards can die in different and weird ways. Here’s my summary of the ways those can die. Perhaps not the most comprehensive, but comes from personal experience:

They become read-only if something fails inside them. Perhaps the most harmless way.

Some blocks become unreadable (my case), unless you at least write something there to fix the bad blocks. <- My case!

Some cards just don’t show up anywhere (card readers and mmc hosts alike) and show no life signs. (In case some internal connection under the epoxy is broken, the interface doesn’t work anymore). Sometimes data can be recovered in SPI mode.

After all all the crucial parts were just a few OpenVPN configs and scripts, so I had some chances.

Why you shouldn’t use a card-reader

SD card readers are complex devices that basically map SDIO bus (or even something else as well, e.g. memory stick) to USB storage and have a lot of things going on inside them. They are based off microcontrollers that run their own firmware, do buffering, error handling, etc. and we basically know very little about them. When I first plugged the card into the card reader I couldn’t do a thing – the reader just disconnected itself from the USB bus whenever it encountered a bad block. A different reader just hung there for a long time until reporting an unreadable block of data.

Lesson learned: SD card readers do a lousy job when it comes to error handling. Most likely those who actually made them never took an effort to test them with BAD cards.

What makes things worse, most laptops have their built-in card-readers connected over USB internally.

In any case, all we needed is to get rid of that buggy hardware abstraction layer called the card reader before we can do actual recovery. A raspberry or any other SBC with a built-in SD card slot would do fine, if they boot off some other device.

SD card slots in them are usually hooked to the kernel via mmc_host subsystem and therefore don’t have that bug, thanks to the perfectionists in charge of the linux kernel.

But what made the situation worse, I had no ready-to-use SBC at my disposal at that moment. But I did have a rooted android phone and a shitty internet connection 😉

ddrescue to the rescue.

I had an arm linux cross-toolchain at my disposal, but no Android NDK. The latter would be a huge download at my shitty internet connection. So I got the ddrescue tarball, cross-compiled it as a static binary (Android has a different incompatible libc, so we’ll have to be creative) and pushed to the device.

Now goes the most vital part of this show. Make sure your internal phone storage is bug enough to store all the recovered data, insert the damaged SD and fire up ddrescue. By default ddrescue will scrape the unreadable blocks on the SD card. So you’d better use one of this flags:

WTF??? The filesystem turned out to be clean, so I had to add -f flag along with -y and -v to force a verbose check without any stupid questions. Finally e2fsck found a bunch of errors and fixed those.

0 ✓ necromant @ sylwer ~ $ sudo e2fsck /dev/loop2p2

0 ✓ necromant @ sylwer ~ $ sudo e2fsck /dev/loop2p2

/usr/bin was no more, perhaps some other parts were missing, but /etc/ with all my OpenVPN scripts was intact that I consider a win. Once I applied those configs atop of the most recent backup everything went back to normal.