Search This Blog

Unpacking Android backups

One of the less known new features introduced in ICS is the ability to backup a device to a file on your computer via USB. All you have to do is enable USB debugging, connect your phone to a computer and type the adb backup command in a shell. That will show a confirmation dialog on the phone prompting you to authorize the backup and optionally specify a backup encryption password. It looks something like this:

This doesn't require rooting your phone and lets you backup application data, both user installed and system applications (APK's), as well as shared storage (SD card) contents. There are some limitations though: it won't backup apps that have explicitly forbidden backups in their manifest, it won't backup protected (with DRM) apps and it won't backup some system settings such as APN's and WiFi access points. The transfer speed is limited by ADB channel speed (less than 1MB/s), so full backups can take quite some time. There is also a rather annoying bug in 4.0.4 where it will backup shared storage even if you don't request it. With all that said, it's a very useful tool, and will hopefully see some improvements in the next Android version.

The backup command is fairly flexible and lets you specify what apps to backup, whether to include system apps when doing a full backup, and whether to include shared storage (SD card) files. Here's a summary of the available options as displayed by adb's usage:

adb backup [-f ] [-apk|-noapk] [-shared|-noshared] [-all] [-system|-nosystem] [<packages...>]
- write an archive of the device's data to <file>. If no -f option is supplied then the data
is written to "backup.ab" in the current directory.
(-apk|-noapk enable/disable backup of the .apks themselves in the archive;
the default is noapk.)
(-shared|-noshared enable/disable backup of the device's shared storage / SD card contents;
the default is noshared.)
(-all means to back up all installed applications)
(-system|-nosystem toggles whether -all automatically includes system applications;
the default is to include system apps)
(<packages...> is the list of applications to be backed up. If the -all or -shared flags
are passed, then the package list is optional. Applications explicitly given on the command
line will be included even if -nosystem would ordinarily cause them to be omitted.)

The restore command is however quite limited -- there are no options, you can only specify the path to the backup file. One of the features most noticeably lacking is conditional restore: restores are all or nothing, you cannot restore only a subset of the apps (packages), or restore only the contents of the shared storage. Supporting this will require modifying the firmware, but you can extract only the needed data from the backup and copy it manually. Copying apps and app data to your device requires root access, but extracting and copying external storage files such as pictures and music can be done on any stock ICS device. And if you create a backup file containing only the files you need to restore, you wouldn't need root access at all. This post will present the format of Android's backup files and introduce a small tool that allows you to extract and repackage them as needed.

SDK API's for using Android's backup architecture were announced as far back as Froyo (2.2), but it has probably been available internally even before that. As introduced in Froyo, it uses a proprietary Google transport to backup application settings to the "cloud". ICS adds a local transport that lets you save backups to a file on your computer as well. The actual backup is performed on the device, and is streamed to your computer using the same protocol that adb pull uses to let you save a device file locally. When you execute the adb backup command a new Java process (not an activity or service) will be started on your device and it will bind to the system's BackupManagerService and requests a backup with the parameters you specified. BackupManagerService will in turn start the confirmation activity shown above, and execute the actual backup if you confirm (some more details including code referenceshere). You have the option of specifying an encryption password, and if your device is already encrypted you are required to enter the device encryption password to proceed. It will be used to encrypt the archive as well (you can't specify a separate backup encryption password).

After all this is done, you should have a backup file on your computer. Let's peek inside it. If you open it with your favourite editor, you will see that it starts with a few lines of text, followed by binary data. The text lines specify the backup format and encryption parameters, if you specified a password when creating it. For an unencrypted backup it looks like this:

ANDROID BACKUP
1
1
none

The first line is the file 'magic', the second the format version (currently 1), the third is a compression flag, and the last one the encryption algorithm ('none' or 'AES-256').

The actual backup data is a compressed and optionally encrypted tar file that includes a backup manifest file, followed by the application APK, if any, and app data (files, databases and shared preferences). The data is compressed using the deflate algorithm, so, in theory, you should be able to decompress an unencrypted archive with standard archive utilities, but I haven't been able to fine one compatible with Java's Deflater (Update: here's how to convert to tar using OpenSSL's zlib command: dd if=mybackup.ab bs=24 skip=1|openssl zlib -d > mybackup.tar). After the backup is uncompresed you can extract it by simply using tar xvf mybackup.tar. That will produce output similar to the following:

App data is stored under the app/ directory, starting with a _manifest file, the APK (if requested) in a/, app files in f/, databases in db/ and shared preferences in sp/. The manifest contains the app's version code, the platform's version code, a flag indicating whether the archive contains the app APK and finally the app's signing certificate (called 'signature' in Android API's). The BackupManagerService uses this info when restoring an app, mostly to check whether it has been signed with the same certificate as the currently installed one. If the certificates don't match it will skip installing the APK, except for system packages which might be signed with a different (manufacturer owned) certificate on different devices. Additionally, it expects the files to be in the order shown above and restore will fail if they are out for order. For example, if the manifests states that the backup includes an APK, it will try to read and install the APK first, before restoring the app's files. This makes perfect sense -- you cannot restore files for an app you don't have installed. However BackupManagerService will not search for the APK in the archive, and if it is not right after the manifest, all other files will be skipped. Unfortunately there is no indication about this in the device GUI, it is only shown as logcat warnings. If you requested external storage backup (using the -shared option), there will also be a shared/ directory in the archive as well, containing external storage files for each shared volume (usually only shared/0/ for the first/default shared volume).

If you specified an encryption password, things get a little more interesting. It will be used to generate an AES-256 key using 10000 rounds of PBKDF2 with a randomly generated 512 bit salt. This key will be then used to encrypt a randomly generated AES-256 bit master key, that is in turn used to encrypt the actual archive data in CBC mode ("AES/CBC/PKCS5Padding" in JCE speak). A master key checksum is also calculated and saved in the backup file header. All this is fairly standard practice, but the way the checksum is calculated -- not so much. The generated raw master key is converted to a Java character array by casting each byte to char, the result is treated as a password string, and run through the PBKDF2 function to effectively generate another AES key, which is used as the checksum. Needless to say, an AES key would most probably contain quite a few bytes not mappable to printable characters, and since PKCS#5 does not specify the actual encoding of a password string, this produces implementation dependent results (more on this later). The checksum is used to verify whether the user-specified decryption password is correct before actually going ahead and decrypting the backup data: after the master key is decrypted, its checksum is calculated using the method described and compared to the checksum in the archive header. If they don't match, the specified password is considered incorrect and the restore process is aborted. Here's the header format for an encrypted archive:

Based on all this info, it should be fairly easy to write a simple utility that decrypts and decompresses Android backups, right? Porting relevant code from BackupManagerService is indeed fairly straightforward. One thing to note is that it uses SYNC_FLUSH mode for the Defalter which is only available on Java 7. Another requirement is to have the JCE unlimited strength jurisdiction policy files installed, otherwise you won't be able to use 256 bit AES keys. Running the ported code against an unencrypted archive works as expected, however trying do decrypt an archive consistently fails when checking the master key checksum. Looking into this further reveals that Android's PBKDF2 implementation, based on Bouncy Castle code, treats passwords as ASCII when converting them to a byte array. The PKCS#5 standard states that 'a password is considered to be an octet string of arbitrary length whose interpretation as a text string is unspecified', so this is not technically incorrect. However since the 'password' used when calculating the master key checksum is a randomly generated value (the AES key), it will obviously contain bytes not mappable to ASCII characters. Java SE (Oracle/Sun) seems to treat those differently (most probably as UTF-8), and thus produces a different checksum. There are two ways around this: either use a Bouncy Castle library with the Android patches applied, or implement an Android-compatible PBKDF2 function in our decryption code. Since the Android Bouncy Castle patch is quite big (more than 10,000 lines in ICS), the second option is clearly preferable. Here's how to implement it using the Bouncy Castle lower level API's:

This seems to do the trick, and we can now successfully decrypt and decompress Android backups. Extracting the files is simply a matter of using tar. Looking at the archive contents allows you to extract certain files that are not usually user accessible, including app databases and APK's without rooting your phone. While this is certainly interesting, a more useful scenario would be to restore only a part of the archive by selecting only the apps you need. You can do this by deleting the ones you don't need, repacking the archive and then using adb restore with the resulting file. There are two things to watch out for when repacking though: Android expects a particular ordering of the files, and it doesn't like directory entries in the archive. If the restore process finds a directory entry, it will silently fail, and if files are out of order, some files might be skipped even though the restore activity reports success. In short, simply tarring the unpacked backup directory won't work, so make sure you specify the files to include in the proper order by creating a backup file list and passing to tar with the -T option. The easiest way to create one is to run tar tvf against the decompresed and decrypted original backup. Once you create a proper tar file, you can pack it with the provided utility and feed it to adb restore. Another thing you should be aware of is that if your device is encrypted, you need to specify the same encryption password when packing the archive. Otherwise the restore will silently fail (again, error messages are only output to logcat). Here's how to pack the archive using the provided shell script:

$ ./abe.sh pack repacked.tar repacked.ab password

Full code for the backup pack/unpack utility is on github. Keep in mind that while this code works, it has very minimal error checking and might not cover all possible backup formats. If it fails for some reason, expect a raw stack trace rather than a friendly message. Most of this code comes straight from Android's BackupManagerService.java with (intentionally) minor modifications. If you find an error, feel free to fork it and send me a pull request with the fix.

I've been hit with the bug where I thought I had a full backup but so far haven't been able to get anything terribly useful from it. I've tried the Java code, which seems to properly extract the system partition before I get a ZipException, but the SD card portion I haven't had any luck getting. I've also given the perl scripts a go but they don't seem to work for me either. The stacktrace from your Java app is: Exception in thread "main" java.lang.RuntimeException: java.util.zip.ZipException: invalid stored block lengths at org.nick.abe.AndroidBackup.extractAsTar(AndroidBackup.java:181) at org.nick.abe.Main.main(Main.java:35)Caused by: java.util.zip.ZipException: invalid stored block lengths at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:147) at java.io.FilterInputStream.read(FilterInputStream.java:90) at org.nick.abe.AndroidBackup.extractAsTar(AndroidBackup.java:160)(Line numbers might be slightly different as I attempted to do a bit of troubleshooting on the code. )My guesses for the error: my data file's size - it's about 10GB; on the xda post about the perl utility, there's some talk about the first part of the file actually being compressed but the second part isn't - perhaps that's the case for me. When the Java app dies, the file that is created does seem to be a valid tar other than it doesn't contain all the data.

So far have tried the Java utility on my mac and the perl scripts on my mac and a Ubuntu 8.04 box.

Sounds like you've hit the external/shared storage bug indeed. Unfortunately the extractor tool doesn't handle this automatically. What you can do is (carefully) extract the shared storage part from the archive with a binary editor, and that should leave you with a valid backup for /data and /system. Then you should be able to use the Perl script to convert the extracted part to a tar file, so that you can restore /sdcard. I think the XDA post has some hints about this.

If you meant what the openssl, etc. standard command is, there isn't a direct one. You have to read the header and do some processing to derive the decryption password. It could probably be done in a shell script using some combination of sed, grep, awk and openssl, but it won't be pretty. If you don't want to use Java, check out the Perl scripts on XDA, or use your favorite scripting language to port the decryption code.

Not but, I haven't used the --delete option either. In any case, you need to delete not only the apk, and preserve the file order to able to restore. What works for me is unpacking, deleting the file and re-packing. See the last part of the article.

The Java code is using a much larger buffer and compression speed should be comparable to zlib's (I think Java SE uses native code for this), so in general it should perform better for larger backups (I haven't done an actual benchmark though).

I struggled getting the restore to work properly and after many BackupManager exceptions, the procedure below allowed me to backup an application and data, modify files in that backup and restore it successfully. (Of special importance is the way the TAR archive is recreated after adding/replacing/removing files.)

1. Backup application and data on the source:

adb -d backup -f filename.ab -apk javapackagename

2. Before restoring the application, remove it from the target.

adb uninstall javapackagename

3. If you want to modify the backup contents use abe.jar as follows to extract the backup file to

a TAR file:

java -jar abe.jar unpack filename.ab filename.tar

4. Extract the TAR file to a place of your choice. In this example this is the current directory:

tar -xvf filename.tar

5. Now replace any files you want to replace in the apps folder created. For example, I replaced the database files in apps/javapackagename/db. (filename and filename-journal)

6. Create a new TAR file containing the files. Note how each file is added individually AND notice that the apps directory is not preceeded by ./ but is just using the directory name.

I'm getting the error as in Dakota's second post. Running Kubuntu 12.04 with adb 1.0.31 for Linux. Downgraded from Android 4.4.2 to 4.3 on Nexus 7 2012. Trying to restore from adb or Ti Backup does not work.

You should contact Helium support to find out if the backup files it produces are in the same format as native Android backups. You can also look at the header at the file to see how if it looks the same as what is described in this post.

I have a Nexus 5, I've just updated to 4.4.3 and the adb backup version was changed to 2! The backup extractor prints the message "Don't know how to process version 2" Any chance for you to update your extractor please?

Yes, I saw that change. I will update the program to accept ver 2 when I get a chance. For now, you can simply change the version to 1 manually (as you did), it should work even if the backup is encrypted, because the program tries both algorithms if the first try fails. Or, if not encrypted, use the OpenSSL method, it simply skips the header, so the version doesn't matter.

This is not the latest version. The latest version handles backups with ver=2, but you have to build it from source. Why do you want to create an uncompressed backup with adb? There is no such option, so it can't be done without modifying the Android build (ROM).

This refers to Android source code (BackupManagerService.java). It simply means that the tar file inside the backup is not compressed. Helium is probably changing something at runtime or they have their own implementation of the BackupService. Not idea why they would turn off compression though. You can't achieve the same thing without writing code and messing with Android internals.

Unfortunately am in the same position after using adb to take a "full backup" of my nexus 4 before proceeding to wipe/unlock.

Have scoured the XDA forums but only post I found with hints to recover backups with the bug was thishttp://forum.xda-developers.com/showpost.php?p=29072968&postcount=4tarfix.pl doesn't work for me, and I don't have the nous to be able to follow the high level instructions by binaryhero

Any chance you have some pointers over how exactly to extract the shared storage part from the archive with a binary editor?

Unfortunately am in the same position after using adb to take a "full backup" of my nexus 4 before proceeding to wipe/unlock.

Have scoured the XDA forums but only post I found with hints to recover backups with the bug was thishttp://forum.xda-developers.com/showpost.php?p=29072968&postcount=4tarfix.pl doesn't work for me, and I don't have the nous to be able to follow the high level instructions by binaryhero

Any chance you have some pointers over how exactly to extract the shared storage part from the archive with a binary editor?

1. After modifying/editing/deleting files from an extracted android backup using the method in your readme file, do I have to update the manifest file before packing and restoring the modified backup to my device? If yes, how do I modify or update the manifest file? As I saw, it just contains some numbers. What are those numbers?

2. For apps that disallow backups, how do I change the "allowBackup" in the manifest so that I can backup using adb again? Where is the manifest file in the apk? Is it possible to modify this?

You can find detailed guides about repacking backups on XDA, etc., but it's generally tricky because order and format needs to be preserved. IIRC, it should work even if you don't modify the manifest, but do test on a recent Android version.

You cannot change allowBackup unless you have root, but once you do, you can access data directly anyway.

I searched everywhere. Repacking is done using the same method as I did by creating a list of the tar file, editing the list file without changing order and converting it back to ab using star or pax. But there is no information anywhere about the manifest file. What do I do with it? I just want to know the way to update the manifest file. I want to know even if backups work without updating the manifest.

What is IIRC?

By the way, the restore worked with the above method on lollipop. I just want to know how to update the manifest. Please tell me.

I used apktool to unpack an apk and it was converted to readable format. I can edit the androidmanifest.xml now. I can change the allowBackup to true. So, my question is, will the app work with adb if I install the modified apk? I haven't got a device to test all these. So I'm asking. :) Please answer. Thank you. :)

The manifest format is as described in the blog post. If you want to mess around with Android internals, you need to do some work yourself, you won't find guides for everything. Being young actually makes this much easier :)

Thank you. I didn't need a guide, just a hint 'cause I don't know anything technical. I'll use my intelligence to experiment with it. I've to find out how to find those values now, especially the signature part. I have to find out how to obtain the hash of the key so that the manifest can be modified if necessary. If you already know it, please share it. :) Thanks.

Hello, I'm getting this error when I try to pack with star:---------------------------star.exe - Application Error---------------------------The application was unable to start correctly (0xc000007b). Click OK to close the application. ---------------------------OK ---------------------------

Trying to unpack a Sony Z2 encrypted backup. Any idea why it fails and what to do?This backup is encrypted, please provide the passwordPassword:Exception in thread "main" java.lang.RuntimeException: java.security.InvalidKeyException: Illegal key size at org.nick.abe.AndroidBackup.extractAsTar(AndroidBackup.java:203) at org.nick.abe.Main.main(Main.java:35)Caused by: java.security.InvalidKeyException: Illegal key size at javax.crypto.Cipher.checkCryptoPerm(Cipher.java:1039) at javax.crypto.Cipher.implInit(Cipher.java:805) at javax.crypto.Cipher.chooseProvider(Cipher.java:864) at javax.crypto.Cipher.init(Cipher.java:1396) at javax.crypto.Cipher.init(Cipher.java:1327) at org.nick.abe.AndroidBackup.extractAsTar(AndroidBackup.java:118) ... 1 more

Using tar -T ended up including the directories, so that every file was in the tar archive twice - this is because the original listing made with tar -t included both directories and files. Fix was to manually delete the files from the listing (tar -tv | grep '^d' will show just the directories).

Then for some reason just using GNU tar (1.27.1 from Debian) didn't seem to create archives that my phone (Moto G 4G XT1039 Android 4.4.4) was willing to restore, so I used the FreeBSD tar command instead and that worked (from the Debian bsdtar package). The final incantation was:

I don't think it was necessary to force-stop the application in question (the purpose of the exercise was to edit it's database so that I could move a bunch of files from internal to sdcard storage), BICBW...

Popular posts from this blog

One of the new features Android M introduces is adoptable storage. This feature allows external storage devices such as SD cards or USB drives to be 'adopted' and used in the same manner as internal storage. What this means in practice is that both apps and their private data can be moved to the adopted storage device. In other words, this is another take on everyone's (except for widget authors...) favorite 2010 feature -- AppsOnSD. There are, of course, a few differences, the major one being that while AppsOnSD (just like app Android 4.1 app encryption) creates per-app encrypted containers, adoptable storage encrypts the whole device. This short post will look at how adoptable storage encryption is implemented, and show how to decrypt and use adopted drives on any Linux machine.Adopting an USB driveIn order to enable adoptable storage for devices connected via USB you need to execute the following command in the Android shell (presumably, this is not needed if your devic…

It seems we somehow managed to let two months slip by without a single post. Time to get back on track, and the recently unveiled Android maintenance release provides a nice opportunity to jump start things. Official release notes for Android 4.2.2 don't seem to be available at this time, but it made its way into AOSP quite promptly, so you can easily compile your own changelog based on git log messages. Or, you can simply check the now traditional one over at Funky Android. As you can see, there are quite a few changes, and if you want a higher level overview your time would probably be better spent reading some of the related posts by the usual suspects. Deviating from our usually somewhat obscure topics, we will focus on a new security feature that is quite visible and has received a fair bit of attention already. It was even introduced on the official Android Developers Blog, fortunately for us only in brief. As usual, we like to dig a little deeper, so if you are interested i…