rebuild_db

Overview

This little program enables users of older iPod shuffle models to finally get rid of all that iTunes or other complicated playlist management stuff. Due to the simple structure of the shuffle (compared to the »big« iPods), it is possible to use the player almost like any other USB flash MP3 player: You simply copy MP3 files onto it. You only need to run the rebuild_db program after you added or removed files from the iPod. This approach hasnumerous benefits:

You don't need iTunes, ml_ipod or GNUpod anymore. Simply copy some MP3 files into the iPod volume and start this teensy 8k program.

You aren't restricted to store your music in the /iPod_Control/Music subdirectory. Build your own directory and filename structure. Your iPod, your rules!

You may use the iPod on as many computers as you want.

The actual iTunes database is left untouched, your iPod should still work with iTunes, if you like. (Note that this wasn't true for me; my iTunes installation now refuses any co-operation with my iPod. But I'm pretty sure that this is due to my experimentation with the iTunes database, and not a side-effect of using rebuild_db.)

If you do not use the iPod_Control folder for your music, iTunes will not delete your files again without asking.

Supported Devices

Only the first two generations of iPod shuffles are supported. Newer iPod shuffles (third generation and up, i.e. everything from March 2009 on) with »VoiceOver« do not work.

Supported Platforms

rebuild_db comes in two flavors: A platform-independent python script, and a plain C Win32 console application.

The script requires some recent version of Python 2.x and should work on any platform Python supports. This includes Linux, Windows, MacOS X and a large amount of other Unix-like systems.

The Win32 version is made for Windows users (obviously), and requires no additional installed software to work. Please note that it is not supported any longer and lacks many features. However, you may continue to use it if you don't need all the cool stuff like smart shuffle or custom rules.

Usage

Windows users should install Python 2.x first, because (as stated above) the Windows native version is now unsupported. Make sure to get a 2.x version, because 3.x is not backwards compatible! Users of other operating systems (including Mac OS X) are likely to have Python installed by default.

First, initialize your iPod with iTunes (that is, give it a name, and enable its usage as a mass storage device). This is needed because the iPod shuffle, despite not really using the iTunes database, is checking for its presence. You do not need to copy any tracks to the iPod; just make sure that a hidden subdirectory with the name »iPod_Control« appears in the root directory of the iPod volume. You may even delete the iPod_Control/Music directory afterwards. If this does not work for you (e.g. because you don't have access to a computer that runs iTunes), you may also try this dump of my own iPod_Control directory.

Second, put your favorite MP3 files onto the iPod. Give them the names you like and arrange them in directories as you like. Only make sure that the names are not too long. (The total path length must not exceed 256 characters.)

Third, put rebuild_db.py into the iPod's root directory and start it. A double-click should do; if not, you have to open a terminal yourself and chdir to the iPod's root directory (i.e. the drive letter on Windows or /Volumes/NameOfYourIPod on Mac OS X) and start the script manually with `python rebuild_db.py'. The program will scan the iPod volume for playable files. It also creates file called rebuild_db.log.txt that contains the program's output messages. If you don't find error messages there, everything should be OK.

Finally, disconnect your iPod shuffle. Of course, you should do this safely, e.g. using the »safe removal of devices« icon in the Windows system tray, the Finder's eject button, or umount on Linux/BSD. You may now have some fun listening to your music :) If you want to add more music later (or delete other tracks that got too boring ;), go back to step 2.

If this is too confusing for you and you are a Mac OS X user, you may want to read the following (less technical, but slightly outdated) description written by a contributor.

Useful features

Automatic directory change and logging

Availability: rebuild_db.py >= 0.6-pre1

Just after being started, the script changes its current working directory to the path from where it was started. This feature makes it possible to start the program in Mac OS X and some other Unix desktop environments by simply double-clicking the script file. The only requirement is that the script must be stored in the iPod's root directory.

In addition, the status output of the program is written to a log file. This way, you can read what happened even after the terminal output dissapeared (and it does so quite quickly :)

Both features can be disabled using command line options. Use »rebuild_db.py -h« to get a list of available options.

Sorted filenames (old-school)

Availability: rebuild_db.exe; rebuild_db.py < 0.7-pre1

Before filling the iPod database, the program sorts all filenames. This way, the playback order in normal (non-shuffled) playback mode is specified by the file and directory names on the iPod.

The sort order is lexicographical and case-insensitive, with subdirectories first. You can see the order that the program uses in the screen output or log file.

The folder »Cool Album« is put in front of »Another Cool Title«, even though C stands after A in the alphabet.

The track 10 in the album was put before track 9. This is because the sort order is lexicographic, not numeric. 10 starts with a »1«, and so it's before »9«. So, if you want to use numbers, make sure that you pad them with zeroes. In the example, »09« instead of »9« would work just fine.

Sorted filenames (new-school)

Availability: rebuild_db.py >= 0.7-pre1

In 0.7-pre1, the sort order was changed to sort files by number. The example above would now be in the following order:

Note that track 9 now precedes track 10 in the »Cool Album«. You don't need to pad numbers with zeroes any longer.

Smart shuffle

Availability: rebuild_db.py 0.3, rebuild_db.py >= 0.6-pre1

»Real« (pure random) shuffle can be quite unfair: Often songs of the same folder (read: album or musical style) are played consecutively, while other folder aren't played at all for a long period. »Smart shuffle« tries to spread the titles of each folder uniformly across the whole playlist and avoid two consecutive titles from the same folder. This greatly improves the shuffle experience ... at least I think so :)

If you don't like smart shuffle, you can use the command line option »-s« to revert to normal shuffle.

Interactive mode

Optional, Availability: rebuild_db.py >= 0.6-pre1

If started with the command line option »-i«, the program queries the user for each directory whether it should be included or not. So, you may choose to skip a directory (choose »N«), accept it (choose »Y«) or accept it with all subdirectories (choose »A«).

Custom search folders

Optional, Availability: rebuild_db.py >= 0.6-pre1

Normally, rebuild_db searches the whole iPod for playable files. If one or more directory names are specified on the command line, it only searches inside these directories. This is useful e.g. if you keep your music in a folder like »My Music«, but you also have some other folders that happen to contain MP3 files you don't want to hear. On the other hand, this can be used for cheating: If you very, very like the music of one subfolder and you want to hear it with double probability, just include this folder twice, as in »rebuild_db.py "My Music" "My Music/Favorite Music"«.

Auto-renaming

Optional, Availability: rebuild_db.py >= 1.0-rc1

Under some (yet to be discovered) circumstances, the iPod may ignore files whose names contain special characters or spaces. By specifying the -r command-line option, rebuild_db will automatically rename all playable files and all directories to (supposedly) safe names.

Custom file rules

Optional, Availability: rebuild_db.py >= 0.7-pre1

Using a file named rebuild_db.rules in the iPod's root directory, the parameters for each file or file groups can be adjusted in a quite flexible manner. The rule file is a normal text file, containing one rule on each line. Every file that is to be added into the database is checked against every rule first, in the order in which they are specified.

Some useful examples

filename~"/podcasts/*": shuffle=0, bookmark=1

All files in the directory »podcasts« will not be played in shuffle mode and will be bookmarkable. A rule like this very useful if you keep your audiobooks in a single directory.

size>10000000: bookmark=1

Any file that is bigger than 10 MB will be bookmarkable. Useful if you tend to keep very long tracks on the iPod (e.g. DJ mixes).

filename~"/stuff/*", size<1000000: ignore=1

All files in the directory »stuff« which are smaller than 1 MB will not be played at all. This can be useful if you keep a bunch of short WAV sound effects in this directory you don't want to have mixed between your music tracks.

shuffle=0: shuffle=1

If this rule is put at the end of the file, all tracks will be played in shuffle mode, even audio books.

Formal specification

A rule line has the following syntax:

var cmp ref [, var cmp ref ...] : var=value [, var=value]

where var is one of the following variables:

filenamethe current file's name (read-only)

sizethe current file's size in bytes (read-only)

ignore1 if the file shall be ignored (not added to the database), 0 if the file shall be added

shuffle1 if the file shall be part of the shuffle order, 0 if the file shall be ignored in shuffle playback mode

bookmark1 if the file shall be bookmarkable (i.e. the iPod remembers the last playback position when skipped), 0 if the file shall not be bookmarkable

reuse1 if the file's database entry should be reused from an existing entry, 0 if the file entry shall be rewritten from scratch. The default behavior is to reuse entries (value 1). This way, rebuild_db retains encryption keys for .aa files transferred with iTunes.

The ref(erence) value is compared against the variable using one of the following comparison operators (cmp):

=the rule is applied if the variable's value and the reference value are identical

<the rule is applied if the variable's value is less than the reference value

>the rule is applied if the variable's value is greater than the reference value

~The reference value is used as a filename matching pattern with wildcards. Matching is always case-insensitive.

FAQ

Q: Is support for the newer (2009 and later) models planned?
A: No, and there will never be. The database format has changed completely between the second and third generation iPod shuffles. Supporting it would amount to an (almost) full rewrite of the program. Even worse, it's not even certain that it would work at all: There's a high probability that the new database format is not manageable with rebuild_db's simple »just update one file and be done with it« approach.

Q: Is there a version for the other iPod models (classic, mini, nano)?
A: Yes, such a program exists, though it's not as simple to use as rebuild_db and is also limited to pre-2009 models. It is called rePear.

Q: What if I put some non-music files on the iPod? Does it still work?
A: There's no problem with non-music files. Unplayable files are simply ignored when rebuilding the database.

Q: Some files and directories with names containing special or native language characters are not played correctly.
A: These are actually two problems: The first one seems to be a iPod firmware bug that produces some directory name weirdness. (Some names that would work fine for files may not work for directories.) The other one is that I simply didn't figure out how to make the program work in non-Latin1 environments, sorry. You can, however, use the -r command-line option to have rebuild_db rename all files to safe names before adding them to the database. Please note that this process is not reversible. For proper support of arbitrary file names, try rePear.

Q: Why is the Windows native version so old? I need some features from the Python version!
A: Implementing these features in C is a lot harder than in Python. Not that I could not do it, I just don't want to and don't have time to. But if there's somebody out there who is willing to contribute a full-featured C version, he (or she) is welcome to join me!

Help wanted!

This program was developed on a »works for me« basis. It may also work for you (and I truly hope it does so!), but there is no warranty. If it damages the iPod databases in a way that renders your iPod useless, I'm sorry, I can't do anything for you. (Hint: It's always a good idea to have a backup of the iPod_Control directory ... :)

You can, however, help me in improving the quality and robustness of this program. If something does not work, please mail me a (ZIP compressed) copy of your /iPod_Control/iTunes directory so that I investigate the problem. Other things I'd love to hear from you include

bug reports

feature requests

processing rules that you think could be useful to other users

fan mail and PayPal donations :)

Version history

1.0-rc1 (2006-04-26)

finally(!) added the often-requested auto-rename feature (-r option)

0.7-pre1 (2005-06-09)

0.6-final was skipped because of the huge load of new (experimental) features in this version :)

rule files allow for nice and flexible fine tuning

fixed --nochdir and --nosmart bugs

numerical sorting (i.e. "track2.mp3" < "track10.mp3")

iTunesSD entries are now copied over from the old file if possible; this should preserve the keys for .aa files

0.6-pre2 (2005-05-02)

fixed file type bug (thanks to Nowhereman)

improved audio book support (thanks to Nowhereman)

files called $foo.book.$type (like example.book.mp3) are stored as audio books now

the subdirectories of /iPod_Control/Music are merged as if they were a single directory

0.6-pre1 (2005-04-23)

always starts from the directory of the executable now

output logging

generating iTunesPState and iTunesStats so that the iPod won't overwrite iTunesShuffle anymore

-> return of smart shuffle ;)

directory display order is identical to playback ordernow

command line options and help

interactive mode, configurable playback volume, directory limitation

0.5 (2005-04-15)

major code refactoring (thanks to Andre Kloss)

removed "smart shuffle" again -- the iPod deleted the file anyway :(

common errors are now reported more concisely

dot files (e.g. ".hidden_file") are now ignored while browsing the iPod

0.4 (2005-03-20)

fixed iPod crashes after playing the shuffle playlist to the end

fixed incorrect databse entries for non-MP3 files

0.3 (2005-03-18)

Python version now includes a "smart shuffle" feature

0.2 (2005-03-15)

added Python version

0.1 (2005-03-13)

initial public release, Win32 only

Downloads

Disclaimer

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.