Recently I gave Rachel, one of my daughters, a collection of digital documents covering her teenage years and some of her childhood. The collection contains PDF’s, some saved HTML pages, WMV and MOV video’s, a few audio recordings in MP3 format and thousands of digital JPEG photos.

It’s a slice of a growing collection, a collection that encompasses the digitized memories of My Life. Thoughts, songs, clips, snapshots, links.

It’s a collection started in 1997 but by now containing items from long before that time; digitized photographs and video of my childhood and teen years, songs from back then, etc.

As time passed and the collection grew two main challenges emerged:

how do I make sure these items make it to my children?

where or how do they get the information about the items?

Getting Digital Assets to the Next Generation

Although seemingly the simplest problem to overcome, this has turned out the be the most laboursome and worrisome aspect of digitizing memories.

The problems are media and file format.

Media

To put it simply: all digital media is fleetingly temporary into the extreme.

When you decide to keep your memories in digital format, and for many of us that decision is made by the consumer equipment we choose, you agree to keep your files “in the air” as if you’re a juggler.

I have DVD backups from 3-4 years ago which cannot be read. I have CDR’s which cannot be read. In a few cases even recent media cannot be read but on the very device which I created them with.

The only sure way to guarantee your files will be there come next year let alone the next generation is to keep them in a constant “alive” state.

My files, the Golden Copy, are stored on a hard disk which is mirrored to another hard disk.[1]

Hard disks are cheap. Exchange the backup drive for a new one at least every 2 years; delegate the previous one to your current main drive or keep it as an additional copy.

As long as you have a somewhat manageable amount of data, use online backup as well. I prefer Carbonite, having been with them since 2006. I can download about 1 GB/hour: if push comes to shove I can restore my digital photo collection in 2-3 days.

At one point though there’s a tipping point either because your ISP would never let you download so much so fast or because downloading it would take the better part of a season…

You realize that you have so many digital memories since so long that you can’t afford not to consider serious disasters in which everything you consider as your current setup is destroyed: flooding, explosions, fire, theft, vandalism, etc. etc.

Once at that point you will want to start making copies/backups on an external drive which you’ll store in a safety deposit box at the bank or in a media fire proof safe at your own place.[2]

The take-away is that at no time should you feel your information is solid and permanently stored. Instead, think of it as very ethereal.

File Formats

So far this is the one that bugs me the least. Time has actually done a lot to reduce worry for my famliy’s version of the digital dark age.

There are caveats though.

Most if not all of the files I use are not only current but long time standards: QuickTime, JPEG, PDF, HTML, and of course plain text. Other data of interest might be contained in email.[3]

It’s my experience up till now that hugely popular file formats like this have tons of converter applications available. If down the line a format such as PDF would start to fade away you’ll have ample time to convert those documents to something else.

File Applications

A bigger problem.

As soon as we start to store information on the application level through integration with a database, we risk data loss through obsolescence of the application.

It’s not that the final storage place becomes obsolete but rather that it starts to demand that the next generation are geeks with intimate knowledge of how this application stored information where and how — and how to get that information out.

A good example is Evernote[4] which when installed as a local application stores its data in a SQLite database. For geeks it’s trivial to 1) find this out, 2) realize you need a SQL browser of some sort, and finally 3) to find such a SQL browser without being ripped off and to effectively use it to export the information.

Are you absolutely sure your children will be such geeks?

It becomes necessary to export to a more standard file format or to prepare for it — and to possibly prepare such export paths and detailed information. More on that, detailed information, later on.

Sharing the Digital Memories

The difference between the digital asset and the digital memory is the one between data (the file) and meta-data (information about the file and what it contains or represents).

A good example to work with are photos.

Think back to your grand parents’ photo albums or, if you’re about my age, your parents’ albums. Under, on or at the back of photos information was scribbled about the photo: a date, event, maybe names of people in the photo.

But the former, the scribbled data, is simply not there: we don’t write or draw on our digital photos.

To carry information about what is portrayed, shown, described or sung over to the next generation we have to find and use ways to provide such meta-data.

“Truth In The File”

The best way to add meta-data is through the file and in the file itself.

Scott Dart, program manager for Microsoft Windows Live, referred to it as “truth in the file”. I like that description.

The first very basic file-level meta-data is the filename.

Most of my file names are in the format yyyy-mm-dd keyword keyword keyword.file-extension.[5] This format not only allows for correct sorting in Windows in a sort of timeline flow but also helps ensure that the date stamp is carried over: file timestamps will change as backups are restored…

The above format can just as easily be applied to a photo (“2007-12-24 christmas eve.jpg”) as to a PDF (“2001-02-01 drawing audrey for ruud.pdf”), for example.

In contrast to the file name, the second layer of file-level meta-data is not immediately visible. This is data that can be embedded in the file.

As this information is not immediately visible and only accessible at the application level it’s again necessary to prepare detailed information about the existence of the meta-data and how to access it and use it.

Your Digital Will

This is where your digital will comes in.

A digital will contains all information the people you would leave behind will need to finalize any account activity and access and use any kind of information you purposely want to leave behind for them.

Let’s only look at the type of information that’s relevant to what we’re talking about; digital memories.

What should you detail?

You should explain that there is a digital collection, that there are digital memories. As this post has done, you too should instruct the next person on how to keep that collection intact and alive.

You have to detail how to access it. Where are the photos? Where are the videos? Where are the scanned documents?

Suggest applications to access these files. Possibly prepare a folder with the installers of such applications ready for them.

Explain about the meta-data and how to access and use it.

List the applications you have used to build this archive of digital memories. Detail what you did with those applications and why. If you’ve ever ran into a problem with those applications or the files they work on and were able to fix or work around it, detail that here.

Best Practices & Tips

Make things as simple as possible.

Store all your digital memory files together in one folder (with subfolders if you want). It’s simpler to backup, simpler to instruct about (“take this folder and that’s it”). Using My Pictures, My Videos and My Whatnot starts to scatter things around in a folder (My Documents) which will soon enough fill up with files and folders unrelated to your digital memories.

Be wary of applications that “eat” your data/files. File recovery software is one thing; getting files out of some sort of proprietary file format or database is a whole different ball game, one nobody should want to play.

The above pertains to web services as well. Unless you use them as a sort of backup or “also” storage, don’t rely on them. Most people who were online in 1997 can name a whole lists of (online) brands that seemed to never ever go away which simply don’t exist anymore today.[6]

If you do want to use an additional layer, an additional application, use one that works with your files in a non-destructive manner. Personal Brain is a good example: files dropped in it are stored as regular files in regular folders: if the application ever fails it’s somewhat trivial to search through the folders on the disk and access your files.[7]

Two is better than one: I don’t rely on one image application to handle our photo collection, for example. I use both Adobe Photoshop Elements and Picasa.

I think of it as a poor man’s RAID. Use shadowing software such as NTI Shadow or sync software like GoodSync [↩]

Sentry Group has dedicated data protection safes with hard disk inside a safe and what not. Costly but as time progresses and more and more people have this problem, I expect more solutions to come on the market. [↩]

mbox and PST although I plan to do a huge export to EML and/or txt [↩]

I use Evernote quite regularly to add some diary-type information. The application will be part of my digital will. Personal Brain is better as it stores notes as standalone HTML files but here too one can’t rely on the next generation “getting” the application. [↩]

You can’t fully rely on these long file names however. I have a number of files from the early years which at one point had short DOS 8.1 file names due to a problem with the CDR backup [↩]

16 Responses to “Creating & Keeping Persistent Digital Memories”

Thanks for this thoughtful post. Oddly enough, I was just mulling all this over last night and then got to read this! I’ve seen a few attempts at digital safety deposit boxes, and I wonder if that might solve the problem of continual file format updates for non-geeks (like me)

I wouldn’t rely on a digital safety deposit box or not for long. 1) the amount of data will grow very fast, especially if you’re also storing video. 2) you really need to have control over your own data; you don’t want to lose stuff to a recession…

Most file format will remain readable or convertible in one way or another. The problem down the line is one for the next generation; are they geeky enough to get it done?

But the first step is ensuring the information remains available. Either stup a poor man’s raid or get something like a Western Digital MyBook (confirm that it’s one of the models that does RAID)

Excellent post.. I just added a 1.5TB RAID 5+ (<$350) here at home primarily for this reason.. I also use online backups, but as you said, transfer times are pretty sad here in the US.. That leaves losing everything to fire or flood..

@fantomaster Much appreciated indeed. WordStar, oh does that bring back some memories.

At times when conserving these digital items I remember my father who would take reel-to-reel tapes, copy them to compact cassette and finally to DAT. At least the degradation in quality is something we can often avoid nowadays.

Hi Ruud, I just found you via Twitter :-). Thanks for this post, it raises a lot of good points. Because its so in-depth Im surprised though that you didnt mention flash drives? To me thats the absolute future of storage and backup, in a mobile as well as a secure way (think of encrypted or bulletproof flash drives). Also I dont see these sticks become obsolete for a long time (USB 3.0 is around the corner), as a matter of fact they’re already in the process of replacing DVD-Rs and (external) hard drives. With data retentions of more than 10 years they definitely fare better than most media. Ok – now I feel better and yeah, I guess you can call me biased. ;-)

I didn’t and don’t include flash/thumb/usb drives because they’re not reliable enough, yet. Flash memory has a limited amount of read/write cycles in them and unlike with hard disks, there’s no warning at all when it will fail.

They’re not cost efficient yet either. A 1 TB drive is about 100 dollar, give or take a few bucks. That’s hard to beat. Hook up a second that you mirror automatically to and you’re set.

Ruud this is brilliant advice. Just read the ‘Google Reader + Twitter HashTags’ post on SEP/Sphinn. Funny it made me go ‘hmmmmmm wish there was an easy way to do everything like that’ and then… Here you are with another great set of possibilities for more data saving.

To quote Fantomaster ‘I bookmarked it’ :) I even making my kids read it, then save it to. From 12.5″ hardfloppy til now I have lost more than I think I ever learned. Awesome ideas man.

Hi Ruud – this is such an excellent post.. thank you for writing it. I worry so much about losing my data which is scattered all over the place, specially after I lost quite a bit of documents/pictures/files from yahoo briefcase recently.. what a heartache that was! I’ll be referring to this article time and again for tips, so bookmarked it! (of course bookmarked on more than one place – x-marks as well as on delicious ;))

Unsurprisingly, I recently thought about leaving my digital legacy, too. I didn’t find a better solution, than at least a simple RAID. Just two HDDs might work to account for non-tech savvy people (though I believe, should one want to read my files without me, he’ll find a tech savvy person, right?).

Then again, if in 70 years HDDs and RAID become what the shot-through paper cards are today, is there be a guarantee for that? “RAID, – oh, it’s something our 80 year old pal might have heard of.”

Perhaps, it’d make sense to teach ones offsprings to backup and convert the data themselves, so they know how to get it and to recover it? Instead of hoarding your data yourself right now, share it now and you’ll be relieved of the burden to make instructions and worry that they won’t find your data (at least partly). I see you’ve already started going in that direction, so it’s all good :)

Thanks for writing this up, Ruud. I couldn’t wrap my head around this at the first attempt: probably because I don’t have that much digital data to delegate.

When it comes to sharing, I think it’d just make sense to come over or have them come over to you, so they would backup your data themselves to an external HDD and carry it with them.

Yes, it isn’t always fresh, but at least they’d have all the recent and digitized photos and they even wouldn’t have to restore your data, unless *their* HDD dies. The responsibility of converting the data would lie on them, too. Then again, if they are young, they might not bother enough until its too late.

So yeah, I guess it doesn’t replace your triple strength backup process, but I assume it might ease the restoration process considerably.

Wow another great post Rudd with tons of great information. I too have been dealing with this same dilemma. My problems started when I switched from a G4 Mac to the new iMac not only did I loose all applications but older files that could not be read on the new iMac. I have old cd’s that don’t work, old hard drives that cannot be read. CD burners that are useless to me since the application won’t work on my new Computer.

I wonder what do people do with all this information, & stuff? If you cannot recover it do you throw it all away? I mean what use is it to anyone? What I have started doing is saving as much as I can to different servers I have. I don’t trust any attachable device nor do I want to load up on DVD’s that may or may not work in a few years. Plus it takes up so much space regardless of whether they are small or not.

I wonder how long before technology can offer one device where everything we want to save can be saved. The kind of device you can James Bond out of a pair of glasses or maybe one cool gadget I wonder… Will I be alive to see it. ☺