Back Up Your Stuff

My Twitter-friend Chelsea may have lost a truly heartbreaking amount of work when a powercut somehow wiped her hard drive. She and probably anyone following the awful saga have resolved to be more zealous about backing stuff up online, so I thought I’d do a post about what I use and what I think of it.

Dropbox

Dropbox runs in your system tray and keeps everything in your ‘Dropbox’ folder synced, both to their servers and to any other computer you install it on. Since I use a desktop and a laptop regularly, right away that gives me one remote and two local copies of all my most important stuff. I use my Dropbox folder as a ‘My Documents’, generally: it’s all things I’ve created and would hate to lose: docs, photos, and my game.

I pay $100 a year for 1TB of space, plus $40 a year for an add-on that gives unlimited version history: every version of everything I’ve ever put in there is stored forever, even if I delete or over-write it.

Limitations:

It’s not good for syncing stuff that needs to stay in the folder it’s in, like a savegame folder. There’s a third party hack to trick it into doing that, but ‘third party hack’ and ‘my most precious data’ don’t really feel like good partners.

If you regularly dump huge amounts of stuff in there, it’s a little irritating that you can’t change the order in which things are backed up. One 2GB file has changed? It’ll keep trying to upload that and not back up any of the new changes to smaller files until it’s done. Hasn’t been a problem for me since I first set it up.

If you regularly delete thousands of files (I do!), it’s bizarre that it takes as long to upload or download that change as it would if you’d added that many files.

You can get your iPhone to back up your photos to it, but they’ve recently split that functionality off into a separate app called Carousel, which is fucking horrible. It only syncs if you run it, then access your photos while it’s running.

It’s not good for backing up Unity games. Unity seems to lock certain files in a way that prevents Dropbox from backing them up, and as mentioned earlier, it’ll keep trying that same file forever, ignoring all other changes in the meantime.

The version history thing is reassuring, but if I ever lost or over-wrote a folder of thousands of files, I can’t see a way to batch-restore them – you have to click on each file individually and select the version you want to roll it back to. For my game, that’d probably take longer than re-making it.

CrashPlan

CrashPlan runs in your system tray and automatically backs up a number of folders you’ve told it to, encrypted, to any and all locations you select. I use it for absolutely everything I value outside of Dropbox – and Dropbox itself, just to be sure.

The free version will back up to a local or external hard drive, or even to a friend’s computer, so you can do a kind of back-up swap (it’s encrypted).

I pay $60 a year to back up to their servers. There’s no space limit, and I’m using over 300 gigabytes – it took a week to get it all up there when I first installed. Now it just runs silently in the background and I never notice it. It’ll back up all changes right away if you like, or you can tell it to only do it when the computer’s idle.

Limitations:

As far as I know you can’t use it for passively keeping stuff in sync across multiple computers.

I thought it didn’t have versioning but it turns out it does. Gosh, unless you really need syncing you should use this instead of Dropbox.

Version Control

Version control is a truly vital concept that has unfortunately been implemented by madmen. It’s basically “keep every version of my project (usually online)” but with the ability to ‘branch’ out from a version and then merge those changes back in later, which is particularly useful for teams.

I use GitHub for my Unity projects, following this guide, but I find Git itself baffling and mad, even after taking real pains to learn it. More than once it’s told me I’m not allowed to save my work, and must over-write it with the outdated online copy, and I have to go crying to Twitter to find someone who can tell me what buttons to press to let me actually save. I’ve never actually lost work to it, so I’ll keep soldiering on, but I have a hard time claiming this is a sane way to back up your stuff if you work alone.

Limitations:

Fucking mad.

Game Maker Studio has a different version control system built in, Subversion. I tried setting it up once, using a test project, and tested deleting something and rolling back to the old version. I could not, the thing was gone forever. That concludes the past, present and future of my relationship with Subversion for Game Maker.

If you’re using git and think that it was designed by insane people, I highly recommend trying out TortoiseGit. It’s the only git client that I’ve ever actually liked (haven’t tried the mac options, though), and while it’s got a ton of crazy options (because git has a ton of crazy options), it’s sensible enough by default that I feel like I could show a new team member how to use it pretty quickly. Generally you can just right click a file to perform an action, and they do a good job of reminding you what part of the git process you’re supposed to be doing, like including a “push” shortcut inside the “commit successful” window.

Also whoever told all the version control designers that they all had to make up their own vocab terms for the same actions was clearly an evil genius with some kind of nefarious scheme in mind

If sync alone is part of your needs, Bittorrent Sync is cool. It’s what it says on the tin, essentially a continuously updated torrent of some location that all your computers download and upload to with all the concurrency and graceful failure benefits of Bittorrent. Nobody can connect to it without sharing the key file, and your data doesn’t go through anyone’s servers.

I’m aware from Twitter over time that you’ve got a VCSless workflow that you’re happy with. I will nevertheless reiterate that git isn’t really a VCS; it’s a toolset that Linus Torvalds and his fellow kernel monsters can use to create a VCS for any conceivable project size and ecosystem of trust levels and policies. If you ever try again I’d go Mercurial (Hg). It throws out support for nutso edge cases in order to have sensible defaults so you can start with Just Check My Blasted Stuff In With A Note Right Now. It too has a GUI by the Tortoise folks, and the most popular GitHubalike for it is Bitbucket.

It creeps me out how people feel comfortable uploading their “most important stuff” to some company unencrypted.

Step 1 should always simply be another external harddisk. Step 2 is a microsd card or usb stick that you carry around on person with your most important stuff (encrypted of course). This will keep you safe from most data disasters. Step 3 maybe a harddisk at a family/friend’s place that you update every now and then, just so there is a truly external backup.

Git was designed by madmen. It’s great for everything it cares about. Github’s desktop app is designed to put a happy fisher-price face on a massive over-complicated system, and that works until its visual metaphors don’t fit anymore, like when your upstream-copy is more up to date and you need to merge it back.

TortoiseGIT – as mentioned above – is a way around this, it doesn’t hide the complexity, but on the other hand it doesn’t hide the complexity.

As for all the other things, I wish to lobby for every game developer in the universe to follow the OS standard and put saved games in Windows’ Saved Games location, instead of somewhere deep in Steam (Dreamfall, I’m looking at you) Your own directory in My Documents (Bioware) or inside a badly named directory, inside a hidden folder, in the root directory of my user account, behind a sign saying Beware of the Leopard (Minecraft).

– For any creative work (programming, visual design, video editing, etc.), learn how to use version control. You don’t need to understand 100% of it; if you’re working alone, you only need to understand how to commit your work, and how to view and revert to past versions of your work. It’s not hard. Versioning only gets hard if you’re getting into team work, branching, merging, etc, which you don’t need if you work alone.

– Dropbox is expensive and does not encrypt your data, but works well. Alternatives that *do* encrypt your data locally are SpiderOak and Wuala, so if you’re not an American, or care about the privacy of your data (might be particularly relevant if you’re a freelancer uploading your client’s data to a cloud system), these are good alternatives.

– Services like CrashPlan are good, but it might be an even better idea to divorce your data storage from your backup system, so you’re not dependent on a single provider. For example, if you install Arq, you can then pick one (or more than one) data storage provider (Amazon, OneDrive, Dropbox, etc). If another provider offers a cheaper option, you can switch at any time, without changing your backup system. Arq also encrypts your data locally, so there’s no chance of anyone getting access to it.

– Local backup. Cloud backup is incredibly useful, but if your computer dies, you don’t want to have to download 400 GB of data just to get your computer back into a working state. Buy a NAS and back your data up to it (the aforementioned Arq allows you to use storage systems in your local network as backup targets).

– A bootable disk. This is super important if you want to minimize your downtime. On a Mac, use something like SuperDuper or CarbonCopyCloner to clone your computer’s internal disk to an external disk once a day, in the evening, before shutting your computer off. That way, if your internal disk dies, you can boot from the external backup, and immediately be back up and running, never losing more than a day’s work (which you can get back from your other backup system). Then, when you have time, replace the internal disk, and clone the data back to it.

I’d like to throw my hat in the ring and mention OneDrive. You get 15 Gb for free when you sign up for a Microsoft Account and you get unlimited (I’m at 10+ TB now) storage if you buy Office365 for USD 69.99 per year (which of course also includes the Office suite of applications).

It works very much like Dropbox and data is completely encrypted, both in transit and when stored. The client is a bit clumsy on Windows 7 and 8 but with Windows 10 they’ve changed how it works.

I store all my Visual Studio projects in a folder in OneDrive and as soon as I change something it syncs to the cloud. Still not version control though, just a backup.

I’m trying Google Drive for a while now and it works pretty much like dropbox so far.

The reason I stopped using Dropbox will sound really stupid but it does bother me. I was happily using it to back my stuff up and I never needed more space than what you get for free. I used it for documents and stuff that doesn’t take too much space. But then it started gifting me more space and then some more and more and more… and it’s all temporary! stuff like 100GB for free during 6 months because I liked DP on facebook, or whatever. I have no idea what happens if I accidentally use more space than what I have on my free plan and I run out of time on these extra spaces. It’s probably safe to assume no data will be lost, just not accessible, but I still don’t like it.

Call it nitpicking but I would prefer having the option to not accept these gifts, and not having that made me switch to a different service.

If you’re willing to give Git another go then can I recommend looking at git-flow? Simplifies the workflow somewhat with good concepts for branching features / releases / etc. Easy enough from the command line, but SourceTree is also built to handle it if you’re keen on sticking with a UI.

Crashplan sounds a lot like Backblaze, which costs the same ($5 a month.) Backblaze doesn’t do syncing, as best I know, but files can be retrieved through their apps and stuff, just not as simple as dropbox.

git/mercurial are far too complicated unless you’re doing a lot of branching/merging. As a coder I find them just about usable. Trying to explain them to non-coders who may not even be familiar with the idea of version control was too nighmarish for words.

Fell back on Subversion which really isn’t that bad if you use it through TortoiseSVN. I suspect the problem with the Gamemaker integration is at the Gamemaker end, as I’ve been keeping my Gamemaker projects in SVN on assembla.com for years now without any problems (just import the whole game.gmx folder).

Also, version control and backup are very different things.

Backup should be taking timed snapshots regularly, whereas Version Control should be taking named snapshots only when you’ve achieved something significant, otherwise you’ll get version controlled copies of your game in the broken state it’ll be in 99% of the time.