How to Backup All Your Files to Amazon S3

Losing your favorite photos, your entire music collection, or your most important documents is something none us want to experience, yet there are still so many of us that either have an inadequate backup strategy or none at all. Ideally you should have a local backup and an off-site one; the local backup lets you get back to work quickly, but the off-site one guarantees you could restore all your files even in the worse case scenarios.

There's a number of online backup services that you could use to automatically backup your files, but sometimes building your own backup solution that you can manage on your own is the best option. The easiest way to build an off-site backup on your own is by leveraging Amazon's S3 and Glacier cloud storage solutions.

In this tutorial, I'll show you everything you need to know to start backing up your Mac or PC onto Amazon's servers, so you'll have the off-site backup security you need.

Setup an Amazon S3 Account

The first thing you’ll need is an Amazon Web Services—or AWS—account. Head over to the AWS website to get started.

Amazon Web Services Website

Click the “Get Started for Free” or “Sign Up” button to setup your account. You will have the option of signing in with an existing Amazon account or creating a new one to use with AWS. Since this is for your backups, I recommend using your existing Amazon account. One of the great benefits of being completely new to Amazon Web Services is that you get 12 months of access to their “AWS Free Tier”. This free tier includes 5GBs of storage space on S3.

Amazon Web Services Account Setup

Once you create your account and login you’ll have access to your “AWS Management Console”. This can look fairly intimidating as it’s designed for giving you tools that range from setting up simple backups to running an entire large scale web infrastructure. Don’t fret however—you’re only going to need to create some basic user accounts and containers in the S3 area.

Amazon Web Services Account Panel

You’ll first want to click on “IAM” near the bottom of the second column in your AWS Management Console. You’ll be prompted with the option to “Continue to Security Credentials” or to “Get Started with IAM Users”. You’re going to want to select the IAM Users option here.

Amazon Web Services IAM User Panel

Now that you’re on the “IAM Management Console” screen you’ll want to click the “Create New Users” button. Choose a username like s3backup that clearly shows its intent in the name. As you see in my example I’ve created three users for the purposes of creating this tutorial content as well as conducting some other tests.

You’ll want to download and save the resulting credentials that are displayed for later use. Much like a Google application specific password, these cannot be retrieved and you’ll need to generate new credentials should you lose these.

Amazon Web Services IAM User Creation

If you do everything correctly, you’ll see your newly created users listed as shown in the image below.

Amazon Web Services IAM User Listing

The next thing you’ll want to do is setup a group for your user(s) so we can apply the appropriate access permissions for our yet to be created S3 backup buckets.

Amazon Web Services IAM Groups

When prompted, choose a group name that makes sense, such as “backups”.

Amazon Web Services IAM Create New Group

The next step is to select our permissions level. You can get very granular and specific here. We’re going to choose the “Power User” level of permissions. This tutorial is assuming you’re using S3 primarily for backups. If your security requirements are more complex or you want to learn more, I suggest looking at the IAM documentation a little more closely. Here is what your permissions policy should look like.

Amazon Web Services IAM Group Permissions

The next step is to add your existing user(s) to your new group. Simply check the boxes as shown below and click “Continue”.

Amazon Web Services IAM Add Users to Group

You’ll get one final chance to review your group settings before hitting the “Continue” button once more to seal the deal. It should look a little something like this:

Amazon Web Services IAM Verify Group Settings

Now let’s head back to our dashboard and select S3. Being completely new to using S3, you’re going to be staring at a screen like the one below that is just begging you to click “Create Bucket”. Go ahead and click it; it’s ok really.

Amazon Web Services S3 Home

Now, type in a descriptive and unique name. Something like “arq-backup” will already be used, as names are global to the service and someone else will likely have used that. Something like “arq-backup-chadhs-macbookpro” is fairly descriptive and most likely not taken. Go ahead and click “Create” here after entering your name and selecting the closest data center region. If you’re a US customer, I recommend the default.

Amazon Web Services S3 Create Bucket

Things to Consider When Choosing Your Storage Type

There are a couple of things you should carefully consider when choosing your storage options for backing up to Amazon S3 or Glacier.

Do you have other backups?

Are any of your other backup sets offsite?

How much data do you have in your home folder; on your machine in total?

What is your budget for online backups?

Amazon S3 and Glacier have different costs per GB for storage and upload but the same price for restores. It’s a little more expensive to upload to Glacier but cheaper to store the data there. It’s also important to note that there is a 3–5 hour waiting period for restores from Glacier, and a small fee for downloading the full contents of your backup. It’s up to you to bring out your favorite calculator and decide what will work for you.

My quick advice is this: if you have a large amount of data that won’t change a lot, that you are backing up to recover from a catastrophe, use Glacier. If you have a lot of data that changes frequently and that you’ll want quick restore access to, go for S3. If you do the math and the monthly cost is a lot more than you are comfortable spending, you may want to consider giving a service like Backblaze or CrashPlan a try, as these types of services tend to run around $5 USD per month and offer unlimited backup space.

Whew, the Hardest Part is Done…

Ok. The hardest part is honestly done. We’re done with levers, dials, doo–dads, authentication tokens, and calculators. Now that your Amazon AWS account is all setup it’s time to backup your computer. If you’re using a Mac, just scroll down slightly. If you’re on a PC running Windows, skip down to the section on CloudBerry. If you’re using both, then simply scroll and enjoy.

Setting up Arq for Mac

Head on over to the Arq website to download and install the 30 day free trail of Arq. After completing this tutorial, the full version is well worth the 39.99 USD price tag to continue backing up your Mac. You can even backup to other cloud providers or an SFTP server if you like.

Arq Download Page

After you finish the setup process Arq will automatically prompt you to setup a destination for your backup sets. I promised this would get easier didn’t I? Go ahead and choose “Amazon Glacier and/or S3”.

Arq Backup Setup S3

Now you’ll need to refer to the AWS credentials you downloaded or otherwise documented from the first section of the tutorial. “Enter the Access Key ID” and the “Secret Access Key”. You’ll also need to need to select the region you created your bucket in from the drop-down.

Arq Backup Setup S3 Credentials

There is a handy “Create a bucket” option you can use here, but we’ve already taken care of that in the first section. Go ahead and select your bucket from the “Use existing bucket” drop-down and click “Continue”.

Arq Backup Setup Choose S3 Bucket

This next selection screen is all up to you. You can backup your entire home folder or just a selection of data. Scroll back to the “Things to consider when choosing your storage” section one more time and make sure you’re making a choice that you are comfortable with data protection and budget wise first.

Arq Default Backup Selection

For demo purposes only, I chose the “I’ll add folders myself” option.

Arq Add Folder to Backup Selection

When you make your selection you’ll be presented with a choice between “Standard storage class” and “Glacier storage class”. If you’re confused on what to choose here, please go back up to the “Things to consider when choosing your storage” section of the tutorial. Once you make your choice click “Add”.

Arq Choose Storage Type

Now you need to come up with and document a strong encryption password. This password is used to encrypt your data before sending it to S3. If you loose this password there’s no way of retrieving your data. Once you have your password documented and entered in you can click “Continue”.

Arq Set encryption Password

Pay close attention to the resulting note that pops up. You must not alter the contents of your S3 bucket(s) used for backup in any way shape or form once you begin using them for your backups.

Arq Do Not Modify Warning

From the Arq menu bar icon you can select “Back Up Now” to verify everything is working and take an initial backup.

Arq Backup Progress

You’ll know the initial backup was a success once you get to the “Finishing backup” notification.

Arq Backup Finishing

Once the backup is complete go ahead and expand the list on the left panel in Arq to explore your backup sets. Once you select your backup set you’ll be able to browse the files you have backed up on the right panel. You’ll also notice a restore button at the bottom right that you can use to restore any file.

Arq Browse Backups

From the menu bar icon choose “Validate Data and Enforce Budget Now”.

Arq Validate Backup and Enforce Budget

Validating will verify your backup integrity after making sure all your files are backed up. This isn’t something you’ll have to do all the time, but is a good idea when you’re first getting up and running to ensure things are running smoothly.

Arq Backup Validating

Now let’s set up your backup schedule and enforce a budget. Go to preferences by pressing ⌘,. Select the “Targets” tab. Then highlight your backup set and click “Edit”. Definitely spend some time on this configuration page. There is a lot of granularity when it comes to setting up when Arq will and will not backup your system, as well as ensure you will stick within your monthly budget. [1]

Arq Edit Backup Settings

Next take a look at the “Network” tab. Here you can set bandwidth throttling options to ensure your backups don’t consume all your bandwidth, preventing you from streaming video or doing other activities online.

Arq Network Preferences

Next hope on down the line to the “Email” tab. Here are some options if you’d like to be notified via email after every successful backup, or there is even a “no news is good news” approach via the “Only send email when errors occur” check box.

Arq Email Preferences

Finally, the “Advanced” tab has some important options like, “Start Arq Agent at login” and others. Explore these options further if the defaults don’t suit your needs.

Arq Advanced Preferences

That should cover setting up Arq on your Mac and most of the options within the application. If you also want to backup your PC or Boot Camp partition when running Windows on your Mac, continue on to the next section. If you’re only a fan of the Windows in your domicile then skip ahead to the next section after that.

Seting up CloudBerry for Windows

Head on over to the CloudBerry website and download and install the “Desktop Backup for Amazon S3 Cloud” product. You may need to download and install the “C++ 2010 Redistributable Package” for your system if it is not already installed or part of your Windows installation. You’ll know if this is the case as the CloudBerry install will notify you and send you to Microsoft’s site. When running the installation take all the default options and activate the free trial. The full version of the software will run you $29.99 after the free trial is over.

CloudBerry Desktop Download

Once you successfully install CloudBerry, start up the application to display the “Welcome” screen.

CloudBerry Start Screen

From here you’ll click the “Setup Backup Plan” button in the middle of the page.

CloudBerry Backup Wizard

CloudBerry has many options for backup targets. In this tutorial we’re focusing on Amazon’s cloud storage offerings. You’ll want to pick either Amazon S3 or Glacier for your storage. If you’re unsure which option to use, Scroll back to the “Things to consider when choosing your storage” section of the tutorial. For the remainder of this CloudBerry setup we’ll assume Amazon S3.

CloudBerry Backup Wizard Select Storage

On the “Amazon S3 Account” tab you’ll want to give your storage account a “Display name” that is essentially a label of your choosing. Then fill in your “Access key” and “Secret key” you created and documented in the first part of this tutorial. Lastly on this tab, you’ll need to pick the S3 bucket you created from the drop down menu.

CloudBerry AWS S3 Account Info

On the “Cost Estimates” tab you can enable the cost estimates feature and set a threshold based on cost or backup size. This can come in handy if you are setting a specific monthly budget amount for your data backup services.

CloudBerry AWS S3 Cost Estimates

On the “Virtual Disk” tab you have the option of installing a driver and mounting your S3 backup bucket to a drive letter. This could be handy for quick access to your data, however in this tutorial we’ll leave that option up to you. Once you’re finished with all three tabs, click “OK”.

CloudBerry AWS Virtual Disk

Now you will be able to select your S3 account you setup [2] and click “Next”.

CloudBerry Backup Wizard Select S3

The next step is to give your backup plan a name. The other settings on this page can be left at their defaults.

CloudBerry Backup Wizard Plan Name

Next you’ll want to select your backup mode. Unless you have a good reason not to, select “Advanced Mode” here. This will give you the option to encrypt your files client side when performing the backup for an extra layer of security. The rest of the options can remain at their defaults.

CloudBerry Backup Wizard Backup Mode

On the next page you’ll want to select your “Backup Source”. For the purposes of this tutorial I’ve only selected my “Downloads” folder. What you’ll choose depends on how this fits into your overall backup strategy. If this is your sole method of getting data offsite, you should start by at least selecting your user folder under "C:\Users" and any other places you store critical data like, music, photos, documents, etc.

CloudBerry Backup Wizard Source

On the “Advanced Filter” options page you have a number of options to consider. The default options are sensible and safe for the uninitiated. I chose to include the option “Backup empty folders” to preserve organizational structure upon restore.

CloudBerry Backup Wizard Advanced Filter

Next up is “Compression and Encryption Options”. Here you can enter a client side password for encrypting your data—do not lose this password—you need it for data recovery. The other options to consider here are to compress backups to save on storage space, whether or not to encrypt filenames in addition to data, and using “Reduced Redundancy Storage”. Using reduced redundancy storage will save you 33% on your Amazon S3 storage costs—but also as the name suggests—reduces the redundancy / protection of your files in the cloud.

CloudBerry Backup Wizard Compression Encryption

The default options are fine for the “Purge Options” section. If you are really trying to keep your backup budget trim [3], you may want to enable and tweak some of the options here.

CloudBerry Backup Wizard Purge Options

For the backup schedule I recommend “Recurring” as your go to option. This gives you similar behavior to Time Machine or Arq on the Mac. Now click “edit schedule” to tweak these settings.

CloudBerry Backup Wizard Schedule

For “Schedule Recurring Options” I recommend setting “Type” to “Daily” and the “Daily frequency” settings to “Occurs every 2 hours”. If you want a window where backups do not occur, adjust the “From” and “To” times accordingly.

CloudBerry Backup Wizard Recurring Options

The “Notification” section of the backup wizard has some interesting options. It defaults to a “no news is good news” philosophy, but can be easily changed to send an email when all backup jobs complete. You also have the option of having CloudBerry logging this to the Windows Event Log for either failures or all backup statuses.

CloudBerry Backup Wizard Notification

Finally you’ll arrive at the “Summary” screen to review all the options you’ve selected, before hitting the final “Next” button.

From the Welcome screen you’ll be able to see that your backup is currently running, and some live summary information about the backup job.

CloudBerry Backup Status on Welcome Screen

Click on the “Backup Plans” tab and you’ll be presented with a different view of your backup status with a progress indicator bar—ooh shiny!

CloudBerry Backup Status on Backup Plans Screen

When the backup has completed successfully you will see an indication on the “Backup Plans” screen with some additional stats.

CloudBerry Backup Success Confirmation

That should cover setting up CloudBerry on your PC or Windows Boot Camp partition and most of the options within the application.

Other apps and things to consider…

These are not the only tools available for using Amazon’s cloud storage for backup. I focused on the paid for top end software in this tutorial. When it comes to the importance of protecting your data, spending a little money on software shouldn’t be a non-starter. One free software alternative that runs on Windows, Linux, and OS X is Duplicati. If you like to stick to free software, take that for a spin.

Perhaps you don’t want a full single purpose backup application. If you want to interact with S3 storage like you would a traditional FTP server, you can connect to your storage with Transmit or Forklift2 on the Mac and Cyberduck on Mac and Windows.

GUIs? Excuse me Chad, but I run such an obscure Linux “distro” that you haven’t even heard of it. The only reason I even have a Mac is for iTerm2.

Does that sound like you? Then I would highly recommend you look at s3tools or duplicity. You can do some pretty cool stuff if you’re strong with the force when it comes to shell scripting or slinging some Python, and more.

Wrapping it up like a birthday present for your brain

I hope this tutorial has been helpful in guiding you through getting your Amazon AWS account set up and taking care of walking you through getting your Mac and/or PC backed up safely. For those of you curious power users out there, I hope you get a chance to take some of my other recommendations for a spin as well.

All too often backups are put on todo lists and not addressed before it’s too late. Even if you have a good scheduled backup to an external drive, I urge you to use this tutorial or another solution to get that data also safely off-site as well. In the event of a theft or fire, home owners or renters insurance will replace your hardware and backup drive, but certainly not your data.

Provided a single full backup set does not cost more than your monthly budget alone. This is detailed in the interface for your convenience as well. ↩

The selection will be based on the label as you defined it during the storage account setup process. ↩

So… I am king of the accidental pun. I’m seriously not kidding, or bragging. It’s a gift and a curse, but mainly a curse. ↩