If you’re using Amazon S3 to store your website’s files, and you want some of them to be private, only accessible to certain users, and particularly if you’re using Django, here’s how.

It took me the best part of a day to piece together the parts of this so I thought they should all be in one place for the next person. I can’t believe I found this so hard. I may have made mistakes, but it seems to work.

I’m going to assume you’ve already got your site set up to store static files on S3 (which is particularly useful if, say, your site is hosted on Heroku). You should already have:

Your media and static folders

This isn’t essential for the whole private-files-on-S3 gist of this post, but getting your media and static files to end up on S3 in separate folders is a little non-obvious, but very useful. So, a little aside to cover it… I’ve got this working nicely by using something like this Stackoverflow answer:

Those STATIC_DIRECTORY and MEDIA_DIRECTORY settings aren’t standard Django settings, but we need the MEDIA_DIRECTORY value when setting permissions on our private files a little later.

I think you’ll need to manually create the /media/ and /static/ directories in your S3 bucket. Then, if you run the collectstatic Django management command your static files should end up in http://your-bucket-name.s3.amazonaws.com/static/. And any files uploaded through FileField or ImageField attributes on your models should end up in http://your-bucket-name.s3.amazonaws.com/media/ . If those model attributes specify upload_to paths, they will be relative to /media/.

Making files private

By default, those media files are public — if you enter an uploaded file’s URL in your browser, you should be able to access it just fine.

Let’s assume that our model has two kinds of file, one public and one private. So in our models.py we have this:

If you upload a private file with that in place, then you should no longer be able to access it directly. e.g., upload a file called test_file.pdf and visiting http://your-bucket-name.s3.amazonaws.com/media/seekrit/test_file.pdf should get you an XML file containing an AccessDenied error.

UPDATE: (12 Feb 2015) Shamim Hasnath suggests creating a new field class to use for the private file:

This will let us link to files something like http://yourdomain.com/42/secretfile/, referring to the secret file on an object with a pk of 42. This is what you should use when linking to the file, e.g., in a template:

What does this all do? First we make sure we get the object from the pk in the URL. Then we want to make sure this user can access the file. The conditions are up to you. Here we’re making sure the user is logged in, and either satisfies some condition set in an is_very_special() method on the user’s UserProfile, or is a staff member.

If that’s OK, and our object actually has a private_file uploaded, then we set the full filepath — this is why we had to create that MEDIA_DIRECTORY setting earlier on, because it needs to be an absolute path, not relative to /media/.

We then generate the signed, temporary URL. Here the URL will be valid for 60 seconds — after that it will no longer function, so it can’t be passed on to anyone else. Once we’ve got this URL, we redirect to it, and the user should be able to access the file — it should appear in the browser, or start downloading, depending on the file type.

Except, that’s not quite all…

Setting the Bucket Policy

Just because we’ve got a signed URL, this doesn’t yet mean the file will download. The private permissions we set on it earlier still apply. We need to specify a policy that will let us bypass this with the signed URLs.

Go to your S3 console, select your Bucket, and right-click to show its Properties. You should see a link saying “Add bucket policy” (or “Edit bucket policy” if you already have one). A window should open, into which you should put something like this:

In the Principal value, the 12345678901 shown here should be replaced with your AWS Account Number. This should be visible on the AWS Manage Your Account page, currently shown at the top-right. Remove the hyphens and put it in here.

In Resource put your actual Bucket Name in place of your-bucket-name and set the path to point to the folder your private files are in. The asterisk on the end means this policy applies to all the files in that folder.

This seems to work for me, although I can’t claim to understand it in great depth.

That’s it

And there we go. I found most of that via Googling, but none of it was all in one place and it took way too long to piece together. Hopefully it’ll be useful to others.

Bear in mind that I DON’T REALLY KNOW WHAT I’M DOING and may have got things wrong. If you spot anything that could be improved, please do let me know (email or Twitter).

On how much has changed. “…you could supply the NME’s news editor with a piece of information secure in the knowledge that it would not escape into the wider world for the week it took to get through the production and publication process and actually appear on the page.” (via @matlock)