This is a great feature, and allows you to give temporary access to users to upload an object straight to an S3 bucket. The advantages are obvious; your application doesn’t have to handle file uploads any more. Why upload to your app servers and then have your application upload to S3, when you can cut out the middle man?

We recently came across a problem in one of our applications where this feature would have helped.

The application in question allowed users to upload files. In this instance, the application was load balanced behind n (where n is more than 1) Varnish servers, which were in turn load balanced behind ELB. The bug manifested in uploads failing, seemingly randomly. Some quick headbashing revealed the problem to be down to our uploads being multi-part, or rather more specifically, our Varnish instances were configured to use a round-robin director, which mean that different parts of the uploaded file were received by different application servers (‘backends’ in Varnish speak). They would then error when they realised they hadn’t received the whole of the multi-part upload.

We made a quick fix in our Varnish setup to instead use the hash director for the Varnish back-ends that handle uploads. This meant that every multi-part upload routed to the same back-end.

Going back to the original ‘tip’, we can now fix this issue in a nicer way, by bypassing ELB, Varnish, and our application servers entirely, and granting the user temporary access to upload their files straight to S3.