The Multipart upload API enables you to upload large objects in parts. You can use
this
API to upload new large objects or make a copy of an existing object (see Operations on Objects).

Multipart uploading is a three-step process: You initiate the upload, you upload the
object parts, and after you have uploaded all the parts, you complete the multipart
upload.
Upon receiving the complete multipart upload request, Amazon S3 constructs the object
from the
uploaded parts, and you can then access the object just as you would any other object
in
your bucket.

You can list of all your in-progress multipart uploads or get a list of the parts
that
you have uploaded for a specific multipart upload. Each of these operations is explained
in
this section.

Multipart Upload Initiation

When you send a request to initiate a multipart upload, Amazon S3 returns a response
with
an upload ID, which is a unique identifier for your multipart upload. You must include
this upload ID whenever you upload parts, list the parts, complete an upload, or abort
an
upload. If you want to provide any metadata describing the object being uploaded,
you must
provide it in the request to initiate multipart upload.

Parts Upload

When uploading a part, in addition to the upload ID, you must specify a part number.
You can choose any part number between 1 and 10,000. A part number uniquely identifies
a
part and its position in the object you are uploading. Part number you choose need
not be a
consecutive sequence (for example, it can be 1, 5, and 14). If you upload a new part
using
the same part number as a previously uploaded part, the previously uploaded part is
overwritten. Whenever you upload a part, Amazon S3 returns an ETag header in
its response. For each part upload, you must record the part number and the ETag value.
You
need to include these values in the subsequent request to complete the multipart
upload.

Note

After you initiate a multipart upload and upload one or more parts, you must either
complete or abort the multipart upload in order to stop getting charged for storage
of the
uploaded parts. Only after you either complete or abort a multipart upload
will Amazon S3 free up the parts storage and stop charging you for the parts storage.

Multipart Upload Completion (or Abort)

When you complete a multipart upload, Amazon S3 creates an object by concatenating
the parts in
ascending order based on the part number. If any object metadata was provided in the
initiate multipart upload request, Amazon S3 associates that metadata
with the object. After a successful complete request, the parts no
longer exist. Your complete multipart upload request must include the
upload ID and a list of both part numbers and corresponding ETag values. Amazon S3
response
includes an ETag that uniquely identifies the combined object data. This ETag will
not
necessarily be an MD5 hash of the object data. You can optionally abort the multipart
upload. After aborting a multipart upload, you cannot upload any part using that upload
ID
again. All storage that any parts from the aborted multipart upload consumed is then
freed. If any part uploads were in-progress, they can still succeed or fail even after
you
aborted. To free all storage consumed by all parts, you must abort a multipart upload
only
after all part uploads have completed.

Multipart Upload Listings

You can list the parts of a specific multipart upload or all in-progress multipart
uploads. The list parts operation returns the parts information that you have uploaded
for
a specific multipart upload. For each list parts request, Amazon S3 returns the parts
information for the specified multipart upload, up to a maximum of 1,000 parts. If
there
are more than 1,000 parts in the multipart upload, you must send a series of list
part
requests to retrieve all the parts. Note that the returned list of parts doesn't include
parts that haven't completed uploading. Using the list multipart uploads operation, you can obtain a
list of multipart uploads in progress. An in-progress multipart upload is an upload
that
you have initiated, but have not yet completed or aborted. Each request returns at
most
1000 multipart uploads. If there are more than 1,000 multipart uploads in progress,
you
need to send additional requests to retrieve the remaining multipart uploads.
Only use the returned listing for verification. You should not use the result
of this listing when sending a complete multipart upload request.
Instead, maintain your own list of the part numbers you specified when uploading parts
and the corresponding ETag values that Amazon S3 returns.

Concurrent Multipart Upload Operations

In a distributed development environment, it is possible for your application to
initiate several updates on the same object at the same time. Your application might
initiate several multipart uploads using the same object key. For each of these uploads,
your application can then upload parts and send a complete upload request to Amazon
S3 to
create the object. When the buckets have versioning enabled, completing a multipart
upload
always creates a new version. For buckets that do not have versioning enabled, it
is
possible that some other request received between the time when a multipart upload
is
initiated and when it is completed might take precedence.

Note

It is possible for some other request received between the time you initiated a
multipart upload and completed it to take precedence. For example, if another operation
deletes a key after you initiate a multipart upload with that key, but before you
complete it, the complete multipart upload response might indicate a successful object
creation without you ever seeing the object.

Multipart Upload and Pricing

Once you initiate a multipart upload, Amazon S3 retains all the parts until you either
complete or abort the upload. Throughout its lifetime, you are billed for all storage,
bandwidth, and requests for this multipart upload and its associated parts. If you
abort
the multipart upload, Amazon S3 deletes upload artifacts and any parts that you have
uploaded, and
you are no longer billed for them. For more information about pricing, see Amazon S3 Pricing.

Aborting Incomplete Multipart
Uploads Using a Bucket Lifecycle Policy

After you initiate a multipart upload, you begin uploading parts. Amazon S3 stores
these
parts, but it creates the object from the parts only after you upload all of them
and send a successful request to complete the multipart upload (you should verify
that your request to complete multipart upload is successful). Upon receiving
the
complete multipart upload request, Amazon S3 assembles the parts and creates an
object.

If you don't send the complete multipart upload request successfully, Amazon S3 will
not
assemble the parts and will not create any
object.
Therefore, the parts remain in Amazon S3 and you pay for the parts that are stored
in Amazon S3. As a
best practice, we recommend you configure a lifecycle rule (using the
AbortIncompleteMultipartUpload action) to minimize your storage costs.

Amazon S3 supports a bucket lifecycle rule that you can use to direct Amazon S3 to
abort multipart uploads that don't complete within a specified number of days
after
being initiated. When a multipart upload is not completed within the time frame,
it
becomes eligible for an abort operation and Amazon S3 aborts the multipart upload
(and deletes the
parts associated with the multipart upload).

The following is an example lifecycle configuration that specifies a rule with the
AbortIncompleteMultipartUpload action.

In the example, the rule does not specify a value for the Prefix element
(object key name prefix) and therefore it applies to all objects in the bucket for
which
you initiated multipart uploads. Any multipart uploads that were initiated and did
not
complete within seven days become eligible for an abort operation (the action has
no effect
on completed multipart uploads).

if the multipart upload is completed within the number of days specified in the rule,
the AbortIncompleteMultipartUpload lifecycle action does not
apply (that is, Amazon S3 will not take any action). Also, this action does
not
apply to objects, no objects are deleted by this lifecycle action.

The following put-bucket-lifecycle CLI command adds the lifecycle
configuration for the specified bucket.

Save the following example lifecycle configuration in a file (lifecycle.json). The
example
configuration specifies empty prefix and therefore it applies to all objects in the
bucket. You can specify a prefix to restrict the policy to a subset of objects.