Batch Processing

While most services provide synchronous APIs, requiring you to make a request
and then wait for a response, BatchJobService
provides a way to perform batches of operations on multiple services without
waiting for the operations to complete.

Unlike service-specific mutate operations, a single job in BatchJobService can operate
against a mixed collection of campaigns, ad groups, ads, criteria,
labels, and feed items. Submitted jobs run in parallel, and BatchJobService
automatically retries operations that fail due to transient errors such as
RateExceededError.

In addition, BatchJobService allows you to use temporary IDs
within your requests so you can submit dependent operations in a single job.

BatchJobService is not a cure-all solution in all situations. Use of
BatchJobService can reduce the number of requests sent but its use must be
evaluated to see if it's a good fit for your needs.

The main reasons for using batch processing are to minimize the number of
API requests and retry operations that fail due to transient errors. This
doesn't guarantee improved performance resulting in higher
throughput or quicker job completion. You need to experiment with different
combinations of batch and non-batch processing to determine the optimal
solution for your circumstances.

Schema

Although each client library contains utilities
that handle the XML serialization of uploaded operations
and XML deserialization of downloaded results, the complete schema for batch
job requests and responses is available at:

Important: Make sure you capture the uploadUrl from the BatchJob within the returned BatchJobReturnValue. The only time you will get the uploadUrl is in the response for your ADD operation.

At this point in the process, the status
of the job will be AWAITING_FILE.

Create operations for the batch job

For this step, create the operations for your batch job using the same approach
you would use for the synchronous API services. For example, the following
snippet creates CampaignOperation objects for adding new campaigns.

Important: The order of your operations can have a significant impact on your
job's performance. Check out the tip on operation order in the
best practices section below for more information.

Upload operations to the upload URL

Once you've collected the set of operations for your job, the next step is to
send them to the upload URL.

Note: The uploadUrl
for a batch job is valid for one week after the job was created.

If you are using the utility in one of the client libraries,
then you don't need to worry about all of the underlying details. The utility
will handle constructing and sending the requests for you, and will provide
methods for both of the following options:

Poll the batch job status

After you upload your operations, the batch job will be submitted to the job
queue, so you'll want to periodically check the job's status until it is
CANCELED or DONE.
Use an exponential backoff policy to avoid polling too aggressively. The
snippet below will wait 30 seconds on the first attempt, 60 seconds on the
second attempt, 120 seconds on the third attempt, and so on.

Tip: If you want to track the progress of a large set of operations, you can log
the progressStats
returned while the job is running.

Download the batch job results and check for errors

At this stage, your job should be in one of two states.

Status

Description

Actions to take

DONE

BatchJobService successfully parsed and attempted each of the uploaded operations.

Download the results for each operation from the batch job's downloadUrl.

CANCELED

The job was canceled as requested,
BatchJobService could not parse the uploaded operations, or an
unexpected error occurred during job execution.
Note: In this case, some of your operations may have been attempted, so it's best to
check the results in the downloadUrl even though the job has a final status of CANCELED.

The batch job's
processingErrors
will contain all errors encountered while pre-processing your uploaded
operations, such as input file corruption errors. The code below shows one
approach for processing such errors.

Note: The processingErrors attribute will not contain errors thrown when the service attempted to execute successfully
parsed operations -- those will appear in the mutateResult retrieved from the download URL.

Using temporary IDs

A powerful feature of BatchJobService is that it supports the use of temporary IDs.
A temporary ID is a negative number (long) that allows operations in a batch job
to reference the result of an ADD operation from a previous operation in the same
batch job. Simply specify a negative number for the ID in the ADD operation for
the parent object, and then reuse that ID in subsequent mutate() operations for other
dependent objects in the same batch job.

A common use case for temporary IDs is to create a complete campaign in a
single batch job. For example, you could create a single job containing ADD operations
with the following ID assignments on each operand:

This sequence would:

Add a campaign with (temporary) ID -1

Add an ad group with temporary ID -2 for campaign -1

Add an ad group ad for ad group -2

Add multiple ad group criteria (keywords) for ad group -2

Apply a label to campaign -1

Add a campaign negative criterion (keyword) for campaign -1

Note: Dependent operations are applied in the order they appear in your upload.
Therefore, when using temporary IDs, make sure the operation that creates
a parent object comes before the operations that create its child objects.

Use a new temporary ID when creating a new object. If you don't, you will
receive a TaskExecutionError.TEMP_ID_ALREADY_USED error.

Upload requirements

Non-incremental uploads

Each client library utility provides a convenience method for uploading
operations in a single step. However, if you're not using a client library,
please note that non-incremental uploads are not supported.

Incremental uploads

Incremental uploads allow you to send multiple upload requests to the batch job
uploadUrl. Your job will only start executing once you've uploaded the last
set of operations.

Exchange the BatchJob uploadURL for a resumable upload URL

The uploadUrl
of your BatchJobmust be exchanged for a resumable upload URL.
To exchange your uploadUrl for a resumable upload URL,
send a request to the uploadUrl that meets the following
specifications:

Request attributes

Request method

POST

URL

Upload URL returned by BatchJobService.mutate

Content-Type HTTP header

application/xml

Content-Length HTTP header

0

x-goog-resumable HTTP header

start

Request body

no request body required

If your request is successful, the returned response will have a status
of 201 Created, and a Location header whose value
is the resumable upload URL.

Upload operations to the resumable upload URL

Once you have the resumable upload URL, you can start uploading your operations.
Each request sent to the resumable upload URL must meet the following specifications.

Request attributes

Request method

PUT

URL

Resumable upload URL from the initialization step above.

Content-Type HTTP header

application/xml

Content-Length HTTP header

The number of bytes in the contents of the current request.

Content-Range HTTP header

Range of bytes in the request, followed by total bytes. Total bytes will
be * for the first and intermediate requests, but should be
set to the final total bytes when sending the last request.

The size of the request body must be a multiple of 256K
(262144) bytes.

Request body for the resumable upload URL

The BatchJobService will ultimately concatenate all of the XML uploaded to the uploadUrl and parse
it as a single request, so you must take care to include only the start and end mutate elements
on the first and last request, respectively.

Request

Start mutate element

End mutate element

First

Intermediate

Last

In addition, since BatchJobService will parse the uploaded XML as a single document,
the request body for a single request need not contain a complete XML document. For example,
if you were uploading 524305 bytes (256K + 256K + 17), your requests could be as follows:

Content length of 262144, where the e in the last line is the 262144th byte.

Request 3

rations></mutate>
... (padded to 262144 bytes)

Content length without padding of 17 bytes, where the closing > on </mutate>
is the 17th byte. Total content length with padding is 262144 bytes.

Best practices

Consider these guidelines when using BatchJobService:

For better throughput, fewer larger jobs is preferred over many smaller jobs.

When submitting multiple concurrent jobs for the same
clientCustomerId, try to
reduce the likelihood of jobs operating on the same objects at the same time, while
maintaining large job sizes. Many unfinished jobs (with status of ACTIVE or CANCELING)
that try to mutate the same set of objects may lead to deadlock-like conditions resulting
in severe slow-down and even job failures.

Don't submit multiple operations that mutate the same object in the same job. The result will be unpredictable if you do so.

For better throughput, order uploaded operations by operation type. For
example, if your job contains operations to add campaigns, ad groups, and ad group criteria,
order the operations in your upload so that all of the CampaignOperations are first, followed
by all of the AdGroupOperations, and finally all AdGroupCriterionOperations.

Do not poll the job status too frequently or you will risk hitting rate limit errors.

Working with Shopping campaigns

When updating product partition trees using BatchJobService, the following
restrictions apply:

If a list of operations on a product partition tree result in a structurally
invalid product partition tree (for example, a node is subdivided without
creating an other node), the entire list of operations on that
product partition tree will fail.

A BatchJob can contain operations that introduce product partition tree
structural changes for more than two ad groups. The
two ad group limit does not
apply to batch jobs.

When removing a product partition node, set the AdGroupCriterion
object's criterion
field to a ProductPartition
instance. Setting this field to a
Criterion
instance will cause the operation to fail with an
AdGroupCriterionError.CONCRETE_TYPE_REQUIRED error.

Limitations

At any given time, a Google Ads account can have up to 1 GB of uploaded
operations across all of its batch jobs that have not yet completed. Once
your account reaches this limit, you will get a
BatchJobError
with reason DISK_QUOTA_EXCEEDED if you try to add new batch jobs. If you
encounter this error, wait until the size of pending uploaded operations falls
below the limit before creating new jobs.

Test accounts are limited to 250 new jobs per day.

Batch jobs are retained for 60 days after creation. After that, they won't
be retrievable through either get() or query() requests.

Code examples

The following client libraries contain a complete code example that illustrates all of
the features above.