Task Queue Overview

This page describes what task queues are, and when and how to use them.
Task queues let applications perform work, called tasks,
asynchronously outside of a user request. If an app needs to execute work
in the background, it adds tasks to task queues. The tasks are executed later,
by worker services.

Key Point: The Task Queue service is designed for asynchronous work. It does not
provide strong guarantees around the timing of task delivery and is therefore
unsuitable for interactive applications where a user is waiting for the result.

Push queues and pull queues

Task queues come in two flavors, push and pull. The manner in which the
Task Queue service dispatches task requests to worker services is different for
the different queues.

Push queues run tasks by delivering HTTP requests to App Engine worker services.
They dispatch these requests at a reliable, steady rate and guarantee
reliable task execution. Because you can control the rate at which tasks are
sent from the queue, you can control the workers' scaling behavior and hence
your costs.

Because tasks are executed as requests targeted at App Engine
services, they are subject to stringent deadlines. Tasks handled by automatic
scaling services must finish in ten minutes. Tasks handled by basic and manual
scaling services can run for up to 24 hours.

All task queue tasks are performed asynchronously. The application that creates
the task hands it off to the queue. The originating application is not notified
whether or not the task completes, or if it was successful.

If a worker fails to process a task, the Task Queue service provides the queue
with a retry mechanism, so the task can be retried a finite number of times.

Use cases

Push queues

One typical push queue use case is a "slow" operation. Consider a social
network messaging system. Every time a user sends a message, the network needs
to update the followers of the sender. This can be a very time-consuming
operation. Using a push queue, the application can enqueue a task for each
message as it arrives to be dispatched to a worker service for processing. When
the worker receives the task request, it can retrieve the sender's list of
followers and update the DB for each one. The worker can be made even more
efficient by enqueuing another pushtask for each database update.

Another use for push queues is scheduled tasks. Imagine an application that
implements an ad campaign. A group of tasks written to send out emails can be
added to a push queue with instructions to withhold the tasks until a specified
time in the future. When the due date arrives, the Task Queue service will
begin to issue requests to execute the tasks.