5.8 WITH Queries

WITH provides a way to write subqueries for use in a larger
SELECT query. The subqueries can be thought of as defining
temporary tables that exist just for this query. One use of this feature
is to break down complicated queries into simpler parts. An example is:

WITH regional_sales AS (
SELECT region, SUM(amount) AS total_sales
FROM orders
GROUP BY region
), top_regions AS (
SELECT region
FROM regional_sales
WHERE total_sales > (SELECT SUM(total_sales)/10 FROM
regional_sales)
)
SELECT region,
product,
SUM(quantity) AS product_units,
SUM(amount) AS product_sales
FROM orders
WHERE region IN (SELECT region FROM top_regions)
GROUP BY region, product;

which displays per-product sales totals in only the top sales regions.
This example could have been written without WITH,
but we'd have needed two levels of nested sub-SELECTs. It's a bit
easier to follow this way.

The optional RECURSIVE modifier changes WITH
from a mere syntactic convenience into a feature that accomplishes
things not otherwise possible in standard SQL. Using
RECURSIVE, a WITH query can refer to its own
output. A very simple example is this query to sum the integers from 1
through 100:

The general form of a recursive WITH query is always a
non-recursive term, then UNION (or
UNION ALL), then a
recursive term, where only the recursive term can contain
a reference to the query's own output. Such a query is executed as
follows:

Recursive Query Evaluation:

Evaluate the non-recursive term. For UNION (but not
UNION ALL), discard duplicate rows. Include all remaining
rows in the result of the recursive query, and also place them in a
temporary working table.

So long as the working table is not empty, repeat these steps:

Evaluate the recursive term, substituting the current contents of
the working table for the recursive self-reference.
For UNION (but not UNION ALL), discard
duplicate rows and rows that duplicate any previous result row.
Include all remaining rows in the result of the recursive query, and
also place them in a temporary intermediate table.

Replace the contents of the working table with the contents of the
intermediate table, then empty the intermediate table.

Note: Strictly speaking, this process is iteration not recursion, but
RECURSIVE is the terminology chosen by the SQL standards
committee.

In the example above, the working table has just a single row in each step,
and it takes on the values from 1 through 100 in successive steps. In
the 100th step, there is no output because of the WHERE
clause, and so the query terminates.

Recursive queries are typically used to deal with hierarchical or
tree-structured data. A useful example is this query to find all the
direct and indirect sub-parts of a product, given only a table that
shows immediate inclusions:

When working with recursive queries it is important to be sure that
the recursive part of the query will eventually return no tuples,
or else the query will loop indefinitely. Sometimes, using
UNION instead of UNION ALL can accomplish this
by discarding rows that duplicate previous output rows. However, often a
cycle does not involve output rows that are completely duplicate: it may be
necessary to check just one or a few fields to see if the same point has
been reached before. The standard method for handling such situations is
to compute an array of the already-visited values. For example, consider
the following query that searches a table graph using a
link field:

This query will loop if the link relationships contain
cycles. Because we require a “depth” output, just changing
UNION ALL to UNION would not eliminate the looping.
Instead we need to recognize whether we have reached the same row again
while following a particular path of links. We add two columns
path and cycle to the loop-prone query:

Tip: Omit the ROW() syntax in the common case where only one field
needs to be checked to recognize a cycle. This allows a simple array
rather than a composite-type array to be used, gaining efficiency.

Tip: The recursive query evaluation algorithm produces its output in
breadth-first search order. You can display the results in depth-first
search order by making the outer query ORDER BY a
“path” column constructed in this way.

A helpful trick for testing queries
when you are not certain if they might loop is to place a LIMIT
in the parent query. For example, this query would loop forever without
the LIMIT:

This works because PostgreSQL's implementation
evaluates only as many rows of a WITH query as are actually
fetched by the parent query. Using this trick in production is not
recommended, because other systems might work differently. Also, it
usually won't work if you make the outer query sort the recursive query's
results or join them to some other table.

A useful property of WITH queries is that they are evaluated
only once per execution of the parent query, even if they are referred to
more than once by the parent query or sibling WITH queries.
Thus, expensive calculations that are needed in multiple places can be
placed within a WITH query to avoid redundant work. Another
possible application is to prevent unwanted multiple evaluations of
functions with side-effects.
However, the other side of this coin is that the optimizer is less able to
push restrictions from the parent query down into a WITH query
than an ordinary sub-query. The WITH query will generally be
evaluated as stated, without suppression of rows that the parent query
might discard afterwards. (But, as mentioned above, evaluation might stop
early if the reference(s) to the query demand only a limited number of
rows.)