LLVM coroutines are functions that have one or more suspend points.
When a suspend point is reached, the execution of a coroutine is suspended and
control is returned back to its caller. A suspended coroutine can be resumed
to continue execution from the last suspend point or it can be destroyed.

In the following example, we call function f (which may or may not be a
coroutine itself) that returns a handle to a suspended coroutine
(coroutine handle) that is used by main to resume the coroutine twice and
then destroy it:

In addition to the function stack frame which exists when a coroutine is
executing, there is an additional region of storage that contains objects that
keep the coroutine state when a coroutine is suspended. This region of storage
is called coroutine frame. It is created when a coroutine is called and
destroyed when a coroutine runs to completion or destroyed by a call to
the coro.destroy intrinsic.

An LLVM coroutine is represented as an LLVM function that has calls to
coroutine intrinsics defining the structure of the coroutine.
After lowering, a coroutine is split into several
functions that represent three different ways of how control can enter the
coroutine:

a ramp function, which represents an initial invocation of the coroutine that
creates the coroutine frame and executes the coroutine code until it
encounters a suspend point or reaches the end of the function;

a coroutine resume function that is invoked when the coroutine is resumed;

a coroutine destroy function that is invoked when the coroutine is destroyed.

Note

Splitting out resume and destroy functions are just one of the
possible ways of lowering the coroutine. We chose it for initial
implementation as it matches closely the mental model and results in
reasonably nice code.

Let’s look at an example of an LLVM coroutine with the behavior sketched
by the following pseudo-code.

void*f(intn){for(;;){print(n++);<suspend>// returns a coroutine handle on first suspend}}

This coroutine calls some function print with value n as an argument and
suspends execution. Every time this coroutine resumes, it calls print again with an argument one bigger than the last time. This coroutine never completes by itself and must be destroyed explicitly. If we use this coroutine with
a main shown in the previous section. It will call print with values 4, 5
and 6 after which the coroutine will be destroyed.

The entry block establishes the coroutine frame. The coro.size intrinsic is
lowered to a constant representing the size required for the coroutine frame.
The coro.begin intrinsic initializes the coroutine frame and returns the
coroutine handle. The second parameter of coro.begin is given a block of memory
to be used if the coroutine frame needs to be allocated dynamically.
The coro.id intrinsic serves as coroutine identity useful in cases when the
coro.begin intrinsic get duplicated by optimization passes such as
jump-threading.

The cleanup block destroys the coroutine frame. The coro.free intrinsic,
given the coroutine handle, returns a pointer of the memory block to be freed or
null if the coroutine frame was not allocated dynamically. The cleanup
block is entered when coroutine runs to completion by itself or destroyed via
call to the coro.destroy intrinsic.

The suspend block contains code to be executed when coroutine runs to
completion or suspended. The coro.end intrinsic marks the point where
a coroutine needs to return control back to the caller if it is not an initial
invocation of the coroutine.

The loop blocks represents the body of the coroutine. The coro.suspend
intrinsic in combination with the following switch indicates what happens to
control flow when a coroutine is suspended (default case), resumed (case 0) or
destroyed (case 1).

One of the steps of coroutine lowering is building the coroutine frame. The
def-use chains are analyzed to determine which objects need be kept alive across
suspend points. In the coroutine shown in the previous section, use of virtual register
%n.val is separated from the definition by a suspend point, therefore, it
cannot reside on the stack frame since the latter goes away once the coroutine
is suspended and control is returned back to the caller. An i32 slot is
allocated in the coroutine frame and %n.val is spilled and reloaded from that
slot as needed.

We also store addresses of the resume and destroy functions so that the
coro.resume and coro.destroy intrinsics can resume and destroy the coroutine
when its identity cannot be determined statically at compile time. For our
example, the coroutine frame will be:

%f.frame=type{void(%f.frame*)*,void(%f.frame*)*,i32}

After resume and destroy parts are outlined, function f will contain only the
code responsible for creation and initialization of the coroutine frame and
execution of the coroutine until a suspend point is reached:

A particular coroutine usage pattern, which is illustrated by the main
function in the overview section, where a coroutine is created, manipulated and
destroyed by the same calling function, is common for coroutines implementing
RAII idiom and is suitable for allocation elision optimization which avoid
dynamic allocation by storing the coroutine frame as a static alloca in its
caller.

In the entry block, we will call coro.alloc intrinsic that will return true
when dynamic allocation is required, and false if dynamic allocation is
elided.

In this case, the coroutine frame would include a suspend index that will
indicate at which suspend point the coroutine needs to resume. The resume
function will use an index to jump to an appropriate basic block and will look
as follows:

If different cleanup code needs to get executed for different suspend points,
a similar switch will be in the f.destroy function.

Note

Using suspend index in a coroutine state and having a switch in f.resume and
f.destroy is one of the possible implementation strategies. We explored
another option where a distinct f.resume1, f.resume2, etc. are created for
every suspend point, and instead of storing an index, the resume and destroy
function pointers are updated at every suspend. Early testing showed that the
current approach is easier on the optimizer than the latter so it is a
lowering strategy implemented at the moment.

In the previous example, setting a resume index (or some other state change that
needs to happen to prepare a coroutine for resumption) happens at the same time as
a suspension of a coroutine. However, in certain cases, it is necessary to control
when coroutine is prepared for resumption and when it is suspended.

In the following example, a coroutine represents some activity that is driven
by completions of asynchronous operations async_op1 and async_op2 which get
a coroutine handle as a parameter and resume the coroutine once async
operation is finished.

In this case, coroutine should be ready for resumption prior to a call to
async_op1 and async_op2. The coro.save intrinsic is used to indicate a
point when coroutine should be ready for resumption (namely, when a resume index
should be stored in the coroutine frame, so that it can be resumed at the
correct resume point):

A coroutine author or a frontend may designate a distinguished alloca that can
be used to communicate with the coroutine. This distinguished alloca is called
coroutine promise and is provided as the second parameter to the
coro.id intrinsic.

The following coroutine designates a 32 bit integer promise and uses it to
store the current value produced by a coroutine.

A coroutine author or a frontend may designate a particular suspend to be final,
by setting the second argument of the coro.suspend intrinsic to true.
Such a suspend point has two properties:

it is possible to check whether a suspended coroutine is at the final suspend
point via coro.done intrinsic;

a resumption of a coroutine stopped at the final suspend point leads to
undefined behavior. The only possible action for a coroutine at a final
suspend point is destroying it via coro.destroy intrinsic.

From the user perspective, the final suspend point represents an idea of a
coroutine reaching the end. From the compiler perspective, it is an optimization
opportunity for reducing number of resume points (and therefore switch cases) in
the resume function.

The following is an example of a function that keeps resuming the coroutine
until the final suspend point is reached after which point the coroutine is
destroyed:

Usually, final suspend point is a frontend injected suspend point that does not
correspond to any explicitly authored suspend point of the high level language.
For example, for a Python generator that has only one suspend point:

defcoroutine(n):foriinrange(n):yieldi

Python frontend would inject two more suspend points, so that the actual code
looks like this:

Intrinsics described in this section are used to manipulate an existing
coroutine. They can be used in any function which happen to have a pointer
to a coroutine frame or a pointer to a coroutine promise.

When possible, the coro.destroy intrinsic is replaced with a direct call to
the coroutine destroy function. Otherwise it is replaced with an indirect call
based on the function pointer for the destroy function stored in the coroutine
frame. Destroying a coroutine that is not suspended leads to undefined behavior.

When possible, the coro.resume intrinsic is replaced with a direct call to the
coroutine resume function. Otherwise it is replaced with an indirect call based
on the function pointer for the resume function stored in the coroutine frame.
Resuming a coroutine that is not suspended leads to undefined behavior.

The first argument is a handle to a coroutine if from is false. Otherwise,
it is a pointer to a coroutine promise.

The second argument is an alignment requirements of the promise.
If a frontend designated %promise = alloca i32 as a promise, the alignment
argument to coro.promise should be the alignment of i32 on the target
platform. If a frontend designated %promise = alloca i32, align 16 as a
promise, the alignment argument should be 16.
This argument only accepts constants.

The third argument is a boolean indicating a direction of the transformation.
If from is true, the intrinsic returns a coroutine handle given a pointer
to a promise. If from is false, the intrinsics return a pointer to a promise
from a coroutine handle. This argument only accepts constants.

Using this intrinsic on a coroutine that does not have a coroutine promise
leads to undefined behavior. It is possible to read and modify coroutine
promise of the coroutine which is currently executing. The coroutine author and
a coroutine user are responsible to makes sure there is no data races.

definei8*@f(i32%n){entry:%promise=allocai32%pv=bitcasti32*%promisetoi8*; the second argument to coro.id points to the coroutine promise.%id=calltoken@llvm.coro.id(i320,i8*%pv,i8*null,i8*null)...%hdl=callnoaliasi8*@llvm.coro.begin(token%id,i8*%alloc)...storei3242,i32*%promise; store something into the promise...reti8*%hdl}definei32@main(){entry:%hdl=calli8*@f(i324); starts the coroutine and returns its handle%promise.addr.raw=calli8*@llvm.coro.promise(i8*%hdl,i324,i1false)%promise.addr=bitcasti8*%promise.addr.rawtoi32*%val=loadi32,i32*%promise.addr; load a value from the promisecallvoid@print(i32%val)callvoid@llvm.coro.destroy(i8*%hdl)reti320}

Depending on the alignment requirements of the objects in the coroutine frame
and/or on the codegen compactness reasons the pointer returned from coro.begin
may be at offset to the %mem argument. (This could be beneficial if
instructions that express relative access to data can be more compactly encoded
with small positive and negative offsets).

A frontend should emit exactly one coro.begin intrinsic per coroutine.

The ‘llvm.coro.free’ intrinsic returns a pointer to a block of memory where
coroutine frame is stored or null if this instance of a coroutine did not use
dynamically allocated memory for its coroutine frame.

This intrinsic is lowered to refer to a private constant coroutine frame. The
resume and destroy handlers for this frame are empty functions that do nothing.
Note that in different translation units llvm.coro.noop may return different pointers.

The first argument provides information on the alignment of the memory returned
by the allocation function and given to coro.begin by the first argument. If
this argument is 0, the memory is assumed to be aligned to 2 * sizeof(i8*).
This argument only accepts constants.

The second argument, if not null, designates a particular alloca instruction
to be a coroutine promise.

The third argument is null coming out of the frontend. The CoroEarly pass sets
this argument to point to the function this coro.id belongs to.

The fourth argument is null before coroutine is split, and later is replaced
to point to a private global constant array containing function pointers to
outlined resume and destroy parts of the coroutine.

The purpose of this intrinsic is to tie together coro.id, coro.alloc and
coro.begin belonging to the same coroutine to prevent optimization passes from
duplicating any of these instructions unless entire body of the coroutine is
duplicated.

The first argument should refer to the coroutine handle of the enclosing
coroutine. A frontend is allowed to supply null as the first parameter, in this
case coro-early pass will replace the null with an appropriate coroutine
handle value.

The second argument should be true if this coro.end is in the block that is
part of the unwind sequence leaving the coroutine body due to an exception and
false otherwise.

The purpose of this intrinsic is to allow frontends to mark the cleanup and
other code that is only relevant during the initial invocation of the coroutine
and should not be present in resume and destroy parts.

This intrinsic is lowered when a coroutine is split into
the start, resume and destroy parts. In the start part, it is a no-op,
in resume and destroy parts, it is replaced with ret void instruction and
the rest of the block containing coro.end instruction is discarded.
In landing pads it is replaced with an appropriate instruction to unwind to
caller. The handling of coro.end differs depending on whether the target is
using landingpad or WinEH exception model.

For landingpad based exception model, it is expected that frontend uses the
coro.end intrinsic as follows:

ehcleanup:%InResumePart=calli1@llvm.coro.end(i8*null,i1true)bri1%InResumePart,label%eh.resume,label%cleanup.contcleanup.cont:; rest of the cleanupeh.resume:%exn=loadi8*,i8**%exn.slot,align8%sel=loadi32,i32*%ehselector.slot,align4%lpad.val=insertvalue{i8*,i32}undef,i8*%exn,0%lpad.val29=insertvalue{i8*,i32}%lpad.val,i32%sel,1resume{i8*,i32}%lpad.val29

The CoroSpit pass replaces coro.end with True in the resume functions,
thus leading to immediate unwind to the caller, whereas in start function it
is replaced with False, thus allowing to proceed to the rest of the cleanup
code that is only needed during initial invocation of the coroutine.

For Windows Exception handling model, a frontend should attach a funclet bundle
referring to an enclosing cleanuppad as follows:

The ‘llvm.coro.suspend’ marks the point where execution of the coroutine
need to get suspended and control returned back to the caller.
Conditional branches consuming the result of this intrinsic lead to basic blocks
where coroutine should proceed when suspended (-1), resumed (0) or destroyed
(1).

The first argument refers to a token of coro.save intrinsic that marks the
point when coroutine state is prepared for suspension. If none token is passed,
the intrinsic behaves as if there were a coro.save immediately preceding
the coro.suspend intrinsic.

The second argument indicates whether this suspension point is final.
The second argument only accepts constants. If more than one suspend point is
designated as final, the resume and destroy branches should lead to the same
basic blocks.

If a coroutine that was suspended at the suspend point marked by this intrinsic
is resumed via coro.resume the control will transfer to the basic block
of the 0-case. If it is resumed via coro.destroy, it will proceed to the
basic block indicated by the 1-case. To suspend, coroutine proceed to the
default label.

If suspend intrinsic is marked as final, it can consider the true branch
unreachable and can perform optimizations that can take advantage of that fact.

Separate save and suspend points are necessary when a coroutine is used to
represent an asynchronous control flow driven by callbacks representing
completions of asynchronous operations.

In such a case, a coroutine should be ready for resumption prior to a call to
async_op function that may trigger resumption of a coroutine from the same or
a different thread possibly prior to async_op call returning control back
to the coroutine:

The ‘llvm.coro.param’ is used by a frontend to mark up the code used to
construct and destruct copies of the parameters. If the optimizer discovers that
a particular parameter copy is not used after any suspends, it can remove the
construction and destruction of the copy by replacing corresponding coro.param
with i1 false and replacing any use of the copy with the original.

The optimizer is also allowed to replace it with i1 false provided that the
parameter copy is only used prior to control flow reaching any of the suspend
points. The code that would be DCE’d if the coro.param is replaced with
i1 false is not considered to be a use of the parameter copy.

The frontend can emit this intrinsic if its language rules allow for this
optimization.

The pass CoroEarly lowers coroutine intrinsics that hide the details of the
structure of the coroutine frame, but, otherwise not needed to be preserved to
help later coroutine passes. This pass lowers coro.frame, coro.done,
and coro.promise intrinsics.

The pass CoroElide examines if the inlined coroutine is eligible for heap
allocation elision optimization. If so, it replaces
coro.begin intrinsic with an address of a coroutine frame placed on its caller
and replaces coro.alloc and coro.free intrinsics with false and null
respectively to remove the deallocation code.
This pass also replaces coro.resume and coro.destroy intrinsics with direct
calls to resume and destroy functions for a particular coroutine where possible.

A coroutine frame is bigger than it could be. Adding stack packing and stack
coloring like optimization on the coroutine frame will result in tighter
coroutine frames.

Take advantage of the lifetime intrinsics for the data that goes into the
coroutine frame. Leave lifetime intrinsics as is for the data that stays in
allocas.

The CoroElide optimization pass relies on coroutine ramp function to be
inlined. It would be beneficial to split the ramp function further to
increase the chance that it will get inlined into its caller.

Design a convention that would make it possible to apply coroutine heap
elision optimization across ABI boundaries.