Blackbird

Blackbird is a promise implementation for Common Lisp. Its purpose is to make
asynchronous operations (such as non-blocking IO or background jobs) simpler to
comprehend and manage.

On top of promises, it provides a number of macros to make using promises as
natural as writing normal lisp code. This lowers the mental barrier to using
asynchronous methods in your app and makes it easier to understand what your
code is doing.

Intro to promises
—————-
A promise is a representation of a value that may exist sometime in the future.
The idea is that you can attach actions to a promise that will run once its
value is computed, and also attach error handlers to make sure any problems are
handled along the way.

Promises not only give an important abstraction for asynchronous programming, but
offer opportunities for syntactic abstraction that make async
programming a lot more natural.

The blackbird promise implementation supports the concept of promise chaining,
meaning values and errors that happen in your computations become available to
other promises as they progress, allowing you to program naturally even if
you’re doing async operations (which are traditionally callback-based). Here’s
how it’s done:

If a callback is attached to a value that is not a promise,
that callback is called immediated with the value. This makes it so that you can
attach a callback to anything: a promise or a value, and the end result is the
same. This way, the distiction between CPS and normal, stack-based programming
fades slightly because a function can return a promise or a value, and you can
bind a callback to either.

Calling attach, catcher, or finally
always returns a promise. This returned promise gets
fired with the return value of the callback being attached. So if you have
Promise A and you attach a callback to it, attach returns Promise B. Promise B
gets finished/resolved with the return value(s) from the callback attached to
Promise A.

Finishing/resolving a promise with another promise as the first value results
in the callbacks/errbacks from the promise being finished transferring over
to the promise that is passed as the value. This, in addition to
attach/catcher/finally
always returning a promise, makes chaining promises possible. In other words, a
promise can result in a promise which results in a promise, and if the final promise
is finished with a value, the callbacks attached to the first (top-level) promise
will be called with this value. This provides what’s almost a call stack for
asynchronous operations in that you can derive a value from deep within a bunch
of CPS calls back to the top-level, assuming that your operations are in the
tail position (remember, a callback has to return the promise from the next
operation for this to work).

This is all probably greek, so let’s give an example (using
cl-async with the as
nickname):

(use-package:blackbird)(defunpromise-calc(x)"Asynchronously add 1 to x, returning a promise that will be finished when x is computed."(with-promise(resolvereject)(as:delay(lambda()(resolve(+x1))):time1)))(as:with-event-loop()(let((promise(attach(promise-calc0)(lambda(x);; x is 1 here(attach(promise-calcx)(lambda(x);; x is 2 here(attach(promise-calcx)(lambda(x);; x is 3 here(*x5)))))))))(attachpromise(lambda(x)(formatt"Final result: ~a"x)))))

This waits 3 seconds then prints:

Final result: 15

Notice how the callback was attached to the top-level promise, but was able to
get the result computed from many async-levels deep. Not only does this mimick a
normal call stack a lot closer than CPS, but can be wrapped in macros that make
the syntax almost natural (note that these macros I speak of are on the way).

It’s important to note that a promise can hold either one set of value(s) or one
error. It cannot hold both, and once it has either a value (or multiple values)
or an error attached to it, the promise essentially becomes read-only.

Promises API
———-

promise (class)

The promise class represents a promise value. For your application, it’s mostly
an opaque object which can be operated on using the functions/macros below. It
currently has no public accessors, and mainly just holds callbacks, errbacks,
values, events, etc.

promisep

promise-finished-p

(defmethodpromise-finished-p(promise))=>T/NIL

This method returns T or NIL depending on whether the given promise has been
finished.

create-promise

(defuncreate-promise(create-fn&keyname))=>promise

Creates and returns a new promise using the given create-fn, which is a
function of exactly two arguments: a function that can be called with any number
of values and finishes the returned promise, or a function of one argument
that signals an error on the promise:

(create-promise(lambda(resolverrejecter);; note that create-promise will catch errors in the form for you, so this;; handler-case isn't strictly necessary but gives an example of promise;; rejection(handler-case(let((my-vals(multiple-value-list(my-operation))))(applyresolvermy-vals))(t(e)(funcallrejectere)))))

This syntax can be a little cumbersome, so see with-promise,
which cleans things up a bit for you.

Note that resolve acts differently depending on the number of arguments passed
to it. If multiple arguments are passed, the promise is finished with the given
values. However, if only one argument is passed, the promise is finished with
the value(s) that the given form returns. In other words:

promisify

Given any value(s) (or a triggered error), returns a promise either finished
with those value(s) or failed with the given error:

(promisify3); finished promise with the value 3(promisify(values"jerry"28)); finished promise with the values "jerry" and 28(promisify(error"oh no")); failed promise with the error "oh no"

attach

(defmacroattach(promise-gencallback))=>new-promise

This macro attaches a callback to a promise such that once the promise computes,
the callback will be called with the promise’s finished value(s) as its
arguments.

attach takes two arguments, promise-gen and callback. promise-gen is a
form that can (but is not required to) return a promise. If the first value of
promise-gen’s return values is a promise, the callback given is attached to that
promise to be fired when the promise’s value(s) are finished. If the first item
in promise-gen is not a promise class, the given callback is fired
instantly with the values passed as promise-values as the arguments.

The reason attach fires the callback instantly is that it’s sometimes nice to
attach a callback to a value when you don’t know whether the value is a promise
or an already-computed value. This allows for some useful syntactic
abstractions.

If attach is called on a promise that has already been finished, it fires
the given callback immediately with the promise’s value(s).

attach returns one value: a promise that is finished with the return values
of the given callback once it completes. So the original promise fires, the
callback gets called, and then the promise that was returned from attach is
resolved with the return values from the callback.

Also note that if a promise is finished/resolved with another promise as the
first value, the original promise’s callbacks/errorbacks are transfered to
the new promise. This, on top of attach always returning a promise, makes
possible some nice syntactic abstractions which can somewhat mimick non CPS
style by allowing the results from async operations several levels deep to be
viewable by the top-level caller.

catcher

(defmacrocatcher(promise-gen&resthandler-forms))=>new-promise

The catcher macro is used for catching errors on your promises. It listens for
any errors on your promise chain and allows you to set up handlers for them. The
handler syntax is very much like cl’s handler-case:

Notice that like attach, catcher returns a new promise that
resolves to either

the value of the promise it wrapped, in the case no error happened

the value returned by its handler form, if an error occurred

tap

(defmacrotap(promise-gentap-fn))=>new-promise

Tap is special because it hooks into a promise chain like attach,
however instead of resolving to the return value(s) of its given function, tap
just forwards the value(s)/error it inherited from the promise given in
promise-gen once it completes.

It’s important to note that the promise chain waits for tap to complete if it
returns a promise, however the chain continues as if the original promise was
given back. In other words, tap injects itself into the chain without actually
changing any of the values.

This can be useful for logging or spawning actions whos values you don’t really
need but you need to complete before the chain continues.

(attach(tap(attach(promisify4)(lambda(x)(+x3)));; x is 7 here, but stays 7 even though (format ...) returns nil(lambda(x)(formatt"x is ~a~%"x)))(lambda(x);; continue with our addition(+x12)))

Notice we got what amounts to read-only access to the values in the promise, and
we logged them out without worrying about polluting the values.

finally

(defmacrofinally(promise-gen&bodybody))=>new-promise

The finally macro works much like attach and catcher in
that it attaches to a promise and resolves the promise it returns to the value
of its body form. However, its body form is called whether or not the given
promise resolves or is rejected…think of it as the unwind-protect equivalent
of promises. Its form runs whether the promise chain has an error or a value.
It can be used to close connections or clean up resources that are no longer
needed.

(let((connnil))(finally;; catch any db errors(catcher;; grab a connection and get some users(attach(get-db-connection)(lambda(db)(setfconndb)(attach(get-users-from-dbconn)(lambda(users)(do-something-with-usersusers)))))(connection-failed(e)(formatt"bad connection! ~a~%"e))(wrong-db(e)(formatt"i'm sorry, but the users are in another db ~a~%"e))(t(e)(formatt"unknown error: ~a~%"e)));; always close the db, rain or shine(whenconn(close-db-conn))))

Note that we can use finally to close up the database whether things went well
or not. Also note how annoying it is to type out all those attach calls. Fear
not, better syntax awaits.

Nicer syntax
————
Promises are a great abstraction not only because of the decoupling of an action
and a callback, but also because they can be wrapped in macros to make syntax
fairly natural. The following macros aim to be as close to native lisp as
possible while dealing with asynchronous operations.

alet

(defmacroalet(bindings&bodybody))=>new-promise

This macro allows (let) syntax with async functions that return promises. It
binds the promise return values to the given bindings (in parallel), then runs
the body when all the promises have finished.

It’s important to note that alet returns a promise from its form, meaning it
can have a callback attached to it, just like any other
promise-generating form.

Also know that the binding forms do not not not have to return a promise for
the binding process to work. They can return any value, and that variable will
just be bound to that value.

If an alet binding form results in multiple values, the first value will be
bound to the variable (just like let).

;; example (x and y compute in parallel)(alet((x(grab-x-from-server1))(y(grab-y-from-server2)))(formatt"x + y = ~a~%"(+xy)));; alet can bind to nil, meaning that the promise is run, but the result is;; thrown out(alet((x(grab-x-from-server))(nil(run-command-i-dont-need-the-return-val-for)))(formatt"got: ~a~%"x))

Note:alet is a useful tool for running operations in parallel, however
use caution when running multiple commands on the same socket, since many
drivers will get confused as to which response goes to which request. Sometimes
opening N connections is easier than trying to match request/response pairs.

alet*

(defmacroalet*(bindings&bodybody))=>new-promise

This macro allows (let*) syntax with async functions that return promises. It
binds the promise return values to the given bindings (in sequence), allowing
later bindings to be able to use the values from previous bindings, and then
runs the body when all promises have calculated.

It’s important to note that alet* returns a promise from its form, meaning it
can have a callback attached to it, just like any other
promise-generating form.

Also know that the binding forms do not not not have to return a promise for
the binding process to work. They can return any value, and that variable will
just be bound to that value.

If an alet* binding form results in multiple values, the first value will be
bound to the variable (just like let*).

;; example (note we calculate uid THEN name)(alet*((uid(grab-user-id-from-server))(name(get-user-name-from-iduid)))(formatt"Dear, ~a. Please return my pocket lint you borrowed from me. My grandfather gave it to me and it is very important. If you do not see fit to return it, be prepared to throw down. Seriously, we're going to throw down and I'm going to straight wreck you.~%"name));; alet* can bind to nil, meaning that the promise is run, but the result is;; thrown out(alet*((x(grab-x-from-server))(nil(save-valx)))(formatt"got: ~a~%"x))

aif

(defmacroaif(promise-gentrue-formfalse-form))=>new-promise

This macro provides if for asynchronous values. It is a very simple wrapper
around alet that provides a nice syntax for making decisions based on what a
promise will return:

;; `grab-user-from-db` can return a promise here. if the promise is finished with;; any value other than NIL, "User exists!" will be printed. If NIL, then "User;; does not exist..." will print.(aif(my-app:grab-user-from-dbuser-id)(formatt"User exists!~%")(formatt"User does not exist...~%"))

multiple-promise-bind

Like multiple-value-bind but for promises. Allows wrapping around a promise that
finishes with multiple values.

It’s important to note that multiple-promise-bind returns a promise, meaning it
can have a callback attached to it, just like any other
promise-generating form.

Also note that the promise-gen value does not have to evaluate to a promise, but
any value(s), and the bindings will just attach to the given value(s) (in which
case it works exactly like multiple-value-bind.

Utils
—–
The following utility functions allow us to easily perform operations over lists
of values/promises, or promises of lists of values/promises (it’s turtles all
the way down).

aeach

(defunaeach(functionpromise-list))=>promise

Given a list of promises/values, waits for each promise in the list to resolve,
calls function on the value(s) returned from the promise, and waits for the
promise/value returned from the function. aeach does this in sequence for
each item in the promise-list.

This is a great way to loop over items one-by-one, each one waiting for the
previous to finish before continuing.

adolist

(defmacroadolist((itemitems&optionalpromise-bind)&bodybody))=>promise

This macro allows looping over items in an async fashion. Since it can be tricky
to iterate over a set of results that each does async processing but need to
happen in sequence, this macro abstracts all this away.

Here are some toy examples:

;; define a timer function(defunmydelay(time)(let((promise(make-promise)))(formatt"delay for ~a~%"time)(as:delay(lambda()(finishpromise)):timetime)promise));; loop over a list of integers, waiting for each one to finish before;; triggering the next one.;;;; this prints:;; delay for 1 (1s pause);; delay for 2 (2s pause);; delay for 3 (3s pause);; DONE!(wait(adolist(item'(123))(mydelayitem))(formatt"DONE!~%"));; to get more control over how the promise finishes, specify the promise-bind arg(adolist(item'(123)promise)(wait(mydelayitem);; continue the loop explicitely(finishpromise)))

amap

(defunamap(functionpromise-list))=>new-promise

Maps over a list of values (or promises of values) and runs the given function
on each value, resolving its returned promise as the list of mapped values. Note
they if function returns a promise, amap waits for the promise to resolve
before continuing, allowing you to run promise-returning operations on all the
values in the collection.

Note that an error on any of the promises being iterated will signal an error
on the promise returned from amap.

all

(defunall(promise-list))=>promise

Waits on all of the given values in the promise-list to complete before
resolving with the list of all computed values. Like amap, the promise-list
can be a promise to a list of promises. All will be resolved before continuing.

Note that an error on any of the promises being iterated will signal an error
on the promise returned from amap.

areduce

Runs over a list of values (or promises of values) and runs the given
function on each value, while accumulating the result, similar to
reduce.

initial-value is being passed as the first accumulator argument, but
note that it should not be a promise itself. By default this is nil;
the function will also not be called if there were zero elements in the
list, instead the initial-value will be returned.

In contrast to amap the function should also not be returning a
promise itself (unless it’s prepared to deal with it as an argument in
the next iteration).

Notice how our chain has a top-down syntax vs the more lispy inside-out syntax.
Which you prefer is a matter of preference, but chain can conceivably used to
string together many simple operations a lot more clearly than the alternative.
It’s important to note that alet/alet* could just as
easily handle most of the binding stuff, the real utility is being able to set
up catcher/finally wrappers at the end of your chain.