Context Navigation

Twisted's
​deferred generator
capability allows you to run iterations in a way that is more intuitive and
similar to what you may be used to seeing in the world of blocking code.

Here's a simple example, courtesy of Alex Levy, that may help you understand
how the concept works:

fromtwisted.internetimport defer, reactor
# The reactor is running and we are going to obtain a Deferred "d"defgetSomeDeferred():"""Some function that returns a Deferred."""
d = defer.Deferred()
reactor.callLater(1, d.callback,'A string that yells "foo!"')return d
defanotherDeferred(needle, haystack):"""Some other function that returns a Deferred."""
d = defer.Deferred()
reactor.callLater(1, d.callback, haystack.find(needle))return d
@defer.deferredGeneratordeffind(needle):"""A Deferred generator function"""print"I am going to find %s in a haystack."% needle
# After yielding waitForDeferred, our generator# will be put on hold for a while.
wfd = defer.waitForDeferred(getSomeDeferred())yield wfd
# The reactor will call .next(), and we resume here.
haystack = wfd.getResult()print"I got my haystack back from a deferred."# We're going to wait for another deferred result.
wfd = defer.waitForDeferred(anotherDeferred(needle, haystack))yield wfd
# When we get our next result, the procedure resumes here.print"I found %s at character %d"%(repr(needle), wfd.getResult())
reactor.stop()# We call the deferred generator like any other function and immediately# get a Deferred that fires when the generator is done.
d = find('foo!')
reactor.run()

The find function is a generator because it contains yield
statements. In the blocking world, you would use it to iterate over multiple
values, like this:

for thing in find('foo'):print"'%s' is either a needle or a haystack"% thing

However, waiting for a function to yield something is a form of blocking, which
is a no-no when you're writing asynchronous code. So we instead wrap the
find function in a defer.deferredGenerator. Now calling it
immediately returns a Deferred that fires when it is done iterating.

The actual stuff that gets done as part of the iteration is incorporated into
the generator function itself. You yield whatever it is you want to be an
iterator of, as usual, with some important differences.

You don't yield the actual object, but rather something based on
a Deferred that will eventually fire with the object. (If you
could get the object immediately, there would be no reason to go to all
this trouble.)

To get the actual object from what you yielded (a Deferred that
has been specially packaged by defer.waitForDeferred), after the
Twisted event loop has taken time off to go do other stuff, by calling a
getResult method of the yielded object.

Now let's consider a more detailed example, the deferred generator that types
fake keystrokes, one after the other, in the WinDictator application.

fromtwisted.internetimport defer
# The reactor is running and we are inside a class instance that has# attributes referencing a "keyer," an "observer," and a "history."@defer.deferredGeneratordefpressAndReleaseKeys(self, keyList):"""
This method is decorated with defer.deferredGenerator to immediately
return a Deferred that fires upon completion of an iteration over
key combinations, which are supplied as sub-sequences of the 'keyList'
sequence. The deferred fires with 'True' if all keys were observed to
have been pressed and reseased, 'False' otherwise.
The generator sends fake X key events for each key of each
sub-sequence, a keypress followed by a release, and waits for
confirmation of each faked key event before proceeding with the next
one.
"""
running =TruedefgotConfirmation(confirmed):
running = confirmed
return confirmed
defkeyAndConfirm(func, key):
d = func(key)
d.addCallback(self.observer.confirm)
d.addCallback(gotConfirmation)return d
stack =[]while running and keyList:
keySequence = keyList.pop(0)# Depress each key in orderfor key in keySequence:
stack.append(key)
d = keyAndConfirm(self.keyer.pressKey, key)
wfd = defer.waitForDeferred(d)yield wfd

Here the first waitForDeferred instance is yielded from within a
while loop that is doing a first batch of iterations. Note that we are
yielding a wrapped Deferred that has a fairly complicated callback
chain. Each key of a combination (e.g., "Shift_L + a" = "A") is being
virtually pressed in turn via the hidden machinations of the keyer and
confirm objects.

Processing of the callbacks occurs right inside our generator function, but
it's done whenever the Twisted event loop can get around to it. You may wonder
how that is possible given that Twisted code runs in a single thread. The
answer lies in the yield statement, which is an invitation for code
outside the generator to do stuff between iterations. See Norman Matloff's fine
​tutorial on
generators for details.

At this point, we have obtained the value that finally resulted from the calls
to self.keyer.pressKey and self.observer.confirm. The result is a
Boolean indicating that everything went OK in those operations. If it didn't,
we set a flag to prevent further iterations and break out of the current one.

Again, we yield the Deferred that we've packaged up with
defer.waitForDeferred, let the Twisted event loop do its thing, and get
the deferred result. Again, the result is a Boolean status value and we quit
iterating if there's been a problem.

Now we come to a part of the generator function where we want to deliver a
final result to the caller. But how do we do that with all this yielding going
on? How do you even return anything that could be construed as the "result" of
a Python generator, when all you can normally do is iterate over the values it
yields?

Here Twisted's deferred generator adds some functionality to generator
functions. The last object yielded that is not wrapped in a call to
defer.waitForDeferred, is used as the deferred result of the
defer.deferredGenerator call itself!

# Final yield of a Boolean, rather than a wfd, is supplied to the# deferredGenerator's callbackyield running

The running variable holds a status that is True unless made
False by any of the key press and release operations. That's also the
status of the entire key generator operation, and whatever function called it
gets that as the deferred result.

A method in the same class as pressAndReleaseKeys is an example of such a
function:

The method calls self.pressAndReleaseKeys() and immediately gets a
Deferred as the result even while the generator function is hammering out
backspaces one by one. The backspace method, by the way, is used for
making corrections to dictated text, which works surprisingly well.

Here's another example of a method that uses our deferred generator, again in
the same class: