Deferred Object In Twisted

Twisted uses the Deferred object to manage the callback sequence. The client application attaches a series of functions to the deferred to be called in order when the results of the asychronous request are available(this series of functions is known as a series of callbacks, or a callback chain), together with a series of functions to be called if there is an error in the asychronous request(known as a series of errbacks or an errback chain). The asychronous library code calls the first callback when the result is available, or the first errback when an error occurs, and the Deferred object then hands the results of each callback or errback function to the next function in the chain.

 

The Problem that Deferreds Solve

It is the second class of concurrency problem - non-computationally intensive tasks that involve an appreciable delay - that Deferreds are designed to help solve. Functions that wait on hard drive access, database access, and network access all fall into this class, although the time delay varies.

Deferreds are designed to enable Twisted programs to wait for data without hanging until that data arrives. They do this by giving a simple management interface for callbacks to libraries and applications.  Libraries know that they always make their results available by calling Deferred.callback and error by calling Deferred.errback. Applications set up result handlers by attaching callbacks and errbacks to deferreds in the order they want them called.



The basic idea behind Deferreds, and other solutions to this problem, is to keep the CPU as active as possible. If one task is waiting on data, rather than have the CPU(and the program!) idle waiting for that data(a process normally called "blocking"), the program performs other operations in the meantime, and waits for some singnal that data is ready to be processed before returning to that process.

In Twisted, a function signals to the calling function that it is waiting by returning a Deferred. When the data is available, the program activeates the callbacks on that Deferred to process the data.



Deferreds - a signal that data is yet to come

In our email sending example above, a parent functions calls a function to connect to the remote server. Asynchrony requires that this connection function return without waiting for the result so that the parent function can do other things. So how does the parent function or its controlling program know that the connection doesn't exist yet, and how does it use the connection once it does exist?

Twisted has an object that signals this situation. When the connection function returns, it signals that the operation is imcomplete by returning a twisted.internet.defer.Deferred object.

The Deferred has two purposes. The first is that it says "I am a signal that the result of whatever you wanted me to do is still pending.". The second is that you can ask the Deferred to run things when the data does arrive.



Callbacks

The way you tell a Deferred what to do with the data once it arrives is by adding a callback - asking the Deferred to call a function once the data arrives.

One Twisted library function that returns a Deferred is twisted.web.client.getPage. In this example, we call getPage, which returns a Deferred, and we attach a callback to handle the contents of the page once the data is available:

from twisted.web.client import getPage
from twisted.internet import reactor

def printContents(contents):
    '''
    This is the 'callback' function, added to the Deferred and called by it when the promised data is available
    '''

    print "The Deferred has called printContents with the following contents:"
    print contents 

    #Stop the Twisted event handling system -- this is usually handled
    #in higher lever ways

    reactor.stop() 

   #call getPage, which returns immediately with a Deferred, promissing to
   #pass the page contents onto our callbacks when the contents are available

deferred = getPage('http://www.google.com')

#added a callback to the deferred -- request that it run printContents when
#the page content has been downloaded

deferred.addCallback(printContents)

#Begin the Twisted event handling system to manage the process -- again
# this isn't the usual way to do this

reactor.run()

A very common use of Deferreds is to attach two callbacks. The result of the first callback is passed to the second callback:

from twisted.web.client import getPage
from twisted.internet import reactor

def lowerCaseContents(contents):
     '''
     This is a 'callback' function, added to the Deferred and called by
     it when the promised data is available. It converts all the data to lower
     case.
     '''

     return contents.lower()

def printContents(contents):

    '''
    This a 'callback' function, added to the Deferred after lowerCaseContents
    and called by it with the results of lowerCaseContents
    '''
    print contents
    reactor.stop()

deferred = getPage('http://www.google.com')

#add two callbacks to the deferred -- request that it run lowerCaseContents
#when the page content has been downloaded, and then run printContents with
#the result of lowerCaseContents

deferred.addCallback(lowerCaseContents)

deferred.addCallback(printContents)

reactor.run()

Error handling: errbacks

Just as a asynchronous function returns before its result is available, it may also return before it is possible to detect errors: failed connections, erroneous data, protocol errors, and so on. Just as you can add callbacks to a Deferred which it calls when the data you are expecting is available, you can add error handlers(errbacks) to a Deferred for it to call when an error occurs and it cannot obtain the data:

from twisted.web.client import getPage
from twisted.internet import reactor

def errorHandler(err):
    '''
    This is an 'errback' function, added to the Deferred which will call it in 
    the event of an error
     '''
    #this isn't a very effective handling of the error, we just print it out:
 
    print "An error has occurred: <%s>" % str(err)
    #and then we stop the entire process:
    reactor.stop()

def printContents(contents):
    '''
    This is a 'callback' function, added to the Deferred and called by it which 
    the page content
    '''
    print contents
    reactor.stop()

#we request a page which doesn't exist in order to demonstrate the
#error chain

deferred = getPage("http://www.google.com/does-not-exits")

#add the callback to the Deferred to handle the page content

deferred.addCallback(printContents)

#add the errback to the Deferred to handle any errors

deferred.addErrback(errorHandler)

reactor.run()