Restarting a Twisted Reactor

I recently started using twisted a couple of weeks ago. For those who don't know, twisted is "event-driven networking engine written in Python". The learning curve is pretty steep if you've never done asynchronous programming before. During the project I was working on, I ran into a condition where I thought I needed to restart the twisted reactor. According to everything I found online, restarting the reactor is not supported. But I can be stubborn so I tried to find a way anyway.

Restarting a Twisted Reactor

Let's start by creating a pretty standard twisted server. We'll subclass LineReceiver which basically makes a TCP server that accepts full lines of text, although it can also do raw data too. Let's take a look at the code:

from twisted.internet import reactor
from twisted.protocols.basic import LineReceiver
from twisted.internet.protocol import Factory

PORT = 9000

class LineServer(LineReceiver):

    def connectionMade(self):
        """
        Overridden event handler that is called when a connection to the 
        server was made
        """
        print "server received a connection!"

    def connectionLost(self, reason):
        """
        Overridden event handler that is called when the connection 
        between the server and the client is lost
        @param reason: Reason for loss of connection
        """
        print "Connection lost"
        print reason

    def lineReceived(self, data):
        """
        Overridden event handler for when a line of data is 
        received from client
        @param data: The data received from the client
        """
        print 'in lineReceived'
        print 'data => ' + data


class ServerFactory(Factory):
    protocol = LineServer


if __name__ == '__main__':
    factory = ServerFactory()
    reactor.listenTCP(PORT, factory)
    reactor.run()

All of the camel-cased methods are overridden twisted methods. I just stubbed them out and have them print to stdout whenever they get called. Now let's make a client that has a reactor that we restart a few times:

import time
import twisted.internet

from twisted.internet import reactor, protocol
from twisted.protocols.basic import LineOnlyReceiver

HOST = 'localhost'
PORT = 9000

class Client:
    """
    Client class wrapper
    """
    def __init__(self, new_user):
        self.new_user = new_user

        self.factory = MyClientFactory()

    def connect(self, server_address=HOST):
        """
        Connect to the server
        @param server_address: The server address
        """
        reactor.connectTCP(server_address, PORT, self.factory,
            timeout=30)

class MyProtocol(LineOnlyReceiver):

    def connectionMade(self):
        """
        Overridden event handler that is fired when a connection
        is made to the server
        """
        print "client connected!"
        self.run_long_running_process()

    def lineReceived(self, data):
        """
        Gets the data from the server
        @param data: The data received from the server
        """
        print "Received data: " + data

    def connectionLost(self, reason):
        """
        Connection lost event handler
        @param reason: The reason the client lost connection 
            with the server
        """
        print "Connection lost"

    def run_long_running_process(self):
        """
        Run the process
        """
        print 'running process'
        time.sleep(5)
        print "process finished!"
        self.transport.write('finished' + '\r\n')
        reactor.callLater(5, reactor.stop)

class MyClientFactory(protocol.ClientFactory):
    protocol = MyProtocol


if __name__ == '__main__':
    # run reactor multiple times
    tries = 3
    while tries:
        client = Client(new_user=True)
        client.connect('localhost')
        try:
            reactor.run()
            tries -= 1
            print "tries " + str(tries)
        except Exception, e:
            print e
            import sys
            del sys.modules['twisted.internet.reactor']
            from twisted.internet import reactor
            from twisted.internet import default
            default.install()

Here we create a simple client that accepts only lines of text too (LineOnlyReceiver). The magic for restarting the reactor lies in the while loop at the end of the code. I actually found the code in the exception handler inside of twisted's reactor.py file which gave me the idea. Basically what we're doing is importing Python's sys module. We then delete the reactor from sys.modules which allows us to re-port it and reinstall the default reactor. If you run the server in one terminal and the client in another, you can see the client reconnect three times.

Wrapping Up

As I mentioned at the beginning, I'm still pretty new to twisted. What you should probably do instead of restarting the reactor is run it in another thread. Or you might use one of its delayed calls or deferred threads to get around the "need" to restart the reactor. Frankly, the method in this article doesn't even work in some conditions. I was actually trying to restart the reactor once inside a function that was decorated with contextlib's contextmanager decorator and that somehow prevented this code to work correctly. Regardless, I thought it was an interesting way to reload a module.