Our Strategy to Migrate to Django

In 2010, the Eventbrite engineering team began early work on an effort to move to the Django web-framwork. At the time, Eventbrite was built using some in-house frameworks, that a layman might describe as no framework.

We believed that in the long-term, moving to a feature-rich framework with a vibrant community would be a huge help to productivity and hiring. We were right.

In the short-term, some of the immediate benefits of moving to Django were:

  • unit-test framework
  • database model abstraction (Django ORM)
  • request middleware
  • URL routing
  • form building
  • python-based template language and i18n support

Moving from no framework to Django was an obvious choice, but it’s much easier to say than to do. Even in 2010, Eventbrite was powered by hundreds of thousands of Python and template code. Simply re-writing it would mean shutting progress down for months, and even then, the task would be to painstakingly replicate every feature in brand-new Django code. In other words, re-writing this much code is an option, but not one that many startups would make.

Homepage on Django

As a first step, the 2010 version of the Eventbrite homepage was written entirely from scratch using Django. The existing database tablets were introspected, and turned in to Django ORM code. Templates were written in Django’s template language (we now use a combination of Mako and various client-side template frameworks). It was beautiful.

This homepage was deployed on a brand-new set of servers, and the traffic was sent there from our main load-balancers (split between django servers and legacy “core” servers). It worked great.

figure-01

However, at Eventbrite, the homepage and a feature page were different animals. This effort showed us using Django could be great, but having separate code-bases would make re-using old code nearly impossible, and thus would have to be re-written. Most of our unit-tests and web-driver have been written since that time, so we would have to verify our porting efforts by hand.

We needed a path forward that didn’t segments our code and servers, and let us reap the benefits of Django as

Python Glue Layer

The answer to this problem was to create a virtualization or adaptation environment that let our CORE code run on top of Django. This would let us get the immediate benefits above, while allowing us to port functionality on an as-needed basis. We call this virtualization “Django-Core” (or “DJ-Core” for short).

figure-02

Under the adaptation environment, URL routing passed through Django’s router, and if it didn’t find a match it set up the DJ-CORE adaptation environment and passed control to CORE. The CORE code could run on both types of machines, and due to the adaptation, it didn’t “know” what kind of machine it was on.

We combined our core and django git repositories (see git filter-branch) and continued to run both versions of the site on separate sets of machines. After a few weeks of verification, we were able to slowly point CORE URLs from to the DJ-CORE.

figure-03

Eventually the CORE servers we decommissioned. After this point, all code was known to be running under Django, and engineers were free to start using Django features directly.

Our final step was to port all of our templates from Clearsilver to Mako. See a future blog post for details.

Details on glue code

There were a few ways we could accomplish the migration listed above. The method we chose was to trick the CORE code, when running under DJ-CORE. There was set of classes and modules, in our CORE application, that interacted with the system and would not work under Django. We wrote new versions of classes and modules, that had the same interface, but did the right thing under Django (frequently these were subclasses of the originals); we called them “Adapators”. When running under Django, those modules and classes were replaced using the method specified in PEP-0302 “New Import Hooks.”

Image our CORE code as a car that uses tires and drives on the pavement. Imaging Django code as running on tracks with steel wheels. When CORE code would import a tire object, we would hand it a similar looking object that under-the-covers actually works on train-tracks.

Here’s a simplified version of the import-hook we wrote (it is definitely incomplete, but you can use it for inspiration). Think of this as a kind of dependency injection framework.

"""Importer hook that allows arbitrary module replacement."""

import sys

class ModuleReplacer(object):
    def __init__(self, *args, **kargs):
        self.__module_map = {}
        sys.meta_path.append(self)

    def add(self, *args, **kwargs):
        return self.update(*args, **kwargs)

    def update(self, from_name, to_module, original=None):
        self.__module_map[from_name] = to_module

        try:
            del sys.modules[from_name]
        except KeyError:
            pass

    def find_module(self, module_name, package_path=None):
        if module_name in self.__module_map:
            return self

    def load_module(self, module_name):
        try:
            to_module = self.__module_map[module_name]
            return to_module
        except KeyError:
            return None

    def replace(self, from_name, to_name):
        """simplifies replacing modules from the caller's perspective

        XXX: Python's flexible import syntax provides multiple ways to
        spell a given import, and this makes it difficult to
        completely replace a module.

        For example, if you invoke ModuleReplacer like this:

        ModuleReplacer.replace(
            'foo.Bar',
            'bar.BarStub'
        )

        This import will give you the replacement module
            import foo.Bar

        But this import will give you the original module Bar:
            from foo import Bar

        """
        def imp(name):
            if '.' in name:
                mod = __import__(
                    name,
                    # a.b.c => ('a.b',)
                    fromlist=('.'.join(name.split('.')[:-1]),),
                )
            else:
                mod = __import__(name)
            return mod
        from_mod = imp(from_name)
        to_mod = imp(to_name)
        self.update(from_name, to_mod, original=from_mod)

ModuleReplacer = ModuleReplacer()

Note that if your replacement module needs access to the original, you will have to work around this import-hook. We did it by importing things in the right order, and by saving the originals (omitted for brevity).

Finally

After conversion to DJ-CORE, we ripped out most of these adaptations and our application has been migrating to be a proper Django app. Since then, we have been moving many parts of our application service-oriented-architecture. The ModuleReplacer lives on to help us with dependency-injection tasks, mostly in special testing environments.