Here at Imaginary Landscape, we have an initative to explore solutions to some of our on-going deployment meta-problems. Those problems include things like standardizing our Django project structures and the hosting environments in which we host them. Another challenge that we have been considering is testing our sites locally.
After some initial research for existing solutions to these problems, some useful-looking candidates appeared. Specifically for provisioning and deployment, packages like Puppet and Chef revealed themselves as strong utilities that are currently gaining popularity and already have large, established communities. These packages provide a Domain Specific Language which is used to define the configuration of your servers or virtual-machines.
But alas, we are a Python shop and, for all implied reasons, we hoped that there would be an open-source Python citizen in this context. To our pleasure we discovered that the Provy project is just that. As a bonus, it is built upon the truly excellent Python library Fabric. Fabric provides a very simplistic API for manipulating remote SSH sessions. Provy uses Fabric to provide an API for describing provisioning configuration.
For local development and testing we knew we wanted some way to replicate as much of our production environment that we could. Usually, the standard answer to this is Virtualenv. However, not only Python packages come into consideration when we're talking about a full-fledged site. Some sites require non-trivial infrastructure to be in place such as rabbitmq, Celery, or non-standard databases. We settled on utilizing Vagrant virtual-machines as this would allow us to fully mock our production servers locally so that testing is authentic as possible.
So now that we have a standardized Django layout, a provisioning library (Provy), and provisioning target (Vagrant, but also our servers), what we need is a process to tie it all together. In my next post, I will explain our working implementation of a system that should streamline each of these steps and leverage more upfront investment with lasting dividends in saved time and effort with regards to provisioning, deployment, and testing.