In September I was fortunate to be able to give a talk at DjangoCon US, the largest annual conference about the Django web framework. The topic of the talk was on Vagrant, a free open-source tool facilitating the manipulation of virtualized environments, and how it may benefit the development of Django applications. This talk was aimed at Django developers of all levels who were interested in getting an overview of the great possibilities Vagrant offers to support teamwork and quality assurance. In this article, I provide the same overview of the issues and solutions as addressed in the talk, albeit in broader terms that should be compelling to all kinds of web development teams, regardless of their framework or programming language of choice.
Common issues for web development teams
Over the last couple of years, a number of things have contributed to making the lives of developers easier, in particular the emergence of cloud computing, of both backend and frontend testing frameworks, and of distributed version control systems such as Git. However, the web stack specifications have also become more complex in order to provide the features that are now commonly expected by web users and to respond to ever increasing levels of traffic on modern web sites and applications. Indeed, it is now often necessary to use components such as RabbitMQ, Hadoop, Solr or Redis, just to name a few. As a result, web teams nowadays tend to be recurrently confronted with two types of issues:
- Code sometimes unexpectedly breaks in production whereas it worked on development machines prior to being deployed. This type of unanticipated problems is often due to the inconsistencies that exist between the team members’ platforms, and to the inconsistencies between the development and production environments. Indeed, many web developers nowadays use Mac and Windows computers as their primary development platform, whereas the vast majority of websites eventually get deployed to Linux machines.
- On-boarding new developers tends to be difficult and time-consuming. This is true especially if the person that you are on-boarding does not have much experience with systems administration, for example a frontend developer or a designer. If you are well organized, you might have long wiki pages describing every step required to manually install all of the technical requirements for your project. However, keeping those wiki pages up to date is hard, tedious and uninteresting, and so is having to manually execute all the required steps every single time a new person joins your team. What’s more, the web developers who work on Mac or Windows platforms often have to go through extra pain as the installation of certain software, such as MySQL or PostgreSQL for example, are generally much more difficult on their platforms than it is on Linux platforms. The net result is that developers end up having to spend a lot of time administering their systems, whereas they would rather spend this time writing code and building features.
Vagrant to the rescue
The issues described above would not occur in an ideal world where all developers of a same team worked on the same, pre-built platform: there would be no need to manually install anything and the working environment would automatically be assembled so that developers may solely focus on their development tasks. In that ideal world, the development and production platforms would also closely match so that no more bad surprise would occur when new code got pushed live. The good news is that this world already exists — and it’s called virtualization.
Virtualization is nothing new, really. It has been around for a long time. However, in the last couple of years it has become much more approachable and useful to developers, especially thanks to a project called Vagrant. Essentially, Vagrant is a developer-friendly interface for managing virtual machines (VM). Currently the only VM manager supported by Vagrant is VirtualBox, but there are plans for it to support VMWare in the future as well. In a nutshell, with Vagrant the workflow for a given developer is as follows:
The developer interacts with Vagrant by running a few simple commands. The main command is ‘vagrant up’ (1) for booting the VM up, that is the equivalent of pressing the ‘on’ button of a physical computer. The developer also has to provide a configuration file named ‘Vagrantfile’ (1), which contains some basic directives to specify the characteristics of the VM that needs to be built. Then Vagrant will talk to VirtualBox (2) and instruct it to actually create the VM (3). Vagrant will also talk to some provisioners (4), which are responsible for provisioning the VM, that is, installing and configuring all the software required for running the project’s application (5). Once the VM has been created, booted and provisioned, the developer may directly access the VM (6), for example via SSH, exactly in the same way as they would access any remote server, except that in this case the server actually lives in a local VM inside the developer’s host machine.
Multiple types of provisioners may be used in conjunction with Vagrant. Chef and Puppet, both ruby-based projects, are probably the most popular ones. Salt Stack, a more recent project, is also quickly gaining popularity, in particular amongst the Python community. Regular shell scripts can also be used, so if you are familiar with writing shell commands then you will feel right at home. Chef, Puppet and Salt Stack can be regarded as provisioning frameworks in that they abstract out the logic for installing and configuring software on multiple types of platforms (e.g. Ubuntu, Debian, Solaris or AWS), allowing you to reuse the same provisioning scripts across any platforms needed for your project. Those provisioners use different syntaxes and follow slightly different paradigms but they all are very powerful. So, choosing a provisioner for your project really comes down to personal taste. Personally I have mostly used Chef as it has worked really well for me so far. I recommend trying them all out and selecting the one that feels the most comfortable to work with.
The Vagrantfile is what glues everything together. It references all the provisioning scripts and lists the specifications for the VM’s platform, for example what operating system the VM should be based on or how much memory it should be allocated. It also specifies the folders to be shared between the VM and the host machine — this is very important as it allows in particular to share the folder containing the project’s source files so that you may modify the code using your favourite editor on your host machine and let the VM access those same files to serve your application.
New team workflow
So, you have decided to give Vagrant a try and you are now wondering about how your team workflow will evolve. Here is how it could be summarized:
- First of all, everybody in your team (whether they use a Linux, Windows or Mac computer) will need to install Vagrant and VirtualBox. This only needs to be done once and is really easy as both tools offer a one-click install process.
- At least one person in your team will be responsible for writing the Vagrantfile and the provisioning scripts. All of those configuration files need to be checked in the project’s code repository so that all team members can easily access them.
- Once a team member has checked out the repository on their computer, all they need to do is run the ‘vagrant up’ command. This command will then follow all the instructions specified in the Vagrantfile and in the provisioning scripts to automatically build a new VM with all the software requirements. Depending on the complexity of the web stack for your project, this step may take between 10 and 30 minutes. Quite importantly, this step would likely not require any input from the user, so it can be left running in the background or during your coffee break. Then, when the initial build of the VM is complete, every subsequent call to ‘vagrant up’ for booting the VM will only take around 20 seconds. Once the VM is built and booted, the entire development environment will be set up and the team member will be ready to get going with their work.
- If the configuration requirements have to change during the lifecycle of the project, then modifications can be applied to the Vagrantfile or to the provisioning scripts and be pushed up the repository. Then, depending on the nature of the change, each team member’s VM may need to be re-booted or re-built in order to automatically bring everybody’s development environment in sync.
Finally, the individual workflow for each team member will remain similar to what they were used to in the past, as they may continue to edit code with their favourite editor or IDE and continue to browse their development copy of the web application using their favourite browser on their host machine.
In closing
When I asked Mitchell Hashimoto, the creator of Vagrant, how he chose the term “Vagrant” to name this tool, he replied, “The idea is that developers would be working in these transient, portable boxes. They would move around and would have no permanent residence. Like a vagrant.” This is a really important concept as the process driven by Vagrant allows the development environment to be automated and reproducible. In particular, this system scales really well as the steps for creating the development environment will remain simple for all team members over time, regardless of the current or future size of your team. Also, in the case where something accidentally breaks in any given person’s VM (for example, some software get corrupted or some important configuration files get deleted), then it is easy to tear down the VM and re-create it from scratch to start again with a pristine working environment. Having the entire software installation’s instructions coded in the provisioning scripts is also an opportunity to ensure that every team member works in an environment that reflects that of the production server, since the same provisioning scripts may be used for building the production platform.
Vagrant has completely transformed the way I approach teamwork for web development and I now systematically use it in all my projects. I recommend you try it and see what benefits this tool can bring to your own projects!