In Search of a Better Local Development Server

The problem with development environments

If you're a developer and you're like me, you have probably tried out a lot of different solutions for running development web servers. A list of the tools I've used includes:

• Full LAMP stack on an Ubuntu workstation,
• Full LAMP stack on an OS X workstation,
• XAMPP
• MAMP
• Bitnami LAMP Stack
• Acquia Dev Desktop

That's not even a complete list — I know I've also tried other solutions from the wider LAMP community, still others from the Drupal community, and I've rolled my own virtual-machine based servers too.

All of these tools have their advantages, but I was never wholly satisfied with any of them. Typically, I would encounter problems with stability when multiple sites on one server needed different configurations, or problems customizing the environment enough to make it useful for certain projects. Even the virtual-machine based solutions often suffered from the same kinds of problems — even when I experimented with version-controlling critical config files such as vhost configurations, php.ini and my.cnf files, and building servers with configuration management tools like Chef and Puppet.

Drupal VM

Eventually, I found Drupal VM, a very well-thought-out virtual machine-based development tool. It’s based on Vagrant and another configuration management tool, Ansible. This was immediately interesting to me, partly because Ansible is the tool we use internally to configure project servers, but also because the whole point of configuration management is to reliably produce identical configuration whenever the software runs. (Ansible also relies on YAML for configuration, so it fits right in with Drupal 8).

My VM wishlist

Since I've worked with various VM-based solutions before, I had some fairly specific requirements, some to do with how I work, some to do with how the Chromatic team works, and some to do with the kinds of clients I'm currently working with. So I wanted to see if I could configure Drupal VM to work within these parameters:

1. The VM must be independently version-controllable

Chromatic is a distributed team, and I don't think any two of us use identical toolchains. Because of that, we don't currently want to include any development environment code in our actual project repositories. But we do need to be able to control the VM configuration in git. By this I mean that we need to keep every setting on the virtual server outside of the server in version-controllable text files.

Version-controlling a development server in this way also implies that there will be little or no need to perform administrative tasks such as creating or editing virtual host files or php.ini files (in fact, configuration of the VM in git means that we must not edit config files in the VM since they would be overridden if we recreate or reprovision it).

Furthermore, it means that there's relatively little need to actually log into the VM, and that most of our work can be performed using our day-to-day tools (i.e. what we've configured on our own workstations, and not whatever tools exist on the VM).

2. The VM must be manageable as a git submodule

On a related note, I wanted to be able to add the VM to the development server repository and never touch its files—I'm interested in maintaining the configuration of the VM, but not so much the VM itself.

It may help to explain this in Drupal-ish terms; when I include a contrib module in a Drupal project, I expect to be able to interact with that module without needing to modify it. This allows the module to be updated independently of the main project. I wanted to be able to work with the VM in the same way.

3. The VM must be able to be recreated from scratch at any time

This is a big one for me. If I somehow mess up a dev server, I want to be able to check out the latest version of the server in git, boot it and go back to work immediately. Specifically, I want to be able to restore (all) the database(s) on the box more or less automatically when the box is recreated.

Similarly, I usually work at home on a desktop workstation. But when I need to travel or work outside the house, I need to be able to quickly set up the project(s) I'll be working on on my laptop.

Finally, I want the VM configuration to be easy to share with my colleagues (and sometimes with clients directly).

4. The VM must allow multiple sites per server

Some of the clients we work with have multiple relatively small, relatively similar sites. These sites sometimes require similar or identical changes. For these clients, I much prefer to have a single VM that I can spin up to work on one or several of their sites at once. This makes it easier to switch between projects, and saves a great deal of disk space (the great disadvantage to using virtual machines is the amount of disk space they use, so putting several sites on a single VM can save a lot of space).

And of course if we can have multiple sites per server, then we can also have a single site per server when that's appropriate.

5. The VM must allow interaction via the command line

I've written before about how I do most of my work in a terminal. When I need to interact with the VM, I want to stay in the terminal, and not have to find or launch a specific app to do it.

6. The VM must create drush aliases

The single most common type of terminal command for me to issue to a VM is drush @alias {something}. And when running the command on a separate server (the VM!), the command must be prefixed with an alias, so a VM that can create drush aliases (or help create them) is very, very useful (especially in the case where there are multiple sites on a single VM).

7. The VM must not be too opinionated about the stack

Given the variations in clients' production environments, I need to be able to use any current version of PHP, use Apache or Nginx, and vary the server OS itself.

My VM setup

Happily, it turns out that Drupal VM can not only satisfy all these requirements, but is either capable of all of them out of the box, or makes it very straightforward to incorporate the required functionality. Items 4, 5, 6, and 7, for example, are stock.

But before I get into the setup of items 1, 2, and 3, I should note that this is not the only way to do it.

Drupal VM is a) extensively documented, and b) flexible enough to accommodate any of several very different workflows and project structures than what I'm going to describe here. If my configuration doesn't work with your workflow, or my workflow won't work with your configuration you can probably still use Drupal VM if you need or want a VM-based development solution.

For Drupal 8 especially, I would simply use Composer to install Drupal, and install Drupal VM as a dependency.

Note also that if you just need a quick Drupal box for more generic testing, you don't need to do any of this, you can just get started immediately.

Structure

We know that Drupal VM is based on Ansible and Vagrant, and that both of those tools rely on config files (YAML and ruby respectively). Furthermore, we know that Vagrant can keep folders on the host and guest systems in sync, so we also know that we'll be able to handle item 1 from my wishlist--that is, we can maintain separate repositories for the server and for projects.

This means we can have our development server as a standalone directory, and our project repositories in another. For example, we might set up the following directory structure where example.com contains the project repository, and devserver contains the Drupal VM configuration.

Servers
└── devserver/
Sites
└── example.com/

Configuration files

Thanks to some recent changes, Drupal VM can be configured with an externalconfig.ymlfile, a local config.yml file, and an externalVagrantfile.localfile using a delegatingVagrantfile.

The config.yml file is required in this setup, and can be used to override any or all of the default configuration in Drupal VM's own default.config.yml file.

The Vagrantfile.local file is optional, but useful in case you need to alter Drupal VM's default Vagrant configuration.

The delegating Vagrantfile is the key to tying together our main development server configuration and the Drupal VM submodule. It defines the directory where configuration files can be found, and loads the Drupal VM Vagrantfile.

This makes it possible to create the structure we need to satisfy item 2 from my wishlist --that is, we can add Drupal VM as a git submodule to the dev server configuration:

Server/
├── Configuration/
|   ├── config.yml
|   └── Vagrantfile.local
├── Drupal VM/
└── Vagrantfile

Recreating the VM

One motivation for all of this is to be able to recreate the entire development environment quickly. As mentioned above, this might be because the VM has become corrupt in some way, because I want to work on the site on a different computer, or because I want to share the site—server and all—with a colleague.

Mostly, this is simple. To the extent that the entire VM (along with the project running inside it!) is version-controlled, I can just ask my colleague to check out the relevant repositories and (at most!) override the vagrant_synced_folders option in alocal.config.yml with their own path to the project directory.

In checking out the server repository (i.e. we are not sharing an actual virtual disk image), my colleague will get the entire VM configuration including:

• Machine settings,
• Server OS,
• Databases,
• Database users,
• Apache or Nginx vhosts,
• PHP version,
• php.ini settings,
• Whatever else we've configured, such as SoLR, Xdebug, Varnish, etc.

So, with no custom work at all—even the delegating Vagrant file comes from the Drupal VM documentation—we have set up everything we need, with two exceptions:

1. Entries for /etc/hosts file, and
2. Databases!

For these two issues, we turn to the Vagrant plugin ecosystem.

/etc/hosts entries

The simplest way of resolving development addresses (such as e.g. example.dev) to the IP of the VM is to create entries in the host system's /etc/hosts file:

192.168.99.99  example.dev

Managing these entries, if you run many development servers, is tedious.

Fortunately, there's a plugin that manages these entries automatically, Vagrant Hostsupdater. Hostsupdater simply adds the relevant entries when the VM is created, and removes them again (configurably) when the VM is halted or destroyed.

Databases

Importing the database into the VM is usually a one-time operation, but since I'm trying to set up an easy process for working with multiple sites on one server, I sometimes need to do this multiple times — especially if I've destroyed the actual VM in order to save disk space etc.

Similarly, exporting the database isn't an everyday action, but again I sometimes need to do this multiple times and it can be useful to have a selection of recent database dumps.

For these reasons, I partially automated the process with the help of a Vagrant plugin. "Vagrant Triggers" is a Vagrant plugin that allows code to be executed "…on the host or guest before and/or after Vagrant commands." I use this plugin to dump all non-system databases on the VM on vagrant halt, delete any dump files over a certain age, and to import any databases that can be found in the dump location on the first vagrant up.

Note that while I use these scripts for convenience, I don't rely on them to safeguard critical data.

With these files and a directory for database dumps to reside in, my basic server wrapper now looks like this:

Server/
├── Vagrantfile
├── config/
├── db_dump.sh
├── db_dumps/
├── db_import.sh
└── drupal-vm/

My workflow

New projects

All of the items on my wishlist were intended to help me achieve a specific workflow when I needed to add a new development server, or move it to a different machine:

1. Clone the project repo.
2. Clone the server repo.
3. Change config.yml:
   • Create/modify one or more vhosts.
   • Create/modify one or more databases.
   • Change VM hostname.
   • Change VM machine name.
   • Change VM IP.
   • Create/modify one or more cron jobs.
4. Add a database dump (if there is one) to the db_dumps directory.
5. Run vagrant up.

Sharing projects

If I share a development server with a colleagues, they have a similar workflow to get it running:

1. Clone the project repo.
2. Clone the server repo.
3. Customize local.config.yml to override my settings:
   • Change VM hostname (in case of VM conflict).
   • Change VM machine name (in case of VM conflict).
   • Change VM IP (in case of VM conflict).
   • Change vagrant synced folders local path (if different from mine).
4. Add a database dump to the dumps directory.
5. Run vagrant up.

Winding down projects

When a project completes that either has no maintenance phase, or where I won't be involved in the ongoing maintenance, I like to remove the actual virtual disk that the VM is based on. This saves ≥10GB of hard drive space (!):

vagrant destroy

But since a) every aspect of the server configuration is contained in config.yml and Vagrantfile.local, and b) since we have a way of automatically importing a database dump, resurrecting the development server is as simple as pulling down a new database and re-provisioning the VM:

scp remotehost:/path/to/dump.sql.gz /path/to/Server/db_dumps/dump.sql.gz
vagrant up