Automated Servers and Deployments with Ansible & Jenkins

In a previous post, Dave talked about marginal gains and how, in aggregate, they can really add up. We recently made some infrastructure improvements that I first thought would be marginal, but quickly proved to be rather significant. We started leveraging Ansible for server creation/configuration and Jenkins to automate our code deployments.

We spend a lot of time spinning up servers, configuring them and repeatedly deploying code to them. As a Drupal-focused shop, this process can get repetitive very quickly. The story usually goes something like this:

1. Project begins
2. Dev/Staging server(s) built from scratch (usually on Linode)
   1. Install Ubuntu
   2. Install Apache
   3. Install PHP
   4. Install MariaDB
   5. Install Git
   6. Install Drush
   7. Install Redis, etc.
3. Deploy project codebase from GitHub
4. Development happens
5. Pull Requests opened, reviewed, merged
6. Manually login to server via SSH, git pull
7. More development happens
8. Pull Requests opened, reviewed, merged
9. Manually login to server via SSH, git pull
10. and so on…

All of this can be visualized in a simple flowchart:

View in Google Docs

This story is repeated over and over. New client, new server new deployments. How does that old programmer’s adage go? “Don’t Repeat Yourself?” Well, we finally got around to doing something about all of this server configuration and deployment repetition nonsense. We configured a Jenkins server to automatically handle our deployments and created Ansible roles and playbooks to easily spin up and configure new servers (specifically tuned for Drupal) at will. So now our story looks something like this:

View in Google Docs

What is Ansible?

“Ansible is an IT automation tool. It can configure systems, deploy software, and orchestrate more advanced IT tasks such as continuous deployments or zero downtime rolling updates.”

Sounds like voodoo magic doesn’t it? Well, I’m here to tell you it isn’t, that it works, and that you don’t have to be a certified sysadmin to use it. Though you may need one to set it all up for you. The basic premise is that you create “playbooks” to control your remote servers. These can be as complex as a set of steps to build a LAMP server up from scratch (see below), or as simple as a specific configuration that you wish to enforce. Typically, playbooks are made up of “roles”. Roles are “reusable abstractions” as their docs page explains. You might have roles for installing Apache, adding Git, or adding a group of user’s public keys. String your roles together in a YAML file and that’s a playbook. Have a look at the official Ansible examples GitHub repo to see some real life examples.

Automate Server Creation/Configuration with Ansible

We realized we were basically building the same Drupal-tuned servers over and over. While the various steps for this process are well documented, doing the actual work takes loads of time, is prone to error and really isn’t all that fun. Ansible to the rescue! We set out to build a playbook that would build a LAMP stack up from scratch, with all the tools we use consistently across all of our projects. Here’s an example playbook:

Benefits:

• Consistent server environments: Adding additional servers to your stack is a piece of cake and you can be sure each new box will have the same exact configuration.
• Quickly roll out updates: Update your playbook and rerun against the affected servers and each will get the update. Painless.
• Add-on components: Easily tack on custom server components like Apache Solr by adding a single line to a server’s playbook.
• Allow your ops team to focus on real problems: Developers can quickly create servers without needing to bug your ops guys about how to compile PHP or install Drush, allowing them to focus on higher priority tasks.

What is Jenkins?

“Jenkins is an award-winning application that monitors executions of repeated jobs, such as building a software project or jobs run by cron.”

Think of Jenkins as a very well-trained, super organized, exceptionally good record-keeping ops robot. Train Jenkins a job once and Jenkins will repeat it over and over to your heart’s content. Jenkins will keep records of everything and will let you know should things ever go awry.

Deploy Code Automatically with Jenkins

Here’s the rundown of how we’re currently using Jenkins to automatically deploy code to our servers:

1. Jenkins listens for changes to master via the Jenkins GitHub Plugin
2. When changes are detected, Jenkins automatically kicks off a deployment by SSHing into our box and executing our standard deployment commands:
   1. cd /var/www/yourProject/docroot/
   2. git pull
   3. drush updb -y
   4. drush fra -y
   5. drush cc all
3. Build statuses are reported to Slack via the Slack Notification Plugin Here’s a full view of a configuration page for a deployment job:

The biggest benefit here is saving time. No more digging for SSH credentials. No more trying to remember where the docroot is on this machine. No more of the, “I can’t access that server, Bob usually handles…” nonsense. Jenkins has access to the server, Jenkins knows where the docroot is, and Jenkins runs the exact same deployment code every single time. The other huge win here, at least for me personally, is that it takes the worry out of deployments. Setting it up right the first time means a project lifetime of known workflow/deployments. No more worrying about if pushing the button breaks all the things. What else is great about using Jenkins to deploy your code? Here’s some quick hits:

1. Historical build data: Jenkins stores a record of every deployment. Should a deploy fail, you can see exactly when things broke down and why. Jenkins records everything that happened in a Console Output tab.
2. Empower non server admins: Jenkins users can login to Jenkins and kick off manual deployments or jobs at the push of a button. They don’t need to know how to login via ssh or even how to run a single command from the command line.
3. Enforce Consistent Workflow: By using Jenkins to deploy your code you also end up enforcing consistent workflow. In our example, drush will revert features on every single deployment. This means that devs can’t be lazy and just switch things in production. Those changes would be lost on the next deploy!
4. Status Indicators across projects: The Jenkins dashboard shows a quick overview of all of your jobs. There’s status of the last build, an aggregated “weather report” of the last few builds, last build duration, etc. Super useful.
5. Slack Integration: You can easily configure jobs to report statuses back to Slack. We have ours set to report to each project channel when a build begins and when it succeeds or fails. Great visibility for everyone on the project.

Other possible automations with Jenkins

• Automate scheduled tasks (like Drupal’s cron, mass emailing, report generation, etc.)
• Run automated tests

Both of these tools have done wonders for our workflow. While there was certainly some up-front investment to get these built out, the gains on the back end have been tremendous. We’ve gained control of our environments and their creation. We’ve taken the worry and the repetition out of our deployments. We’ve freed up our developers to focus on the work at hand. Our clients are getting their code sooner. Our team members are interrupted less often. Win after win after win. If you’re team is facing similar, consider implementing one or both of these tools. You’re sure to see similar results.