Ansible?
One of my roles at Chromatic is as a member of the devops team. Among other things, this involves working with our servers and our clients' servers. This in turn means I spend a lot of time working with Ansible, a popular tool for provisioning, configuring and deploying servers and applications.
In very simple terms, a machine running Ansible (usually) runs commands on another machine, (usually) over SSH. These commands are specified declaratively (not imperatively), using small parcels of YAML called tasks. These tasks invoke Ansible modules that specialize in performing options with specific components such as files, databases etc.
For example, the following task uses the File module (docs, code) to create a specific directory if it doesn't already exist, and changes its attributes if they are not already correctly set:
- file:
path: /home/jenkins/.ssh
state: directory
owner: jenkins
group: jenkins
mode: 700Multiple tasks related to a single concern are grouped together into roles, and multiple roles can be grouped together into playbooks. A playbook can then be used to perform the exact same configuration steps on any number servers simultaneously.
Declarative?
Ansible tasks are written declaratively, meaning we don't specify which underlying implementation must be used to accomplish the task. This is useful since it allows for a high level of abstraction, very readable and relatively easy-to-write code, and in some cases, allows us to use the same task on different platforms. For example, there’s Ansible’s Copy module which is used to copy files to the destination machine. In the following task, Ansible copies a configuration file to the correct directory on the remote machine and sets the file’s owner, group, and file permissions:
- name: Copy SSH config file into Alice’s .ssh directory.
copy:
src: files/config
dest: /home/alice/.ssh/config
owner: alice
group: alice
mode: 0600To accomplish the same result, we could, for example, have written a series of commands, or a function in bash using scp, chown, and chmod. With Ansible, we can focus on the desired configuration without being overly concerned by the details.
On the other hand, it also means that the available tools sometimes seem strange or unusual—mainly since developers usually have access to imperative tools for those cases when no declarative option is suitable.
One place I noticed this in Ansible was when repeatedly performing the same task with a set of different items. Specifically, I found Ansible's looping tools a little bizarre, not least because there are sixteen of them—compare to PHP which has four kinds of loops.
There's actually a reason for this if you're interested in Ansible's internals. The Loops page in the documentation points out that, "[l]oops are actually a combination of things with_ + lookup(), so any lookup plugin can be used as a source for a loop." Lookups are a type of Ansible plugin used to "access data in Ansible from outside sources," and if you compare the Loops documentation and Ansible's plugin directory on GitHub you'll see many of the same names.
However, Ansible's documentation refers to lookups as an "advanced topic," and it's not necessary to delve into the source code to learn to use the loops themselves. The remainder of this post outlines several of Ansible's more commonly used loops, along with some of the things I've learned about how to use them.
Ansible's Loops
The tasks performed in the following examples are more or less arbitrary examples related to creating users and their directories, but they are closely related to actual tasks that might need to be performed on production servers (but note: the data we have to work with and the number of tasks we use to achieve the proper configuration are obviously unrealistic!)
The examples build on one another to perform the following simple tasks on a hypothetical server:
- Ensure four users are present,
alice,bob,carol, anddan. - Ensure that each user's home directory contains two directories,
.ssh/andloops/. - Ensure that each of the four users' home directories contains one directory for each other user. For example, the user
alice's home directory should look like this when we're done:
/home/alice/
├── .ssh/
├── bob/
├── carol/
├── dan/
└── loops/Loop 1: create users using with_items
A typical task in Ansible might look something like this one, which removes the user ‘chuck’ from the system(s) the task runs on:
- name: Remove user ‘Chuck’ from the system.
user:
name: chuck
state: absent
remove: yesTo repeat this task for multiple users—say we need to remove the users ‘Chuck’ and ‘Craig’—we simply add a with_items parameter to the task. with_items accepts either a list (shown here), or a variable (as in the rest of the following examples):
- name: Remove users ‘Chuck’ and ‘Craig’ from the system.
user:
name: "{{ item }}"
state: absent
remove: yes
with_items:
- chuck
- craigReturning to our first loop example, we can use with_items to create the first users in our list, ‘alice’ and ‘bob’:
Loop 1: Variables
users_with_items:
- name: "alice"
personal_directories:
- "bob"
- "carol"
- "dan"
- name: "bob"
personal_directories:
- "alice"
- "carol"
- "dan"Loop 1: Tasks
- name: "Loop 1: create users using 'with_items'."
user:
name: "{{ item.name }}"
with_items: "{{ users_with_items }}"Here, we use Ansible's User module to loop over a variable named users_with_items. This variable contains names and information about two users, but the task only ensures the users exist on the system, it does not create directories contained in each user's list of personal_directories (note that personal_directories is just an arbitrary key in the array of data for our examples).
This is a notable feature of Ansible's loops (and of Ansible in general): since tasks call specific modules with a specific area of concern, it's not usually possible to do more than one kind of thing in a task. In this specific case, that means we can't make sure the user's personal_directories exist from this task (i.e. because we’re using the User module not the File module).
The with_items loop works essentially like this PHP loop:
<?php
foreach ($users_with_items as $user) {
// Do something with $user...
}We've written the task as usual, except that:
- We've substituted the variable name
item.namefor the username, and - We've added the
with_itemsline specifying the variable to iterate over.
It's also worth noting that from inside Ansible's loop's the current iteration is always item, and any given property of it is accessed with item.property.
Loop 1: Results
/home/
├── alice/
└── bob/Loop 2: create common users' directories using with_nested
In this example, we use two variables, Loop 1's users_with_items, and a new one, common_directories which is a list of all the directories that should be present in every user's directory. This means that (returning to PHP again), we need something that works like this:
<?php
foreach ($users_with_items as $user) {
foreach ($common_directories as $directory) {
// Create $directory for $user...
}
}In Ansible, we can use the with_nested type of loop. with_nested loops take two lists, the second of which is iterated over during each iteration of the first:
Loop 2: Variables
users_with_items:
- name: "alice"
personal_directories:
- "bob"
- "carol"
- "dan"
- name: "bob"
personal_directories:
- "alice"
- "carol"
- "dan"
common_directories:
- ".ssh"
- "loops"Loop 2: Tasks
# Note that this does not set correct permissions on /home/{{ item.x.name }}/.ssh!
- name: "Loop 2: create common users' directories using 'with_nested'."
file:
dest: "/home/{{ item.0.name }}/{{ item.1 }}"
owner: "{{ item.0.name }}"
group: "{{ item.0.name }}"
state: directory
with_nested:
- "{{ users_with_items }}"
- "{{ common_directories }}"As the task above shows, the two lists in with_nested can be accessed using item.0 (for users_with_items) and item.1 (for common_directories) respectively. This allows us to e.g. create the directory /home/alice/.ssh on the very first iteration.
Loop 2: Results
/home/
├── alice/
│ ├── .ssh/
│ └── loops/
└── bob/
├── .ssh/
└── loops/Loop 3: create personal users' directories using with_subelements
In this example, we use another kind of nested loop, with_subelements to create the directories listed in the users_with_items variable from Loop 1. In PHP, the loop might look something like this:
<?php
foreach ($users_with_items as $user) {
foreach ($user['personal_directories'] as $directory) {
// Create $directory for $user...
}
}
Note that we are looping over the $users_with_items array and $user['personal_directories'] for each user.
Loop 3: Variables
users_with_items:
- name: "alice"
personal_directories:
- "bob"
- "carol"
- "dan"
- name: "bob"
personal_directories:
- "alice"
- "carol"
- "dan"Loop 3: Tasks
- name: "Loop 3: create personal users' directories using 'with_subelements'."
file:
dest: "/home/{{ item.0.name }}/{{ item.1 }}"
owner: "{{ item.0.name }}"
group: "{{ item.0.name }}"
state: directory
with_subelements:
- "{{ users_with_items }}"
- personal_directoriesThe with_subelements loop works almost exactly the same way as with_nested, except that instead of a second variable, it takes a variable and the key of another list contained in that variable—in this case, personal_directories. Similarly to Loop 2, the first iteration of this loop creates (or verifies the existence of) /home/alice/bob.
Loop 3: Results
/home/
├── alice/
│ ├── .ssh/
│ ├── bob/
│ ├── carol/
│ ├── dan/
│ └── loops/
└── bob/
├── .ssh/
├── alice/
├── carol/
├── dan/
└── loops/Loop 4: create users using with_dict
Loop 3 completed the setup of the home directories belonging to alice and bob, but there are still two outstanding users to create, carol and dan. This example creates those users using a new variable, users_with_dict and Ansible's with_dict loop.
Note that the data structure here contains meaningful keys (dict or dictionary is Python's name for an associative array); with_dict can be the best option if you are compelled to use data with this type of structure. The loop we create here in Ansible is rather like this in PHP:
<?php
foreach ($users_with_dict as $user => $properties) {
// Create a user named $user...
}Loop 4: Variables
users_with_dict:
carol:
common_directories: "{{ common_directories }}"
dan:
common_directories: "{{ common_directories }}"Loop 4: Tasks
- name: "Loop 4: create users using 'with_dict'."
user:
name: "{{ item.key }}"
with_dict: "{{ users_with_dict }}"The with_dict type of loop is quite brief and allows access to the variable's keys and the corresponding values. Unfortunately, it has one practical drawback, namely that it is not possible to loop over the subelements of a dict using with_dict (so, e.g. we can not use with_dict to create each user’s common directories).
Loop 4: Results
/home/
├── alice/
│ ├── .ssh/
│ ├── bob/
│ ├── carol/
│ ├── dan/
│ └── loops/
├── bob/
│ ├── .ssh/
│ ├── alice/
│ ├── carol/
│ ├── dan/
│ └── loops/
├── carol/
└── dan/Loop 5: create personal directories if they don't exist
Since we can't easily use the users_with_dict, we need to make use of Ansible's available tools to do it in a different way. Since we have now created the required users alice, bob, carol, and dan, we can re-use the with_nested loop along with the contents of the /home/ directory. This example uses several new non-loop features to show how loops can be integrated into relatively complex tasks:
Loop 5: Variables
common_directories:
- ".ssh"
- "loops"Loop 5: Tasks
- name: "Get list of extant users."
shell: "find * -type d -prune | sort"
args:
chdir: "/home"
register: "home_directories"
changed_when: false
- name: "Loop 5: create personal user directories if they don't exist."
file:
dest: "/home/{{ item.0 }}/{{ item.1 }}"
owner: "{{ item.0 }}"
group: "{{ item.0 }}"
state: directory
with_nested:
- "{{ home_directories.stdout_lines }}"
- "{{ home_directories.stdout_lines | union(common_directories) }}"
when: "'{{ item.0 }}' != '{{ item.1 }}'"Here we have two tasks, one using the shell module to execute a find command on the server, and another using the file module to create directories.
When executed in the directory /home, the command find * -type d -prune | sort (performed by the shell module) will return only the names of the directories found inside /home—in other words, the names of all the users whose directories need to be provisioned.
The output of this command is stored in a variable, home_directories, by the register: "home_directories" line in the task. The important part of this variable—which we will use in the subsequent task—looks like this:
"stdout_lines": [
"alice",
"bob",
"carol",
"dan",
],
The second task in this example (the actual loop) is almost exactly the same as the with_nested loop in the second example, but there are two differences to note:
-
The second line under
with_nestedlooks a bit unusual:- "{{ home_directories.stdout_lines | union(common_directories) }}" -
There's another line beginning with
whenat the end of the task:when: "'{{ item.0 }}' != '{{ item.1 }}'"
Let's go through these one at a time. The odd line under with_nested is applying a Jinja2 filter to the new list of directories from the first task above (that's the home_directories.stdout_lines part). The basic syntax of Jinja's filters is:
- object to filter (
home_directories.stdout_lines) - apply filter (
|) - filter name plus arguments, if any (
union(common_directories))
So in other words, we're using the filter to combine home_directories.stdout_lines and the common_directories variable from the beginning of this example into a single array:
item:
- .ssh
- alice
- bob
- carol
- dan
- loopsThis means that our with_nested loop will loop through each of home_directories.stdout_lines (the first with_nested line), and ensure that each of the directories in the second line exists in each user's home directory.
Unfortunately, that would give us the wrong output—if we relied on the loop alone, we'd find that each user's home directory would contain a directory with the same name as the home directory! (i.e. /home/alice/alice, /home/bob/bob etc.) This is where Ansible conditionals—the when—come in:
when: "'{{ item.0 }}' != '{{ item.1 }}'"This line prevents the task from creating a directory when the current item in home_directories.stdout_lines and the current item in our union of home_directories.stdout_lines are identical (as the Ansible Loops documentation points out, "...when combining when with with_items (or any other loop statement), the when statement is processed separately for each item"). In PHP, what we're doing in the second task would look something like this:
<?php
$users = ['alice', 'bob', 'carol', 'dan'];
$common_directories = ['.ssh', 'loops'];
$directories = $user + $common_directories;
foreach ($users as $user) {
foreach ($directories as $directory) {
if ($directory != $user) {
// Create the directory…
}
}
}This gives us the set of results shown below, and completes the provisioning of our test example.
Loop 5: Results
/home/
├── alice/
│ ├── .ssh/
│ ├── bob/
│ ├── carol/
│ ├── dan/
│ └── loops/
├── bob/
│ ├── .ssh/
│ ├── alice/
│ ├── carol/
│ ├── dan/
│ └── loops/
├── carol/
│ ├── .ssh/
│ ├── alice/
│ ├── bob/
│ ├── dan/
│ └── loops/
└── dan/
├── .ssh/
├── alice/
├── bob/
├── carol/
└── loops/Conclusions
Ansible's loops are pretty weird. Not only are they declarative (like everything else in Ansible), but there are many different types, some of whose names (with_nested? with_subitems?) are difficult to disentangle.
On the other hand, they are powerful enough to get things done—though it may take a slight shift in thinking (much like language functions such as array_filter, array_reduce, and array_map and other similar functions do when you first come across them). It took some time before I really started to understand that it was necessary to attach a loop to a task—even though this sometimes means looping over the same data more than once—instead of performing one or more tasks inside a loop.
Hopefully, this post will help you to benefit from my initial bafflement. To that end, I've packaged up the Vagrant virtual machine (Vagrant natively supports the use of Ansible for provisioning) and Ansible playbook that I used to create and test these examples). Just follow the directions in the README to run the examples in this post or to try your own. If you have any questions or comments, drop us a line on Twitter at @chromaticHQ!
