Strong Quad

Programmers often try to set up a development environment for their projects that matches the production environment. However, upon starting up the finished software, they might discover some required libraries are missing or a slightly modified configuration leads to unexpected side effects. Vagrant [1] prevents this kind of situation. The tool creates a virtual machine with a suitable development environment at the push of a button. The machines produced in this way are not only portable, but Vagrant can always reproduce them precisely. Finished machines can also be passed on to other team members and relocated to the production environment, if needed.

On the Rails

Mitchell Hashimoto began to develop Vagrant in his spare time, but because of the fast-growing numbers of users, he founded the one-man company HashiCorp in the fall of 2012, so he could work full-time on Vagrant. The company primarily earned money through support contracts, training, and the development of commercial add-ons. According to HashiCorp, current users of Vagrant are the BBC, O'Reilly, and Expedia, in addition to Mozilla and Disqus.

To set up a development environment with Vagrant, you first create a small configuration file in your project directory. Within this file, called Vagrantfile (case not important), you need to describe the required software, configure the virtual machine, and determine how you would like to access the guest system. A PHP programmer might want a Debian system with a web server, for example.

Vagrant even works as a wrapper for the virtualization software or provider. Vagrant natively supports VirtualBox, Docker, and Microsoft's Hyper-V. You can add more providers in the form of plugins. If you want to run Vagrant with VMware, you will find a suitable plugin on the HashiCorp site [2]. As a commercial product, this costs around US$ 80 per license.

Cloudy Boxes

To start the virtual machine as quickly as possible, Vagrant just clones a complete disk image and then modifies it. Vagrant calls these base images "boxes." Prebuilt boxes can be downloaded free of charge via the community collection on Vagrantbox.es [3] (Figure 1) or from the Vagrant Cloud powered by HashiCorp [4]. The latter provides a virtual machine with Debian 7.4 and Chef pre-installed.

If you register for free in the Vagrant Cloud, you can also take advantage of the Vagrant Share service. It offers a kind of dynamic DNS service for boxes: At the command line, the Vagrant user can combine a running virtual machine with a new subdomain on Vagrantshare.com. Its development environment is subsequently accessible on the Internet (e.g., on http://hulking-chameleon-3934.vagrantshare.com). In this way, you can test logging into the new web application with OAuth under real conditions and give team members in branch offices an insight into ongoing development.

If you want your boxes in the Vagrant Cloud to be accessible to certain people only, you need a paid subscription. Prices start at US$ 6 per month [5]; on top of that, each box download costs 12 cents (Figure 2). If you don't trust Vagrant Cloud, you can create your own boxes.

Ruby Control

The configuration lines in Vagrantfile are Ruby code, but users do not need to know all about the scripting language. Listing 1 shows an example of a minimal configuration file: The second line uses

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/trusty64"
  config.vm.network :forwarded_port, host: 9999, guest: 80
end

config.vm.box = "ubuntu/trusty64"

to stipulate that a box in the Vagrant Cloud should run an Ubuntu 14.04 server in the 64-bit variant. The third line also sets up port forwarding, making the web server on the guest accessible on port 9999 of the host computer.

Once you have adapted the Vagrantfile to suit your needs, a vagrant up command is all it takes to have a working virtual machine. Typing vagrant ssh opens an SSH connection to the guest system.

If you have ever set up a virtual machine manually, including SSH access, you will probably be jumping for joy right now. As an added bonus, you can simply pass on the configuration file to other team members.

Assembly Service

When you first start the virtual machine, Vagrant can run arbitrary commands and programs automatically (e.g., to install and set up a web server retroactively using a shell script), and it makes provisioning a quick and easy process. Vagrant not only runs shell scripts, it also works with the configuration systems Chef, Puppet, Ansible, Salt, and CFEngine. On request, the tool even creates Docker containers. Plugins let you retrofit more of these provisioners.

By default, Vagrant passes the developer's directory through to the guest machine, where it appears in the /vagrant directory. Similarly, the tool can synchronize any directory on the hard disk with any on the virtual machine (synced folders). In this way, the developer sees the effects of changes directly in the filesystem.

Vagrant is not just available for Linux, but also for Windows and Mac OS X. Prebuilt binary packages in the Debian and RPM format are available from the project homepage. Additionally, the tool is available from the repositories of various distributions. Ubuntu only has the slightly older version 1.4.3. Vagrant itself is written in Ruby; installing as a Rubygem is no longer intended in the current versions of Vagrant. The tool is available under the liberal MIT license, and the open source code is on GitHub [6].

Packer

Normally, it is a good idea to provide the complete application as a virtual machine (appliance). If customers use different virtualization solutions, though, this can quickly become a huge amount of work. Here is where Packer steps in [7]. The tool not only sets up a virtual machine automatically, it also builds images for cloud services such as Amazon EC2, DigitalOcean, Google Compute Engine, and OpenStack, as well as the virtualization solutions VirtualBox, VMware, Qemu (for KVM and Xen), Parallels, and Docker. On request, Packer also creates a box for Vagrant. Plugins let you add support for additional platforms. Packer can also draw on Chef and Puppet to set up the guest system.

In contrast to Vagrant, HashiCorp provides the Packer program as a distribution-independent ZIP archive, with versions for 32-bit, 64-bit, and ARM systems. Besides Linux, Packer runs on Windows, Max OS X, FreeBSD, and OpenBSD. You have to install the tool manually by moving the content of the ZIP archive to a suitable directory (e.g., /usr/local/bin on Linux). The source code is written in the Go programming language, lives on GitHub [8], and is subject to the Mozilla public license version 2.0.

Bindlestiff

Developers use a configuration file in JSON format to describe the desired output format and content of the virtual machine. Listing 2 shows an example of a template. The first of the two sections defines how Packer should create the image(s). A builder tool that specializes in a specific output format handles the image-building process. The third line in Listing 2 enables a builder called virtualbox-ovf. It first picks up an existing image in the OVF interchange format, the one in Listing 2 goes by the name of debian7.ovf. From this, the builder creates a copy, which it then starts with VirtualBox (Figure 3).

01 {
02 "builders": [{
03   "type": "virtualbox-ovf",
04   "source_path": "debian7.ovf",
05   "ssh_username": "root",
06   "ssh_password": "123456",
07   "ssh_wait_timeout": "30s",
08   "shutdown_command": "echo 'Done' | sudo -S shutdown -h now"
09   }],
10 "provisioners": [{
11   "type": "shell",
12   "inline": [
13     "sleep 30",
14     "sudo apt-get update",
15     "sudo apt-get install -y sqlite3"
16     ]
17   }]
18 }

The subsequent configuration of the guest system is handled by provisioners in Vagrant. Listing 2 starts the shell provisioner, which executes the commands in the Lines 13 to 15. This, in turn, installs SQLite on the guest system. You can then log in via SSH with a username of root and a password of 123456. Listing 2 has this information in lines five and six. Packer uses shutdown_command to tell the virtual machine to shut down.

Finally, builder steps in again and exports the guest system to OVF format. In addition to virtualbox-ovf Packer has a number of other builders. For example, virtualbox-iso draws on VirtualBox to install a completely new virtual machine and install a fresh system on it. The source is an ISO image that it downloads from a server; it even checks the checksum in the process. Of course, this process only works if the complete installation can be handled without user input. If you like, you can retroactively install your own builders and provisioners via the plugin system.

Consul

Usually, a developer will tell their application which machine has the database, but the program could alternatively ask Consul [9]. This service provides information about what services are available where at the current time; in other words, it is used for service discovery. Consul also offers failure detection by monitoring the current services and the system. For example, the tool alerts if a web server repeatedly returns HTTP status 200 or suddenly runs out of free main memory. Administrators can define appropriate rules to describe the necessary tests. When Consul detects a problem, it can automatically redirect requesting applications to another server that is still running. The web application would then not even notice that the database on the primary server had failed.

Finally, Consul offers a key/value store. In this mini-database, applications in particular can store their configuration data in JSON format. However, distributed applications can also exchange data via the key/value store and, thus, for example, elect a master. The data query takes place via a HTTP REST-style interface.

Agent Exchange

Consul itself comprises several components. The basis is one or more Consul servers that store information about the current services and answer requests from applications. HashiCorp recommends that you have at least three to five Consul servers to enhance data security. The Consul servers independently form a cluster and select a leader that is responsible for coordination. If you run multiple data centers, you can set up a separate cluster at each one. If a Consul server cannot answer an application request, it forwards the request to the other clusters and returns the response from them to the application.

Moreover, administrators can start Consul servers on other computers that simply redirect all requests to the right Consul server. This approach is designed to simplify load balancing, among other things. HashiCorp confusingly refers to this lower ranking Consul server as a client. Applications can query a Consul server directly, as well as detour via a Consul client.

Like Packer, Consul is written in Go and is available under the Mozilla Public License 2.0. Source code, available from GitHub [10], comes in packages for Windows, Mac OS X, and Linux. Just as with Packer, you'll find a ZIP archive that just contains a command-line utility. This agent launches either a Consul server or a Consul client, depending on the parameters you pass in.

You can elicit information from a consul server either via its HTTP interface or its built-in DNS server (Figure  4), and HashiCorp offers a simple user interface in the form of a web application (Figure 5).

Serf

If a production web server fails, operators of online stores quickly lose money. To keep this from happening, you can turn to Serf [11] for help (see also the "Overlap" box). The tool monitors the computing nodes in a cluster.

As soon as one fails, it runs an action defined by the administrator. For example, if a web server goes on strike, Serf can automatically notify the upstream load balancer, which in turn distributes the load to other web servers.

Serf itself is designed as a distributed system: You first need to install a Serf agent on each computer node. These agents monitor their respective neighbors in the cluster. Should one of the agents fail, the monitoring agent informs the surviving agents in the cluster, which in turn inform their neighbors.

The Serf documentation compares the approach to a zombie invasion: It starts with one zombie, which then infects all the other healthy people. In this way, all the agents in a cluster hear about the failure within a very short time.

The procedure also has the advantage that it scales freely and efficiently. The agents communicate via a Gossip protocol that sends UDP messages. Administrators therefore need to drill holes in their firewalls. Finally, each agent needs just 5 to 10MB of main memory.

Radio Transmitters

Serf not only determines whether nodes are down, it can also dispatch events predefined by the admin. These events then trigger certain actions on the node. For example, the nodes could search for updates and install them.

Serf is written in Go and licensed under the Mozilla Public License 2.0. Source code is available on GitHub [12], with prebuilt packages for Windows, Mac OS X, Free BSD, OpenBSD, and Linux. For Linux, you have the choice of x86 or x86_64 variants and even an ARM version. Commissioning and operation of Serf strongly resembles the process for Consul: Again, you unpack the ZIP archive to reveal a command-line utility – the agent. You need to start the agent on each node to be monitored and introduce the agents to one another: Just introduce an agent to its neighbor; it then discovers all the others automatically.

Conclusions

HashiCorp offers four extremely useful tools for the DevOps lifecycle: Vagrant helps simplify development and Packer helps with deployment, whereas Consul and Serf take care of operations and maintenance. Using these tools is extremely simple: In Vagrant, for example, a single command line is all it takes to launch a custom development environment.

The documentation provided by HashiCorp is exemplary for all four tools. It is complete, comprehensive, and above all understandably written. Useful introductions help you get started. Users in the appropriate Google groups [14] or in IRC chats on Freenode help clear up any questions you might still have. For Vagrant, the IRC channel is #vagrant. Additionally, O'Reilly has released a book on Vagrant, and the author is none other than Mitchell Hashimoto [15].