Action on the Hook

Virtualization speeds up the pace of a system administrator's life. With KVM, Xen, and VMware, admins can provision systems far more quickly than they could have procured and installed new hardware back in the days of dedicated servers. You can use the extra time for cloning image configurations, copying directories, and mounting storage resources. However, things get tricky when hardware issues mean you need to shut down, migrate, and restart virtual machines.

If you want to reduce the tedium of working with virtual machines, you could resort to an Infrastructure-as-a-Service (IaaS) cloud solution. The cloud handles the most important tasks of server virtualization, manages the basic infrastructure (e.g., DNS and DHCP), and provides a web interface. Admins can choose from a gallery of open source products, such as OpenStack [1], openQRM [2], Eucalyptus [3], or Ganeti [4], each with its specific functionality and concepts.

Because of the diversity of scenarios, most cloud stacks behave in a complex way and require a lengthy learning curve. OpenNebula [5], which originated in Spain, sets out to prove that things can happen more quickly, if your users have typical requirements.

Lean Cloud Management

OpenNebula spent its childhood in the academic field of grid computing and is now considered a lightweight enterprise solution. The Fermilab particle accelerator uses OpenNebula to manage hundreds of virtual machines, and industrial customers include Telefonica and Akamai. The main developer from C12G Labs offers commercial support and is currently establishing an international network of partners [6].

OpenNebula is characterized by its simple elegance. Whereas other IaaS competitors prompt you to install and configure half a dozen services and components, OpenNebula manages with only a few components and with a single front end for controlling and configuring the private cloud, OpenNebula Sunstone. The "OpenNebula Installation" box outlines the setup, and Figure 1 shows the architecture.

Versions and Features

OpenNebula names its releases after interstellar nebulae. Stable version 4.0, published in May 2013, was named "Eagle" – after the Eagle Nebula (a star cluster discovered in the mid-18th century that is about 7,000 light years from Earth). Compared with the 3.x series, which is roughly two years older, it mainly fixes bugs but also revamps the Sunstone web interface. The changes affected, for example, VNC access to virtual machines.

The latest version 4.2, code-named "Flame," was released in early August 2013. (The eponymous Flame Nebula lies in the Orion constellation; its light takes about 1,000 years to reach earth.) The creators of the project have smoothed off the rough edges in the Sunstone interface for Flame. One new feature is a simplified view that explicitly addresses users who only want to provision one VM – not set up an entire cluster. Admins can now monitor image sizes in an improved way and intervene if a VM is threatening to spiral out of control. To do this, OpenNebula now monitors this resource.

The developers have written the back end for VMware virtualization, which can now be installed easily and has fewer dependencies. The Xen driver now uses the xl-style commands available with version 4.0 of the hypervisor.

OpenNebula users have become accustomed to running the latest version in test environments, while relying on the penultimate version for production operation. Although the programming interface described below existed in version 3, its entire functionality was only reached in version 4.0.

On the Hook

An important reason for virtualizing servers is the pursuit of automation. If your own development department has built a new release of its PHP, Perl, Ruby, or Java application using the Jenkins CI server [8], for example, it can automatically generate a stage VM with the necessary packages and frameworks. Once the requesting department has tested and approved the application, you just need one click to release the associated resources in an ideal world.

The glossy brochures on cloud computing fail to mention that the devil again lies in the details. In many cases, you'll need to update a database schema or start a particular service on a virtual machine. Fortunately, OpenNebula provides hooks for such tasks [9]. The software uses them on many occasions to call external programs that admins can compose in the programming or scripting language of their own choice.

Following Up on Events

The basic principle is, if an OpenNebula user wants to know about changes on a virtual machine, a host, the configured networks, users, groups, or images, they can add their own scripts to the execution sequence. You store your tools in the /var/lib/one/remotes/hooks directory (for Debian, /var/lib/opennebula/remotes/hooks).

To define the events for which you need a response, the cloud administrator identifies the triggering object and event in Table 1 or Table 2 and enters it in /etc/one/oned.conf. For example, Listing 1 registers the shell script from Listing 2 to measure VM uptimes.

01 VM_HOOK = [
02     name      = "vm_switchon",
03     on        = "RUNNING",
04     command   = "account.sh",
05     arguments = "on $ID" ]
06
07 VM_HOOK = [
08     name      = "vm_switchoff",
09     on        = "STOP",
10     command   = "account.sh",
11     arguments = "off $ID" ]
12 [...]

01 #!/bin/bash
02
03 ACCOUNTING=/var/log/one/accounting.log
04
05 case "$1" in
06     "on"|"off")
07         mode=$1
08         shift
09         ;;
10      *)
11         echo "$0 error: wrong mode." >&2
12         exit 1
13         ;;
14 esac
15
16 if [ "$1" = "" ]
17 then
18     echo "$0 error: ID missing." >&2
19 fi
20
21 id=$1
22 shift
23
24 echo "$id $(date +%s) $mode" >> $ACCOUNTING

The VM_HOOK construct can appear more than once in the configuration. The on parameter defines the event to which OpenNebula responds (Listing 1, lines 3 and 9). The command entries specify the script to be called; arguments defines a string containing its call parameters. As connoisseurs will immediately see from the syntax, the IaaS management system itself is written in Ruby, but in terms of scripting languages, admins have a choice.

The shell script in Listing 2 checks (lines 5 to 9) for a running VM and then (in line 24) records the Unix epoch timestamp, along with the VM ID, in a configurable logfile. With an Awk script that reads logfiles and parses values, you could easily discover the total uptime of each VM and pass the results on to billing software, a department, or a customer of the IaaS cloud, for example.

The $TEMPLATE Variable

The example passes parameters only in a very simple way. To discover more information about the affected resources, users can pass the content of the $TEMPLATE variable to the script. It uses a kind of XML dialect to store a wide variety of details about the VM, the network, or the storage system. To avoid having to line up the big guns in the form of an external XML parser, the developers provide a tool that divides the content into handy key-value pairs and filters the results. For example, the following bash snippet:

X=/var/lib/one/remotes/datastore/xpath.rb
mem=$($X -b $template HOST_SHARE/MAX_MEM)

returns the contents of the <HOST_SHARE><MAX_MEM> container below the root node. This is the field to which OpenNebula sends the value for total available memory on the virtualization host; Listing 3 shows an example with a number of values.

01 <HOST>
02   <ID>1</ID>
03   <NAME>one-sandbox</NAME>
04   <STATE>2</STATE>
05   <IM_MAD>kvm</IM_MAD>
06   <VM_MAD>qemu</VM_MAD>
07   <VN_MAD>dummy</VN_MAD>
08   <LAST_MON_TIME>1377716985</LAST_MON_TIME>
09   <CLUSTER_ID>-1</CLUSTER_ID>
10   <CLUSTER/>
11   <HOST_SHARE>
12     <DISK_USAGE>0</DISK_USAGE>
13     <MEM_USAGE>65536</MEM_USAGE>
14     <CPU_USAGE>10</CPU_USAGE>
15     <MAX_DISK>0</MAX_DISK>
16     <MAX_MEM>502568</MAX_MEM>
17     <MAX_CPU>100</MAX_CPU>
18     <FREE_DISK>0</FREE_DISK>
19     <FREE_MEM>259512</FREE_MEM>
20     <FREE_CPU>87</FREE_CPU>
21     <USED_DISK>0</USED_DISK>
22     <USED_MEM>243056</USED_MEM>
23     <USED_CPU>12</USED_CPU>
24     <RUNNING_VMS>1</RUNNING_VMS>
25   </HOST_SHARE>
26   <VMS>
27     <ID>16</ID>
28   </VMS>
29   <TEMPLATE>
30     <ARCH><![CDATA[x86_64]]></ARCH>
31     [...]
32   </TEMPLATE>
33 </HOST>

Monitoring as the Main Application

This simple interface provides a variety of options: If you are prepared to take a detour, in the form of polling logfiles or status files, you can tie in an existing availability monitoring system like Nagios or Zabbix. Alternatively, OpenNebula itself includes a simple performance monitoring system in the form of Ganglia [10] that lets the system administrator keep an eye on important system parameters, such as CPU load or free memory.

If you hook into the CREATE event of a VM, you can install the monitoring agent software via configuration management, or you can roll out a complete application using a continuous integration trigger. Puppet, Chef, and Jenkins offer interfaces for this, and administrators can control them using wget or curl.

Insights and Future

You can set up your own private OpenNebula cloud with comparatively little effort. In the simplest case, you only need two systems, a management console with web access and a virtualization host. Because of its easy-to-handle and open interfaces, OpenNebula expands in a modular way. The way to do this – as described in this article – is to use hooks that latch onto critical events.

Additionally, the IaaS management system provides a number of other APIs for those interested: You can use drivers to integrate new hypervisors in a similarly easy way. EC2 or OCCI (Open Cloud Computing Interface) connectors [11] – a generalized set of API calls that try to unify the different APIs provided by specific IaaS stacks  – let cloud captains launch and shut down their own virtual applications. Although you do need to create storage resources manually and are left to your own devices in terms of VLAN management, with OpenNebula, you do get a transparent and understandable private cloud for your own solar system.