Remote Diagnostics

In enterprise environments, you often see extensive management frameworks used to manage the collection of virtual systems and other resources. If you want to right-size your approach and prefer to rely on the Libvirt virtualization framework instead, you can automate many tasks via its API.

I recently had such a case: A provisioning front end had been used to generate a variety of virtual systems on a number of hypervisors. I wanted to know when a machine was available on a host. Of course, I could have changed the front end so that a message was sent when a new system was installed, but, unfortunately, I did not have access to the provisioning system.

I could have turned to a management tool (virsh or virt-manager), but that's a fairly convoluted approach in the long run, especially if you are sitting in front of a workstation and the tools are not installed there. To solve the problem, I decided to simply send a request to the Libvirt API on the existing hypervisors and retrieve a list of active and inactive systems. The output can be garnished with arbitrary information, but I was just interested in seeing which systems were online and which were offline.

The virtualization framework offers a range of authentication methods. Because time is always short, I opted for a simple SSH-based login on the host systems. To do this, I created a virtuser account and defined an SSH key for the account at the same time. The user and the SSH key are managed via a central FreeIPA system (Figure 1).

For access control, you may want to allow the new user to access the Libvirt framework, because only the root user has access without appropriate changes. On recent systems, you can use polkit to do this. As of version 0.112, you even have a separate engine, which you can feed with JavaScript-based rules. You can really do some interesting things – for example, giving a user access only to one particular system, or only if it is running on a particular hypervisor. The actions that a user can perform can also be defined in a very granular way.

But, to return to the topic, I decided to grant access to a whole group. Thus, I can use my personal account to access the systems and do not need to be logged in as root. To allow access for all of the group members in virt, I created a simple rule file on all the host systems (Listing 1). systemctl restart polkit.service then parses the ruleset.

01 polkit.addRule(function(action, subject) {
02  polkit.log("action=" + action);
03  polkit.log("subject=" + subject);
04  var now = new Date();
05  polkit.log("now=" + now)
06  if ((action.id == "org.libvirt.unix.manage" || action.id == "org.libvirt.unix.monitor") && subject.isInGroup("virt")) {
07     return polkit.Result.YES;
08  }
09  return null;
10 });

After ensuring that the user virtuser is a member of the virt group, an initial test shows whether or not access works:

# virsh -c qemu+ssh://virtuser@ernie/system version
Compiled against library: libvirt 1.1.3
Using library: libvirt 1.1.3
Using API: QEMU 1.1.3
Running hypervisor: QEMU 1.6.1

If access works, a few lines of Python are all you need to retrieve the desired information (Listing 2). The hypervisor hosts are defined in the hv array. In the following loop, the connection URI is then assembled in each case; armed with this information, the conn object is then used to open a connection to the existing hosts. The two functions listDomainsID and listDefinedDomains finally let me extract the information and output the names of the virtual systems.

01 #!/usr/bin/env python
02
03 import libvirt
04
05 ## Hypervisor hosts
06 hv = [ "tiffy.tuxgeek.de", "ernie.tuxgeek.de" ]
07
08 for hv_host in hv:
09
10   uri = "qemu+ssh://virtuser@" + hv_host + "/system"
11   conn = libvirt.openReadOnly(uri)
12
13   hypervisor_name = conn.getHostname()
14
15   print "The follwing machines are running on: " + hypervisor_name
16
17   # List active hosts
18   active_hosts = conn.listDomainsID()
19   for id in active_hosts:
20     dom = conn.lookupByID(id)
21     print "System " + dom.name() + " is UP."
22
23   # List inactive Hosts
24   for name in conn.listDefinedDomains():
25     dom = conn.lookupByName(name)
26     print "System " + dom.name() + " is DOWN."
27   print

Now, when I run the script manually, I see the results from Listing 3.

01 $ python list-virt-systems.py
02 The following machines are running on: tiffy.tuxgeek.de
03 System fedora is UP.
04 System rhel65 is down.
05 System ipa1 is down.
06 System ipa2 is down.
07
08 The following machines are running on: ernie.tuxgeek.de
09 System rawhide is UP.
10 System build is down.
11 System rhds1 is down.
12 System rhds2 is down.
13 System devel is UP.

Wonderful! The script works as expected. To make life simple, I now let crond call the script and push the output into my web server's document root. Then, when I call the appropriate URL, I can immediately see which machines are available and which systems have been created on the hypervisor but are not yet online.

This small example is deliberately kept simple. If you want to use the script for serious applications, you should at least add exception handling. If desired, you can, of course, tap the API's feature set not only to query status information but also to create new systems or define other virtual resources, such as networks.

On Fedora systems, the Python bindings are provided by the libvirt-python package. The documentation folder has other examples that can serve as a basis for further experiments. It is also exciting to show the VMware camp how to manage their systems with a small script. In this case, however, you need to remember to equip Libvirt with an appropriate ESX driver.