Automation Tool

If you have to run standard tasks in an environment with a large number of systems (e.g., a compute cluster, a server farm, or a cloud environment) you might want a tool to help you save time and avoid duplication of labor. Logging in on each server and typing your commands hundreds of times manually is too slow, too error prone, and too inefficient. Many admins would rather have a tool that lets them run standard tasks on all clients in parallel, without typos and in a reproducible manner.

Tools such as Puppet, Chef, SaltStack, and Ansible provide this functionality through an agent running on the client, and a special description language lets the user define tasks or the target state. Rex takes a different approach. The Rex configuration management tool uses SSH as the transport medium and Perl as the command language, which means any computer can act as a Rex client without the need for additional software.

The fact that Rex doesn't rely on a client agent program also means the user won't run into conflicts between newer and older Rex versions. And you won't have to learn a new, specialized command language: As long as you know some Perl, you'll be ready to get started.

Installing Rex

Only the command center – the Rex host, which is sometimes called Rex Control Master – needs a few software modules. By the way, the Rex developers call the program (R)?ex, which is totally unpronounceable, so I'll settle for plain old Rex for the rest of the article. You can install these software modules either via a package manager in Linux or FreeBSD or via Git. If you're using a package manager, you'll want to include the Rex [1] repository to leverage automated updates by the distribution. Listing 1 shows how to install via Git. The advantage of using Git is that the Git sources have the freshest version. If you are accustomed to working with Perl, a third way to install Rex is through Perl's CPAN archive [2].

01 git clone https://github.com/krimdomu/Rex.git
02 cd Rex
03 perl Makefile.PL
04 make
05 make test
06 make install

First Steps

Commands are transmitted to the remote Rex client through a so-called Rex file. Listing 2 shows a fairly simple example. The first three lines define the name of the user who executes the commands, the password, and the authentication method.

01 user "root";
02 password "secret";
03 pass_auth;
04
05 group server => "hercules", "sugar";
06
07 desc "Get the uptime of all servers";
08
09 task "uptime", group => "server", sub {
10  my $output = run "uptime";
11  say $output;
12 }

You might notice that Listing 2 holds the root password in unencrypted form. Rex also allows authentication using SSH keys. For encrypted authentication, you need to create an RSA or DSA key pair without a password on the Rex host and then copy the public key over to the client (Listing 3). After that, a test login on the client without password should work.

01 root@hercules:~/.ssh# ssh-keygen -t rsa
02 Generating public/private rsa key pair.
03 Enter file in which to save the key (/root/.ssh/id_rsa):
04 Enter passphrase (empty for no passphrase):
05 Enter same passphrase again:
06 Your identification has been saved in /root/.ssh/id_rsa.
07 Your public key has been saved in /root/.ssh/id_rsa.pub.
08 The key fingerprint is:
09 9b:e4:e2:27:92:04:4a:9b:ee:82:cc:9f:4d:4b:4d:c1 root@hercules
10 The key's randomart image is:
11 +--[ RSA 2048]----+
12 |                 |
13 |    .            |
14 |    E            |
15 |    ..   .       |
16 |    ..o. .S      |
17 |    .o . oo o    |
18 |    = . +..+     |
19 |    o+ B.o..     |
20 |    o..o +.o     |
21 +-----------------+
22 root@hercules:~/.ssh# ssh-copy-id root@sugar
23 root@sugar's password:

Check ~/.ssh/authorized_keys to make sure you haven't added extra keys that you weren't expecting.

Then, change the first lines of the Rex file into the following:

user "root";
private_key "/root/.ssh/id_rsa";
public_key "/root/.ssh/id_rsa.pub";

The CPAN Net::OpenSSH module also supports the possibility of Kerberos as an authentication method.

Installing Packages

Listing 2 shows how you can use the Rex file to query the client for information such as the uptime. You can use the same technique to obtain other client values, such as the amount of free memory (free or vmstat), the fill factor of the hard disks (df), the network utilization (netstat), and the I/O performance (iostat). You can also easily filter and format the output with Perl.

If you want to archive the results, a tool like the System Activity Reporter sar or the Performance Co-Pilot PCP [3] would be more appropriate, because they are designed for long-term data handling. Sar comes with most Linux distributions (package sysstat), but it is not installed by default. So, you have to install it first, and Rex can even help with installing other tools.

Rex comes with several commands written in Perl and one of these commands is install, which you can use to install software package. Append the lines from Listing 4 to the Rex file. After that, type:

rex install_sysstat

01 use Rex::Commands;
02 use Rex::Commands::Pkg;
03
04 desc "Install sar (sysstat)";
05 task "install_sysstat", group => "server", sub {
06  install package => "sysstat";
07 };

The install command automatically takes care of the translation into real commands for each platform on which it runs. Thus, install works well with a heterogeneous group of servers, as long as each group member belongs to the supported systems (CentOS 5/6, Debian 5/6, Fedora, Gentoo, Mageia, openSUSE, RHEL 5/6, Scientific Linux, Ubuntu version 10 or greater, Solaris 10/11, FreeBSD, NetBSD, OpenBSD). Rex knows how to install packages on these platforms and will use the appropriate command (rpm, apt, pkg, emerge, urpmi,opkg, yum, pkgadd or zypper). However, all systems in a group have to use the same package name.

How to Edit config Files

Sar is now installed, but it can't gather any data yet. First of all, you have to change the entry ENABLED= from false to true in /etc/default/sysstat file. You can use Rex either to overwrite the whole sysstat file or to update the file with the necessary change. Use a script like the snippet in Listing 5 to change only the relevant word.

01 task "enable_sar", group => "server", sub {
02  run qq(sed -i 's/ENABLED="false"/ENABLED="true"/' /etc/default/sysstat);
03  service "sysstat" => "restart";
04 };

Rex actually has a special command for replacing text in a file:

task searchreplace => sub {
 sed qr{search}, "replace", "/directory/file.txt";
};

Related commands allow you to delete lines that match a search pattern, to overwrite whole files, or to append lines. After changing a configuration file, you'll need to restart the service you are updating. As Listing 5 shows, Rex will again work with an abstraction. The restart command is translated into a command with the same name under Ubuntu or to an equivalent svcadm call under Solaris.

User Management and Other Tasks

Rex offers numerous commands for typical tasks that occur regularly on the admin's agenda. For example, you can use Rex to create and delete groups, start and stop processes, manage cron jobs, manipulate Iptables rules, load kernel modules, download files with scp, or edit system parameters with sysctl.

Rex also provides detailed information about the client systems. This information goes from a simple query of the operating system (get_operating_system()) to a long list with all kinds of details (as shown in Listing 6).

01 root@hercules:/home/jcb/Rex/test# rex sysinfo
02 [2014-04-25 10:49:34] INFO - Running task sysinfo on sugar
03 [2014-04-25 10:49:34] INFO - Connecting to sugar:22 (root)
04 [2014-04-25 10:49:34] INFO - Connected to sugar, trying to authenticate.
05 [2014-04-25 10:49:35] INFO - Successfully authenticated on sugar.
06 $memory_cached = '89'
07 $Kernel = {
08  kernelversion => '#25~precise1-Ubuntu SMP Thu Jan 30 17:42:40 UTC 2014'
09  architecture => 'i686'
10  kernel => 'Linux'
11  kernelrelease => '3.11.0-15-generic'
12  }
13 $hostname = 'sugar'
14 $operating_system = 'Ubuntu'
15 $operatingsystem = 'Ubuntu'
16 $operating_system_release = '12.04'
17 $eth0_mac = '08:00:27:c4:a1:d8'
18 $VirtInfo = {
19  virtualization_role => 'guest'
20  virtualization_type => 'virtualbox'
21  }
22 $memory_shared = '0'
23 $Network = {
24  networkdevices => [
25   'lo'
26   'eth0'
27  ]
28  networkconfiguration => {
29   lo => {
30   broadcast => ''
31   ip => '127.0.0.1'
32   netmask => '255.0.0.0'
33   mac => ''
34   }
35   eth0 => {
36   broadcast => '192.168.111.255'
37   ip => '192.168.111.188'
38   netmask => '255.255.255.0'
39   mac => '08:00:27:c4:a1:d8'
40   }
41  }
42  }
43 $Swap = {
44  free => '509'
45  used => '0'
46  total => '509'
47  }
48 $eth0_ip = '192.168.111.188'
49 $swap_used = '0'
50 $Host = {
51  kernelname => 'Linux'
52  operating_system => 'Ubuntu'
53  hostname => 'sugar'
54  operatingsystemrelease => '12.04'
55  operatingsystem => 'Ubuntu'
56  domain => ''
57  operating_system_release => '12.04'
58  manufacturer => 'innotek GmbH'
59  }
60 $kernelversion = '#25~precise1-Ubuntu SMP Thu Jan 30 17:42:40 UTC 2014'
61 $memory_total = '494'
62 $kernelrelease = '3.11.0-15-generic'
63 $operatingsystemrelease = '12.04'
64 $architecture = 'i686'
65 $domain = ''
66 $swap_free = '509'
67 $lo_broadcast = ''
68 $kernel = 'Linux'
69 $memory_used = '178'
70 $kernelname = 'Linux'
71 $swap_total = '509'
72 $memory_buffers = '12'
73 $lo_netmask = '255.0.0.0'
74 $lo_ip = '127.0.0.1'
75 $lo_mac = ''
76 $memory_free = '316'
77 $manufacturer = 'innotek GmbH'
78 $Memory = {
79  shared => '0'
80  buffers => '12'
81  free => '316'
82  used => '178'
83  total => '494'
84  cached => '89'
85  }
86 $ETH0_broadcast = '192.168.111.255'
87 $ETH0_netmask = '255.255.255.0'

Rex can also report details about hypervisors, and it can create, configure, start, stop, and destroy virtual machines with VirtualBox, KVM, and Xen. You can even use Rex to monitor a system running in Amazon's Elastic Compute Cloud (EC2).

Summary

The most impressive thing about Rex is its ease of use. An experienced admin only needs a short amount of time to learn the ropes. The requirements for the central Rex host and its clients are minimal, which means it won't take you long to start benefiting from the automation of the Rex environment. Rex helps you avoid typos and gives you access to a useful collection of well-documented system-management commands, but perhaps the biggest benefit is the time you'll save with parallel execution of common tasks on all clients.

One weakness of Rex is the lack of documentation. An online book about Rex isn't much more than an index of contents and a FAQ consists with only of a handful questions. At least the documentation of the API functions seems complete. The error message documentation might help a Perl developer who can study the Perl sources, but it is not very helpful for the ordinary user.