Agile system management in large-scale environments with Func
Get Agile!
Updating the Apache package or retrieving information from the system quickly is more easily said than done, especially if you need to perform the same task on a whole group of servers. Using a for loop to open an SSH connection to each of the servers and running the update command is a quick workaround:
# for i in `seq 1 10`; do ssh www$i update httpd; done
If the administrative SSH key is stored on the system, the call will work without manual authentication, but a more elegant solution would be preferable. Func (Fedora Unified Network Controller) [1] offers a dedicated, authenticated, encrypted communication channel instead (Figure 1). The channel is set up between a centralized management system (Overlord) and all the client systems (Minions). The certmaster tool acts as a centralized PKI instance and handles certificate management.
Func is based on a large number of modules available on any client system. Each of these modules provides various methods that can be called via the centralized management system or from other client systems. The modules contain methods for executing individual programs, querying system information, and even rebooting the systems.
Thanks to the open API, Func is easily extensible for your own customized modules: The "hack potential" is thus virtually unlimited.
Installation and Setup
Func is included in the Fedora and EPEL software repositories. Users with other distributions can pick up the source code from the Func download page [2]. The management system is easily installed with Yum:
yum install func
After the installation, you can type service certmaster start to launch the PKI. The /etc/certmaster/certmaster.conf configuration file lets you decide whether to automate client system certificate requests (autosign=yes) or to handle this task manually:
certmaster-ca --sign www1.virt.tuxgeek.de
The certmaster-ca --list command gives you a list of open certificate requests, whereas certmaster-ca --list-signed lists the certificates issued thus far. The CA certificate used to sign the requests is typically stored as a PEM file in /etc/pki/certmaster/ca/.
You also need to install a func package on the client systems. The /etc/certmaster/minion.conf configuration file provides the system with information, such as the management server name, which is important because a certificate request is automatically sent to the daemon when you start funcd service.
Depending on the Overlord configuration, this will be returned automatically, or after manual signing, and stored in the /etc/pki/certmaster/ directory. From this point on, all communications between the Overlord and its Minions will be encrypted.
In other words, no sensitive data crosses the wire in the clear. Func also gives you an easy approach to testing the client configuration (Listing 1).
Listing 1: Testing the Minion Configuration
01 # func "*" check --minion 02 SCAN RESULTS: 03 * FQDN is detected as www1.virt.tuxgeek.de, verify that is correct 04 * this minion is configured in /etc/certmaster/minion.conf 05 to talk to host 'tiffy' on port 51235 for certs, verify that is correct 06 End of Report.
To allow calls to Func to run without an administrative user account, you should set POSIX ACLs for certain directories (Listing 2).
Listing 2: Access for Non-Privileged Users
01 setfacl -d -R -m 'u:foobar:rX' /etc/pki/certmaster/ 02 setfacl -R -m 'u:foobar:rX' /etc/pki/certmaster/ 03 setfacl -d -R -m 'u:foobar:rX' /var/lib/certmaster 04 setfacl -R -m 'u:foobar:rX' /var/lib/certmaster 05 setfacl -d -R -m 'u:foobar:rX' /var/lib/certmaster/certmaster 06 setfacl -R -m 'u:foobar:rX' /var/lib/certmaster/certmaster 07 setfacl -d -R -m 'u:foobar:rX' /var/lib/certmaster/certmaster/certs 08 setfacl -R -m 'u:foobar:rX' /var/lib/certmaster/certmaster/certs 09 setfacl -d -R -m 'u:foobar:rX' /var/lib/certmaster/peers 10 setfacl -R -m 'u:foobar:rX' /var/lib/certmaster/peers 11 setfacl -d -R -m 'u:foobar:rwX' /var/lib/func 12 setfacl -R -m 'u:foobar:rwX' /var/lib/func 13 setfacl -d -R -m 'u:foobar:rwX' /var/log/func/ 14 setfacl -R -m 'u:foobar:rwX' /var/log/func/
Automating Installation with Cobbler
If you want to automate the process of installing Func on all your systems, you can use Cobbler [3] kickstart profiles.
The Cobbler configuration file has two separate settings for future Func clients:
# grep '^func' /etc/cobbler/settings func_auto_setup: 1 func_master: grobi.tuxgeek.de
Now you can add the Cobbler $func_install_if_enabled template to the %packages section of the kickstart profile, which is used for installing new machines. This step only installs the Func RPM if the func_auto_setup entry in the Cobbler configuration is set to 1. Adding a SNIPPET:func_register_if_enabled line to the %post section ensures that the client system registers with the Func master.
If the master is configured to automate the process of issuing client certificates, the system will immediately become available as a Func Minion, thus removing the need for manual configuration.
If you are using Cobbler, you'll need Fedora Everything available in your distribution for the installation of Func on Fedora to work. If you have Red Hat Enterprise Linux or CentOS, you'll need to set up the EPEL repository:
# cobbler repo add --name=el-5-x86_64-epel --mirror=http://download.fedora.redhat.com/pub/epel/5/x86_64
You can then introduce the repository to the required kickstart profiles to allow the installation of packages from this repository:
# cobbler profile edit --name=el5-ksprofile --repos="el-5-x86_64-epel"
Note that the $func_install_if_enabled template in the %packages section is needed in this case.
Modules, Please!
Once you're sure the management system and clients can talk via a secure channel, you should discover which modules a certain client has:
# func www1.virt.tuxgeek.de call system list_modules
The list should contain at least the standard modules on all your client systems, including yumcmd and command. Listing 3 shows a list of methods for a single module.
Listing 3: Multiple Methods per Module
01 # func www1.virt.tuxgeek.de call yumcmd list_methods
02 {'www1.virt.tuxgeek.de':
03 ['check_update',
04 'save_config',
05 'update',
06 'module_description',
07 'module_version',
08 'get_method_args',
09 'module_api_version',
10 'list_methods']}
Now, to update a certain package on this machine, do the following:
# func www1.virt.tuxgeek.de call yumcmd update httpd
Access to methods in other modules works in a similar way.
Many Hosts
Func really shines when you need to manage a large number of systems. You can leverage the normal shell file name expansions ("globbing") here, but note that all file name patterns must be enclosed in quotes. The following example queries all the LDAP systems in the virt.tuxgeek.de domain for their kernel versions, except the systems that contain the slave string in their hostnames:
# func --exclude "*slave*" "*ldap*.virt.tuxgeek.de" show hardware kernelVersion ldap1-ms.virt.tuxgeek.de: 2.6.18-128.el5 ldap2-ms.virt.tuxgeek.de: 2.6.18-128.el5
Func also lets you group machines for management. You can either edit the /etc/func/groups manually to create the required group, or leave the job to Func. The example shown in Listing 4 creates an LDAP server group with the hosts ldap1 and ldap2, displays the results, then copies the /etc/hosts file to all systems in the group.
Listing 4: LDAP Management
01 # func "*" group --ah "@ldap:ldap[1-2].virt.tuxgeek.de" 02 # func "*" group --la 03 Group : ldap 04 Host : ldap1.virt.tuxgeek.de 05 Host : ldap2.virt.tuxgeek.de 06 # func "@ldap" copyfile --file=/etc/hosts --remotepath=/etc/hosts
Things really get interesting when you access the API provided by the management tool. Basically, you can address the API function for any method you can call at the command line. Bindings exist for the Python scripting language, for example. Listing 5 is a simple example that checks whether the LDAP service is active on all members of the LDAP group and, if not, launches the service:
Listing 5: Func API
01 #!/usr/bin/python
02
03 import func.overlord.client as fc
04
05 results = fc.Client("@ldap").service.status("dirsrv")
06
07 for (host, returns) in results.iteritems():
08 if returns == 0:
09 print host, "LDAP-Server is UP"
10
11 else:
12 print host, "LDAP-Server is DOWN - starting it up"
13 result = fc.Client(host).service.start("dirsrv")
14 if result[host] == 0:
15 print host, "LDAP-Server is now UP"
16 else:
17 print host, "LDAP-Server is still down, check manually..."
# python check-ldap.py ldap1.virt.tuxgeek.de LDAP-Server is UP ldap2.virt.tuxgeek.de LDAP-Server is DOWN - starting it up ldap2.virt.tuxgeek.de LDAP-Server is now UP
If you enjoyed using the editor and would like to develop your own modules, that's no problem. A comprehensive guide for doing so in Python is available [4].
Conclusions
Func can be an enormous help, especially in large-scale system landscapes. Tasks that were formerly the domain of SSH loops are now easily completed at the command line. And, with the comprehensive API, the ability to program your own modules and add functionality makes Func's hack potential more or less infinite.