Management Func Remote Control Lead image: bowie15, 123RF
bowie15, 123RF

Agile system management in large-scale environments with Func

Get Agile!

Running a handful of commands on a large number of systems is a classic admin problem. Func gives administrators a useful alternative to loops and SSH connections. By Thorsten Scherf

Updating the Apache package or retrieving information from the system quickly is more easily said than done, especially if you need to perform the same task on a whole group of servers. Using a for loop to open an SSH connection to each of the servers and running the update command is a quick workaround:

# for i in `seq 1 10`; do ssh www$i update httpd; done

If the administrative SSH key is stored on the system, the call will work without manual authentication, but a more elegant solution would be preferable. Func (Fedora Unified Network Controller) [1] offers a dedicated, authenticated, encrypted communication channel instead (Figure 1). The channel is set up between a centralized management system (Overlord) and all the client systems (Minions). The certmaster tool acts as a centralized PKI instance and handles certificate management.

Overview of features for the Fedora Unified Network Controller (Func).
Figure 1: Overview of features for the Fedora Unified Network Controller (Func).

Func is based on a large number of modules available on any client system. Each of these modules provides various methods that can be called via the centralized management system or from other client systems. The modules contain methods for executing individual programs, querying system information, and even rebooting the systems.

Thanks to the open API, Func is easily extensible for your own customized modules: The "hack potential" is thus virtually unlimited.

Installation and Setup

Func is included in the Fedora and EPEL software repositories. Users with other distributions can pick up the source code from the Func download page [2]. The management system is easily installed with Yum:

yum install func

After the installation, you can type service certmaster start to launch the PKI. The /etc/certmaster/certmaster.conf configuration file lets you decide whether to automate client system certificate requests (autosign=yes) or to handle this task manually:

certmaster-ca --sign

The certmaster-ca --list command gives you a list of open certificate requests, whereas certmaster-ca --list-signed lists the certificates issued thus far. The CA certificate used to sign the requests is typically stored as a PEM file in /etc/pki/certmaster/ca/.

You also need to install a func package on the client systems. The /etc/certmaster/minion.conf configuration file provides the system with information, such as the management server name, which is important because a certificate request is automatically sent to the daemon when you start funcd service.

Depending on the Overlord configuration, this will be returned automatically, or after manual signing, and stored in the /etc/pki/certmaster/ directory. From this point on, all communications between the Overlord and its Minions will be encrypted.

In other words, no sensitive data crosses the wire in the clear. Func also gives you an easy approach to testing the client configuration (Listing 1).

Listing 1: Testing the Minion Configuration

01 # func "*" check --minion
03 * FQDN is detected as, verify that is correct
04 * this minion is configured in /etc/certmaster/minion.conf
05   to talk to host 'tiffy' on port 51235 for certs, verify that is correct
06  End of Report.

To allow calls to Func to run without an administrative user account, you should set POSIX ACLs for certain directories (Listing 2).

Listing 2: Access for Non-Privileged Users

01 setfacl -d -R -m 'u:foobar:rX' /etc/pki/certmaster/
02 setfacl -R -m 'u:foobar:rX' /etc/pki/certmaster/
03 setfacl -d -R -m 'u:foobar:rX' /var/lib/certmaster
04 setfacl -R -m 'u:foobar:rX' /var/lib/certmaster
05 setfacl -d -R -m 'u:foobar:rX' /var/lib/certmaster/certmaster
06 setfacl -R -m 'u:foobar:rX' /var/lib/certmaster/certmaster
07 setfacl -d -R -m 'u:foobar:rX' /var/lib/certmaster/certmaster/certs
08 setfacl -R -m 'u:foobar:rX' /var/lib/certmaster/certmaster/certs
09 setfacl -d -R -m 'u:foobar:rX' /var/lib/certmaster/peers
10 setfacl -R -m 'u:foobar:rX' /var/lib/certmaster/peers
11 setfacl -d -R -m 'u:foobar:rwX' /var/lib/func
12 setfacl -R -m 'u:foobar:rwX' /var/lib/func
13 setfacl -d -R -m 'u:foobar:rwX' /var/log/func/
14 setfacl -R -m 'u:foobar:rwX' /var/log/func/

Automating Installation with Cobbler

If you want to automate the process of installing Func on all your systems, you can use Cobbler [3] kickstart profiles.

The Cobbler configuration file has two separate settings for future Func clients:

# grep '^func' /etc/cobbler/settings
func_auto_setup: 1

Now you can add the Cobbler $func_install_if_enabled template to the %packages section of the kickstart profile, which is used for installing new machines. This step only installs the Func RPM if the func_auto_setup entry in the Cobbler configuration is set to 1. Adding a SNIPPET:func_register_if_enabled line to the %post section ensures that the client system registers with the Func master.

If the master is configured to automate the process of issuing client certificates, the system will immediately become available as a Func Minion, thus removing the need for manual configuration.

If you are using Cobbler, you'll need Fedora Everything available in your distribution for the installation of Func on Fedora to work. If you have Red Hat Enterprise Linux or CentOS, you'll need to set up the EPEL repository:

# cobbler repo add --name=el-5-x86_64-epel --mirror=

You can then introduce the repository to the required kickstart profiles to allow the installation of packages from this repository:

# cobbler profile edit --name=el5-ksprofile --repos="el-5-x86_64-epel"

Note that the $func_install_if_enabled template in the %packages section is needed in this case.

Modules, Please!

Once you're sure the management system and clients can talk via a secure channel, you should discover which modules a certain client has:

# func call system list_modules

The list should contain at least the standard modules on all your client systems, including yumcmd and command. Listing 3 shows a list of methods for a single module.

Listing 3: Multiple Methods per Module

01 # func call yumcmd list_methods
02 {'':
03     ['check_update',
04      'save_config',
05      'update',
06      'module_description',
07      'module_version',
08      'get_method_args',
09      'module_api_version',
10      'list_methods']}

Now, to update a certain package on this machine, do the following:

# func call yumcmd update httpd

Access to methods in other modules works in a similar way.

Many Hosts

Func really shines when you need to manage a large number of systems. You can leverage the normal shell file name expansions ("globbing") here, but note that all file name patterns must be enclosed in quotes. The following example queries all the LDAP systems in the domain for their kernel versions, except the systems that contain the slave string in their hostnames:

# func --exclude "*slave*" "*ldap*" show hardware kernelVersion

Func also lets you group machines for management. You can either edit the /etc/func/groups manually to create the required group, or leave the job to Func. The example shown in Listing 4 creates an LDAP server group with the hosts ldap1 and ldap2, displays the results, then copies the /etc/hosts file to all systems in the group.

Listing 4: LDAP Management

01 # func "*" group --ah "@ldap:ldap[1-2]"
02 # func "*" group --la
03 Group : ldap
04      Host :
05      Host :
06 # func "@ldap" copyfile --file=/etc/hosts --remotepath=/etc/hosts

Things really get interesting when you access the API provided by the management tool. Basically, you can address the API function for any method you can call at the command line. Bindings exist for the Python scripting language, for example. Listing 5 is a simple example that checks whether the LDAP service is active on all members of the LDAP group and, if not, launches the service:

Listing 5: Func API

01 #!/usr/bin/python
03 import func.overlord.client as fc
05 results = fc.Client("@ldap").service.status("dirsrv")
07 for (host, returns) in results.iteritems():
08   if returns == 0:
09    print host, "LDAP-Server is UP"
11   else:
12    print host, "LDAP-Server is DOWN - starting it up"
13    result = fc.Client(host).service.start("dirsrv")
14   if result[host] == 0:
15    print host, "LDAP-Server is now UP"
16   else:
17    print host, "LDAP-Server is still down, check manually..."
# python LDAP-Server is UP LDAP-Server is DOWN - starting it up LDAP-Server is now UP

If you enjoyed using the editor and would like to develop your own modules, that's no problem. A comprehensive guide for doing so in Python is available [4].


Func can be an enormous help, especially in large-scale system landscapes. Tasks that were formerly the domain of SSH loops are now easily completed at the command line. And, with the comprehensive API, the ability to program your own modules and add functionality makes Func's hack potential more or less infinite.