Safely Behind Bars

When managing access privileges on Unix, and thus on FreeBSD, you have basically two types of users: those with and those without administrative privileges. This model clearly reaches its limits, however, if you need to create a web administrator, for example. The web admin needs permissions to be able to change certain configuration files or start the HTTP daemon but should not be permitted to change the system configuration.

A solution to this problem would be adding more granularity to access privileges. In FreeBSD, you have the option of working with File Access Control Lists (FACLs) or deploying the Capsicum capability and sandboxing framework. Unfortunately, these approaches substantially increase the administrative overhead, which in turn could have a negative effect on security.

The chroot environment was invented as a way out of this dilemma, but it has some deficiencies from a security standpoint. For example, it is known that an FTP server that allows anonymous access can let users break out of the chroot environment. In the course of time, some improvements were added to chroot, but problems such as the influence of processes outside of chroot were ultimately not resolved.

The Idea Behind Jails

This is where is the concept of the jail comes in: Jails use the positive properties of chroot and at the same time provide absolute protection against manipulation of processes outside the jail. Because a jail relies on a subdirectory tree, a process within the jail cannot access directories and files on the outside. Furthermore, a process inside a jail cannot manipulate the host processes [1]-[5]. Jails thus offer a useful option for providing network services.

However, a jail will not increase the security of a daemon itself. If an FTP daemon has a vulnerability, then it will still have it in the jail, and an attacker could exploit this vulnerability and thus obtain access to the jail, where they might even escalate to root. However, you would still have a major security advantage, in that the attacker could only go about their dastardly deeds in the jail. They would not have access to the host system! In contrast to more granular access management, this approach does not involve much more overhead. Processes are unable to break out of the jail or influence other processes outside of the jail. (See also the "Tools for Managing Jails" box.)

Within a jail is a root user, albeit with a very restricted account. Among other things, the user cannot manipulate the host or IP address of the jail. Certain sysctl MIBs let you restrict root's powers in the jail. It is possible to assign IPv4 or IPv6 addresses (or multiple IPv4 or IPv6 addresses) to a jail, thus creating a routing-enabled jail. The address is passed in on launching the jail. The loopback address (127.0.0.1) and its IPv6 counterpart (::1) can be linked with a jail. This is referred to as an internal jail.

Inside a jail, the user has access to a complete FreeBSD, which means that jails offer a kind of virtualization, like VMware or Xen. As of FreeBSD 8, you can even create jails within a jail. The concept of jails introduced in FreeBSD 4 has long since convinced vendors of other operating systems, too. In Solaris, for example, jails are known as zones.

Restrictions

Within a jail, important limitations exist as a result of the implementation. Remote Procedure Calls (RPCs) do not work in jail operations for security reasons. Thus, you have no way of using NFS within a jail. Daemon processes on the host must be configured very carefully to avoid address conflicts between the jail and the host. Loading and unloading of kernel modules within a jail are prohibited, as is creating device nodes. Mounting and unmounting of filesystems is not possible. Modifying the network configuration, network addresses, or even routing tables is prohibited. Access to raw sockets on the host system is no longer permitted, but within the jail, it is possible to access these types of sockets. Addressing semaphores of the host system is also not allowed.

On closer inspection, most of these restrictions will result in security gains compared with a chroot environment.

Implementing Jails

The system consists of two components: the jail tool, which is the user interface between the kernel and userland, and kernel code, which is started by system call. The jail tool requires the following parameters: the path to the jail, the hostname of the jail, the corresponding IP address, and the command to be run.

As you can see in the excerpt from the source code (Listing 1), the jail function call contains a jail structure (struct jail: /usr/include/sys/jail.h), in which the above-mentioned parameters are stored.

01 main(int argc, char **argv)
02 {
03   struct jail j;
04   int i;
05   [...]
06   /* Populate the jail struct */
07   memset(&j, 0, sizeof(j));
08   j.version = 0;
09   j.path = <path to jail>;
10   j.hostname = <hostname of jail>;
11   j.ip_number = <IP address of jail>;
12   /* call system call */
13   i = jail(&j);
14   [...]
15   execv(<command>, ...);
16   [...]
17   exit(0);
18 }

The jail tool locks itself up (i = jail(&j)) and creates a child process in this environment that then runs the desired command using execv(). As mentioned previously, the jail kernel function is called:

int jail(struct proc *p, struct *jail_args uap);

This function copies information stored in the kernel memory space uap (struct jail) and stores some of the information in a C struct named prison. Then, chroot() is called with two parameters: the path of the jail and the calling process.

Locking a Process

In FreeBSD, each process is represented by a C struct (struct proc), which is described in /usr/include/sys/proc.h. It contains a pointer that points to the prison structure. The p_flag field with the value P_JAILED shows the process manager that a process belonging to the structure must be run in a jail.

The high degree of protection has its roots in the way the FreeBSD kernel process manager handles the C struct proc. Once a process has been assigned compute time by the time-slice procedure, proc->prison->p_flag is checked to see whether the process belongs to a jail. For processes that were started within a jail, this flag is set unconditionally.

Many other kernel services decide on access on the basis of these flags and on whether and how resources may be accessed. An excerpt from the source code illustrates the sequence (Listing 2). Figure 1 schematically represents the communication pathways.

01 int jail(struct thread *td,
02   struct jail_args *uap)
03 {
04   struct prison *pr, *tpr;
05   struct jail j;
06   struct jail_attach_args jaa;
07   [...]
08   error = copyin(uap->jail, &j,
09     sizeof(j));
10   [...]
11   MALLOC(pr, struct prison *,
12     sizeof(*pr),
13     M_PRISON, M_WAITOK | M_ZERO);
14   [...]
15   error = copyinstr(j.path,
16     &pr->pr_path, sizeof(pr->pr_path), 0);
17   [...]
18   error = copyinstr(j.hostname,
19     &pr->pr_host, sizeof(pr->pr_host), 0);
20   [...]
21   pr->pr_ip = j.ip_number;
22   pr->pr_linux = NULL;
23   pr->pr_securelevel = securelevel;
24   [...]
25   error = jail_attach(td, &jaa);
26   [...]
27   FREE(pr, M_PRISON);
28   [...]
29   return (error);
30 }
31
32 int jail_attach(struct thread *td,
33   struct jail_attach_args *uap)
34 {
35   struct proc *p;
36   struct prison *pr;
37
38   [...]
39   p = td->td_proc;
40   [...]
41   error = change_dir(pr->pr_root, td));
42   [...]
43   change_root(pr->pr_root, td);
44   [...]
45   return (error);
46 }

Installation, Configuration, and Updates

Installing jails is not difficult, but the configuration requires care. After all, it's all about security. Commands can only be executed by the root user.

To set up a jail, you need at least 150MB of free disk space – as you add utilities, you will obviously need more. As an example, I'll look at a jail that houses web services. You need approximately 4GB for this setup, which makes it possible to run a FAMP server (FreeBSD-Apache-MySQL-PHP) or a content management system (CMS). Furthermore, the operating system and kernel source code must be installed. The approach is described in detail in the manual [1]. To create a jail, complete the steps shown in Listing 3.

01 # mkdir -p /jail/www
02 # cd /usr/src
03 # make world DESTDIR=/jail/www
04 # cd etc
05 # make distribution DESTDIR=/jail/www
06 # mount_devfs devfs /jail/www/dev
07 # cd /jail/www
08 # ln -sf dev/null kernel

After everything is compiled and installed, the jail can access devices via the device filesystem (devfs). This mount must complete before the jail starts. Because some programs expect a /kernel file, you can simulate this with a symbolic link. In fact, the kernel on the host system is responsible for all jails.

Before the jail is launched for the first time, you need to check some of the utilities on the host to see whether they are bound to the host IP address. This is done with the simple command:

# sockstat | grep "\*:[0-9]"

The description for rc.conf(5) lists the services that need to be bound to a fixed IP address via parameters. One example is the inetd service:

inetd_flags="-wW -a <IP_address> <host_system>"

The Sendmail service on the host system should be started so that it only listens on the localhost address (sendmail_enable= "NO"). To complete the installation of the jail, you can now configure the network interface on the host system.

For a routing-enabled jail, run the following command to assign an IP address,

# ifconfig netif0 inet alias  <IP_address jail>/32

where netif0 is the network interface. You should add this to the /etc/rc.conf file on the host system:

ifconfig_netif0_alias0="inet <IP_address jail> netmask255.255.255.255"

In contrast, an internal jail is always assigned the IP address of the loopback interface,

# ifconfig lo0 inet alias 127.0.0.1/32

where lo0 is the loopback interface. Again, you should add this to the /etc/rc.conf file on the host system:

ifconfig_lo0_alias0=  "inet  127.0.0.1 netmask 255.255.255.255"

For the following steps, it doesn't matter whether you are setting up an internal or a routing-enabled jail.

After completing the installation to this point, manually launch the jail with the hostname http://www.homenet.net, an IP address of 192.168.1.200, and the shell /bin/sh for the configuration:

# jail /jail/www www.homenet.net 192.168.1.200 /bin/sh

From now on, the administrator is inside the jail. First, set the password for the root user. You will also want to create a user who belongs to the wheel group, like root, for logging in later via SSH.

To avoid warnings about a non-existing filesystem table, create an empty /etc/fstab. The /etc/rc.conf file should now look like Listing 4.

01 rpcbind_enable="NO"
02 network_interfaces=""
03 hostname="www.homenet.net"
04 sshd_enable="YES"
05 sshd_flags="-p <Port Jail>"
06 sendmail_enable="NO"
07 syslogd_enable="YES"
08 syslogd_program="/usr/sbin/syslogd"
09 syslogd_flags="-ss"
10 defaultrouter="<IP Address Router>"

Additionally, you need to change the configuration for the SSH daemon in the /etc/ssh/sshd_config file

ListenAddress <IP_address jail>

and add the domain name server(s) to the /etc/resolv.conf file:

nameserver <IP_address DNS>

The syslogd daemon creates logfiles in a jail just as on the host system. You would not want to log in to every single jail to check what is happening inside. It's easier to redirect the syslogd output to the host. In the system configuration for the jail (/etc/rc.conf), you enable the daemon with syslogd_enable="YES". In /etc/syslog.conf, you can enter:

*.* @<address of Syslog host>

and then restart the daemon by typing /etc/rc.d/syslogd restart. This step completes the configuration within the jail, and you can get out of jail free by typing exit. For a final test, start the jail in the same way it will start later when you boot the system

# jail /jail/www www.homenet.net 192.168.1.200 /bin/sh /etc/rc

To log in to the jail, type:

ssh root@www.homenet.net -p<Port Jail>

If everything was configured correctly, the jail prompt should appear.

On the host system, you still need to complete some final configuration steps. Because the jail will be started the next time the system boots, you need to add the lines shown in Listing 5 to /etc/rc.conf.

01 jail_enable="YES" jail_list="www" jail_www_rootdir="/jails/www"
02 jail_www_hostname="<hostname jail>" jail_www_ip="<IP address jail>"
03 jail_www_exec="/bin/sh /etc/rc" jail_www_devfs_enable="YES"
04 jail_www_devfs_ruleset="devfsrules_jail"

The jail_enable="YES" entry must be set to start a jail. If you want to launch multiple jails on the host system, add them to a space-separated list under the keyword jail_list. They are then launched in the specified order. The last line is interesting; it defines the ruleset for the device filesystem service.

For everything to work correctly, you need to add the contents of /etc/defaults/devfs.rules to the existing configuration in /etc/devfs.rules. The other details are specific to each individual jail. However, you should note that the name specified in jail_list must match the name used elsewhere, as you can see in the example.

You also need to replace or add the details for the syslog daemon in the /etc/rc.conf configuration:

syslogd_flags=  "-a <network IP address>/24 -4-b <host_address>"

Then, restart the syslog daemon.

After entering all the parameters in /etc/rc.conf, you can start the jail:

# /etc/rc.d/jail start <name of jail>

To stop a jail, type stop<name of jail>. Omit the name to stop all jails.

Software can be installed in the same way as on the host system – either in the form of binaries using pkg_add -r <package_name> or via the ports. However, this approach requires the ports tree from the host system to be mounted in the jail, and the /usr/ports directory must exist in the jail. The nullfs filesystem exists for this purpose. On the host system, run

# mount_nullfs -o ro /usr/ports /jail/www/usr/ports

and add the following entry to /etc/fstab on the host:

/jails/www/usr/ports /usr/ports  nullfs rw 0 0

You also need to make two changes to /etc/make.conf in the jail:

WRKDIRPREFIX=/tmp
DISTDIR=/tmp/distfiles

If you update the host system, you also need to update the jail. To do so, change to the /usr/src directory as the root user. Because make buildworld has typically already been run on the host, it is no longer necessary for the jail. You just need:

# cd /usr/src
# mergemaster -p -D <path to jail>
# make installworld DESTDIR= <path to jail>
# mergemaster -D <path to jail>

This completes the jail update.

Sometimes, it is important to see whether processes are actually running in the jail and – if so – which ones. The ps -ax command can help you here. The following excerpt from the process list shows which processes are running in a jail. You can see this by the additional J in the status (STAT) column.

# ps -ax
708 ?? SsJ 0:00,42 /usr/sbin/syslogd ...
722 ?? SsJ 0:17,19 /usr/sbin/named ...
774 ?? SsJ 0:00,12 /usr/sbin/sshd ...
781 ?? IsJ 0:00,87 /usr/sbin/cron -s
...

The jls command lets you view all the currently active jails, as shown in Listing 6.

# jls
JID IP Address    Hostname            Path
1   192.168.1.201 dns.domain.tld      /home/jail/dns
2   192.168.1.202 www.domain.tld      /home/jail/www
3   192.168.1.203 xwindow.domain.tld  /home/jail/xwindow

Jails, Simplified

An alternative and much easier way of creating jails on a host relies on the ezjail script collection. It is located in the sysutils section and can be quickly installed using portinstall or pkg_add -r. The following steps are performed as root. In the first step, you create a basic jail, which acts as a template for all further jails. To do this, run:

# ezjail-admin update -pP

This process takes about two to five hours, depending on the performance of the system, because running make world recompiles everything. Using the -i parameter instead of -pP works around this procedure and only runs a make installworld. For the next steps, again note that the difference between internal and routing-enabled jails lies only in the IP address assignments. The next step creates a jail:

# ezjail-admin create <jail_name> <IP_address>

As described previously, you need to enter the appropriate parameters for the network interface and the jails in /etc/rc.conf. Incidentally, there is no difference. Next, enter:

# Start jails with ezjail
ezjail_enable="YES"

Now you can start the jails using

/usr/local/etc/rc.d/ezjail.sh start

The alternative is ezjail-admin start.

The update process is a breeze with ezjail. A simple call to ezjail-admin update -i is all it takes to update the jail. However, the applications in the jail must be updated in a further step.

As experienced administrators will quickly see, the ezjail script collection takes much of the work off your hands. You no longer have to enter the individual make commands and run them.

Many Services, One Jail

As already described, among other things, a jail is used to protect hosts against attacks on specific services. The best-known examples are the DNS service, bind, and web servers. It is important that all of these services are always and exclusively bound to the IP address of the jail.

The next example installs the DNS service, bind, in a jail. On small networks, you don't strictly need to assign a separate machine for bind; it is sufficient to create a jail on the central FreeBSD server. Primary and secondary DNS should never run in the same jail on a host system. If the host system failed, both primary and secondary DNS would be inaccessible. Add the following entry to the bind configuration file:

listen-on port 53 { <IP_address_of_jail>; };

You still need to modify the /etc/resolv.conf file on each computer on the network – including the host system!

It is possible to operate a DHCP server a jail [7]. To do this, you need to build the ISC-DHCP3 server port from the /usr/ports/net/isc-dhcp3-server directory:

# make install -DDHCP_PARANOIA  -DDHCP_JAIL -DDHCP_SOCKETS
# make clean

In the jail, add the system configuration, /etc/rc.conf, as shown in Listing 7.

01 dhcpd_enable="YES"
02 dhcpd_flags="-q"
03 dhcpd_conf="/usr/local/etc/dhcpd.conf"
04 dhcpd_umask="022"
05 dhcpd_chuser_enable="YES"
06 dhcpd_withuser="dhcpd"
07 dhcpd_withgroup="dhcpd"
08 dhcpd_devfs_enable="YES"
09 dhcpd_makedev_enable="NO"
10 dhcpd_chroot_enable="NO"
11 dhcpd_jail_enable="NO"
12 dhcpd_rootdir="/var/db/dhcpd"
13 dhcpd_hostname="dns"
14 dhcpd_ipaddress="<IP-address-of-jail>"

Add the following line to the DHCP daemon configuration file, /usr/local/etc/dhcpd.conf:

local-address <IP_address jail>;

Then, on the host system, enable the kernel driver for the Berkeley Packet Filter (bpf), and add this entry to the ruleset for the devfs daemon [devfsrules_jail= 4] section: add path 'bpf*' unhide.

You also need to bind the Apache or Lighttp HTTP services to an address. The entry in httpd.conf looks like this:

Listen <IP_address of jail>:80

You can also install a mail server within a jail. However, the configuration is more complex than for the previous examples. I'll briefly look at the procedure for a mail server with Postfix MTA [8] including anti-spam and anti-virus scanners (see the "Rules for the PF and IPFW Packet Filters" box).

You need to replace the localhost address with the jail IP address in the Postfix configuration files, /usr/local/etc/postfix/main.cf and /usr/local/etc/postfix/master.cf. Also, the hostname localhost must be replaced with the jail hostname, unless the jail uses the localhost IP address. The same thing applies to configuring the anti-spam filter and anti-virus scanner. To allow the system to continue to send email on the host system, the DNS service on the network needs an MX record.

Of course, you could install all of the services in separate jails on a host system for a small office or home network. Figure 2 shows the configuration.

I have read in various forums [9] about administrators who run database servers in jails. This is something to avoid, because some database systems use System V IPC to communicate and because the data in a database needs special protection. The recommended approach is to install the database system on the host and communicate with the database via an IP address. Similar precautions also apply to database servers that are managed via a web interface. The WWW service should definitely run in a jail: If a vulnerability in the daemon process is then exploited, at least there is no risk to the database data. Figure 3 shows this configuration.

Services based on the RPC protocol cannot be operated in a jail. This is especially true for Network Information Service (NIS). If you really want user administration within a jail, then you either need to copy the host system's /etc/passwd to the jail or install LDAP. The procedure for LDAP is the same as installing on a client system. It is important to note that the jail cannot use user IDs that exist on the host. The reason is simple: If an SSH service runs on the host and a user account is compromised, the attacker could also easily log in to and damage the host system. The best approach is thus to set up the minimum number of user accounts and avoid using the same usernames. This cannot be avoided for system accounts and the root user. In this context, it is extremely important to use different passwords for the root user on the host and in the jail. (Also see the "Important Flags for Jails" box.)

Windowing with Jails

Jails are not just useful for security purposes, they also provide a useful approach to building training environments. Each participant can be assigned to a jail where they complete their assignments. Graphical user interfaces (GUIs) based on X11 can also be installed in a jail for this purpose. One important point here is that the X server does not work in a jail because it requires direct access to the hardware. Thanks to the X11 protocol, though, you can redirect the output to, for example, a thin client. The configuration is not particularly difficult, but there are a couple of pitfalls.

First, install the desired X11 applications in the jail either via the ports tree or using pkg_add -r. In both cases, all required libraries and programs are automatically installed to resolve dependencies. You also need the xauth program to support trouble-free authentication. It can be found in /usr/ports/X11/xauth. The next step is to change a parameter in the /etc/ssh/sshd_config settings for the SSH daemon: X11Forwarding yes. This forwards all X11 protocol tokens through the SSH tunnel. Note: At this point, name resolution must be working properly!

On the client side, you need to have the xauth and xhost tools, as well as a display manager in place (xdm, gdm, or kdm depending on your preference). Add the following to the SSH client configuration:

Host *
   ForwardAgent yes
   ForwardX11 yes
   XAuthLocation /usr/bin/xauth
   ForwardX11Trusted yes

Next, tell the X server on the clients to accept data from the X11 applications in the jail:

$ xhost +<IP_address of jail>

If you then use ssh user@x11jail <X11_application> to log in to the jail, the window of the X11 application launched in the jail appears on the client desktop (Figure 4).

Building on this setup, you can run Linux applications in a jail with just a few clicks, in case you have applications that are available for Linux, but not FreeBSD. This setup relies on FreeBSD libraries that provide Linux compatibility, so you need to install the linux_base port on the host,

# cd /usr/ports/compat/linux_base
# make install clean

then mount the directory with the libraries in the jail directory:

# mount_nullfs /compat/ <path to jail>/compat/
# mount -t linprocfs linprocfs <path to jail>/compat/linux/proc

It is also important to mount the Linux process filesystem because Linux process management maps to a filesystem. To keep the mount after a restart, you can modify the /etc/fstab on the hosts accordingly with the entries each occupying a single line:

<path to jail>/compat/ /compat nullfs rw 0 0
linprocfs <path to jail>/compat/ linux/proc linprocfs rw 0 0

Also add the following line to the host's /etc/rc.conf system configuration: linux_enable="YES".

Two Worlds, One System

The examples shown in the previous section can be extended to provide a kind of virtualization solution. The jail solution on FreeBSD allows you to create jails that will run 32-bit applications on 64-bit hosts. The procedure for the installation and configuration is the same, except for a few minor differences (Listing 8).

# setenv UNAME_m i386
# setenv UNAME_p i386
# cd /usr/src
# make buildworld TARGET=i386
# make installworld TARGET=i386 DESTDIR=<path to jail>
# cd etc/
# make distribution DESTDIR=<path to jail>
# mount -t devfs devfs <path to jail>/dev

To begin, you need to set the environmental variables so the uname command is output correctly. After changing to the directory /usr/src, you need a couple of calls to make. To create a 32-bit jail, use the TARGET=i386 make parameter. (The expression i386 is a synonym for 32-bit technology.) The remaining steps follow the pattern shown in the listing, and the configuration steps within the jail follow the same approach as described previously.

Conclusions

The jail solution on FreeBSD is not only a security concept, it also provides a small virtualization solution. It gives data center administrators a powerful tool and allows Internet service providers to offer a root shell to their customers. The model's design ensures full security.

Jails also offer administrators on home networks a significant security advantage, for example, by locking up the DNS service and a web browser in a jail. These features all demonstrate that jails on FreeBSD can provide a genuine benefit with versatile applications.