Open Space

In 2005, after an inordinate struggle to set up his own home networked file storage, Olivier Cochard-Labbé decided to create a solution to streamline the process. Much coding and configuring ensued, and he spawned FreeNAS – Free, as in freely available (open source) and based on FreeBSD, and NAS, as in Network Attached Storage.

Before long, a vibrant community formed around the project, and FreeNAS [1] began to take shape as an actual product. It came to include a range of services, from file sharing to media streaming.

By 2010, however, Cochard-Labbé was no longer able to commit sufficient time and resources to keep the FreeNAS project going, so he transferred control of it to iXsystems [2], a company well-known for its support of open source software in general and FreeBSD [3] in particular. A period of considerable development effort followed, which involved updating and modernizing the software and improving the project infrastructure to meet the needs of a growing FreeNAS development community.

The user interface was completely reimplemented, and features like disk encryption for ZFS pools and a third-party plugin system to support the installation of non-core services were added. Today, FreeNAS is under very active development, with hundreds of thousands of downloads for each release – more than 5.5 million in total since its inception.

Although FreeNAS was designed for home and small office environments, it does have a true business-class counterpart: TrueNAS [4]. Based on FreeNAS, TrueNAS is an enterprise-grade unified storage appliance also created by iXsystems, designed with the mission-critical IT environment in mind. TrueNAS adds enterprise features like hardware failover, performance tuning, reporting, 24/7 support, and other high-reliability enhancements that are made possible by having complete control of both software and hardware. However, a surprising number of maverick sys admins do use FreeNAS in business and enterprise environments, because it can be quite effective when employed with the necessary skill and care.

In this article, we introduce FreeNAS and provide an overview of those features most relevant to enterprise users. We also discuss ZFS, which is the primary filesystem used in FreeNAS and a significant source of its power. We will point out hardware or configuration considerations necessary to get the most from FreeNAS and, finally, describe how to set up and maintain a FreeNAS installation.

Storage Made Simple

A key feature of FreeNAS is ZFS (or "Zettabyte" File System). Originally developed by Sun Microsystems, ZFS [5] was designed for large storage capacity and to address many storage issues, such as silent data corruption, volume management, and the RAID 5 "write hole." FreeNAS 9.1.1, the most current release at the time of writing, uses all of the most advanced features ZFS has to offer.

The copy-on-write technology of ZFS makes possible many of its most impressive features. Because block pointers are not updated until the data is written to disk, the typical RAID write hole problem does not apply to ZFS. ZFS also supports a snapshot feature, which works by preventing the deletion of blocks that are part of files present at the time a snapshot is taken. As long as these blocks don't change afterward, preserving the snapshot uses virtually no space. ZFS deduplication works in a similar fashion, maintaining tables of which blocks are used in which files.

ZFS supports a variety of software RAID levels, including stripes, mirrors, and up to triple-parity RAID-Z. The FreeNAS ZFS Volume Manager (Figure 1) guides the user toward a configuration that is optimized for the number of disks available.

The FreeNAS graphical interface also supports the configuration of arbitrarily nested ZFS datasets within filesystems. ZFS datasets are like subdirectories that have configurable properties independent of their parent filesystem, such as the ability to configure compression or deduplication on a per-dataset basis. ZFS datasets can also be configured with quotas and reserved space, making them useful for thin-provisioning scenarios. Snapshots of datasets can be scheduled separately from the parent filesystem, or they can be included recursively in a snapshot of the parent volume.

As of version 8.3.1, FreeNAS also supports the creation of ZFS volumes on encrypted disks. This block-level disk encryption framework is based on Geli from FreeBSD and supports hardware acceleration on systems with AES-NI-capable processors.

Core Services Configuration

FreeNAS provides built-in support for a variety of network file-based protocols, including CIFS and NFS. CIFS (Common Internet File System) is commonly used in Windows environments or those with a mixture of Windows and Unix-like operating systems. NFS (Network File System) is commonly used for high-performance applications and in heterogeneous environments. FreeNAS also supports AFP for Mac OS X environments, including Time Machine backups. Each service can be configured from its corresponding wrench icon (Figure 2).

The data to be shared is configured through the Sharing section of the web UI. ZFS datasets allow for complex sharing scenarios, enabling the administrator to tailor a configuration to meet the network's needs, such as user/group quotas, user/group permissions, data properties, and customized snapshot and backup requirements.

For block-based applications, FreeNAS supports iSCSI target mode for file extents, physical disks, or zvols, a ZFS feature that creates a virtual disk device for which the blocks are stored in the parent filesystem. Regardless of which filesystem the target is formatted, shared block devices will adhere to the snapshot and backup policies on the storage side, with no interaction required from clients. CHAP and mutual CHAP are included for target and initiator authentication.

Built-in directory services support for Active Directory, LDAP, and NIS allows large organizations to integrate FreeNAS into their existing infrastructure. Other built-in services include UPS support, SNMP, and SSH as remote management alternatives, as well as rsync for more specialized replication and backup scenarios.

Save the Data: FreeNAS Backups

FreeNAS supports a wide variety of backup methods. As just mentioned, the rsync service provides built-in incremental backup capability. iSCSI extents can be used with native backup software on applicable platforms. A Mac OS X client can use its FreeNAS AFP share as a Time Machine target; finally, any network share can be used as a destination for an administrator's scripts or programs.

ZFS snapshots also provide a powerful tool for preserving and recovering older versions of data. ZFS snapshots preserve all the blocks in use by files at the time of the snapshot and can be sent to or received from other ZFS-supporting hosts (using SSH or another transport method) as a way of replicating or restoring data from a known point in time.

Snapshots take up only as much space as the metadata used to describe them and, of course, that allocated to the "older blocks" themselves. If you need to recover files from an older snapshot, you can easily clone and share it. If the current version of a filesystem becomes totally useless, the entire filesystem can also be rolled back to the state of a previous snapshot. These capabilities give admins a wide variety of options in creating data preservation and recovery policies.

FreeNAS supports scheduled snapshots in increments as small as five minutes, with preservation times of as little as an hour (Figure 3). As previously mentioned, snapshots can also be scheduled to replicate to a compatible remote system running the same version of ZFS. Snapshot replication only transmits the differences since the last snapshot, conserving bandwidth.

Great Responsibility: A ZFS Primer

A good working knowledge of ZFS is critical to deploying FreeNAS successfully in an enterprise environment. Poorly configured ZFS pools can experience suboptimal performance. Conversely, a well-planned deployment that takes best advantage of ZFS's features can be a definite success.

A ZFS filesystem resides in a pool, which may contain one or more groups of devices called vdevs. The possible types of vdev are individual disks, mirrors, RAID-Z1, RAID-Z2, and RAID-Z3 sets. All of the component groups of a ZFS pool are combined in a manner similar to striping, although the distribution of blocks is not a stripe in the strictest sense of traditional hardware RAID. This means that if even a single group within the pool is damaged beyond its tolerance for recovery, the entire pool is faulted. Mirroring and parity arrangements are only possible within groups, not among them. The upshot is that there is no equivalent of RAID 0+1 or 51 in ZFS – only device arrangements similar to RAID 10, 50, or 60.

By default, ZFS will use any idle RAM in the system as the primary Adaptive Replacement Cache (ARC). A common rule of thumb for ARC is to allow 1GB of RAM for each 1TB of data the system will store. Although solid state drives represent the leading edge of performance in data storage today, they do not yet have the capacity (or cost efficiency) to replace HDDs entirely. ZFS can therefore take best advantage of the higher performance of SSDs as secondary ZFS Intent Log (ZIL) and Level 2 Adaptive Replacement Cache (L2ARC) devices.

Such L2ARC devices can provide more space for caching and significantly improve read latency and throughput, especially random reads. In scenarios where an L2ARC device is used, you'll want to try to match the size of the device to the size of the working set (data that will be read frequently). ZFS supports multiple L2ARC devices, and L2ARC devices may be safely added or removed using the FreeNAS UI. FreeNAS also provides the zilstat command-line utility to help the administrator determine whether the system's workload would benefit from the addition of an L2ARC.

The ZIL records synchronous writes, fulfilling the guarantee that synchronous writes are committed to stable media before returning success. ZFS also allows the creation of a separate intent log (SLOG) on a separate device or set of devices. The SLOG reduces contention for the disk resources of the pool, but if a low-performing device is used for the ZIL, it can become a performance bottleneck. Ideally, you should choose a flash device with low latency and high throughput. Capacity is not critically important for ZIL devices, because the ZIL is flushed to the pool often (usually around every five seconds). Asynchronous writes do not benefit from ZIL devices.

ZFS supports deduplication, or the block-level removal of duplicate data. Deduplication is extremely resource intensive, and a rule of thumb calls for 5GB of RAM per terabyte of storage to be deduplicated. Think very carefully before using dedup on systems intended to back "live" applications, as it is more suitable for backup devices. As an alternative to deduplication, compression can be configured on ZFS pools or datasets.

The default compression method, LZ4, offers nearly real time compression rates. Deduplication and compression share a caveat: Blocks written to a pool where deduplication or compression are enabled will remain that way until the blocks are deleted. Thus, even temporarily enabling deduplication or compression can have long-lasting performance implications, reinforcing the need for proper planning in setting up ZFS and FreeNAS.

All the metadata needed by ZFS is stored in its pool, so a ZFS pool from one system can be exported and then imported by another operating system with a compatible ZFS version. In theory, this means that ZFS pools created on FreeNAS or another system using the same underlying ZFS version can be imported across operating systems and are compatible for the purpose of remote snapshot replication. In practice, you should always test and document such data recovery solutions before making them part of any business continuity or disaster recovery plan.

Encrypted disks are an exception to the rule of portability for ZFS pools. It is recommended that you use another FreeNAS system running the exact same OS (and therefore ZFS) version if the situation calls for mounting an encrypted ZFS pool elsewhere, because encryption methods and key management methodologies can change.

The ZFS Scrubs feature augments the functionality provided by fsck. A ZFS scrub checks every block on the disk, rather than just filesystem metadata. During a scrub, every block is compared with its checksum, which is stored with its block pointer elsewhere on the disk. If the checksum does not match, ZFS will attempt a repair and indicate the number of mismatches that it found in its scrub report. This feature can provide an early indication of a failing disk. Because it reads all the data from disk, a ZFS scrub is an I/O-intensive operation and is best scheduled for low-load periods. Scrubs should be performed regularly; FreeNAS automatically schedules a scrub for every ZFS volume on every fifth Sunday at midnight. You can use the FreeNAS UI to change the schedule as needed.

Putting It Together

When planning for any enterprise or business-critical deployment scenario, you must choose between various requirements for performance, data protection, capacity, and future growth, all within the constraints of a given budget. Much of the performance of FreeNAS rests on the parameters of the ZFS pool and the amount of RAM in the system. Certain use cases benefit from faster CPU or more CPU cores, and naturally, the network can also become a bottleneck without the right switch and NIC hardware.

For smaller working sets, more RAM or a fast L2ARC may significantly increase the performance of random, read-heavy workloads. Highly sequential workloads or workloads that often read a large portion of the stored data will benefit less from this approach. On the write side, synchronous write applications (i.e., NFS or iSCSI for virtualization purposes) will benefit from low-latency, high-throughput ZIL devices.

Regardless of the specific application, any FreeNAS deployment can benefit from using an optimal RAID-Z. When using RAID-Z, you should use equal-sized disks in groups of some power of two, plus the parity level of the group. Thus, the optimal RAID-Z1 group sizes are 3, 5, and 9 disks, RAID-Z2 are 4, 6, and 10 disks, and RAID-Z3 are 5, 7, 11 disks. When combining multiple RAID-Z groups in a pool, it is best to have an even number of identical groups to avoid a performance penalty and to allow ZFS to keep block arrangements as even as possible. Finally, like many filesystems, ZFS performance is proportional to the amount of free space. ZFS begins experiencing noticeable performance degradation when the pool is around 80 percent full. Take this into account when planning for future growth.

For networking, FreeNAS supports both Gigabit and 10 Gigabit Ethernet. Network configuration can be performed from the console as well as from the web user interface. FreeNAS also supports a number of link aggregation protocols, including LACP, link failover, and round-robin. Link aggregation requires a properly configured switch or router. The ability to saturate a given network connection depends on the rest of the performance considerations discussed above.

Setup and Maintenance

One of the easiest parts of a FreeNAS deployment is the actual installation. By design, the FreeNAS boot disk is separate from the disks in the storage pool. This segregation ensures that operating system upgrades do not threaten storage: In the worst case of a completely failed upgrade, the storage disks will be untouched and may be imported on a fresh FreeNAS instance. The boot disk image is typically written to a USB thumb drive or SSD of at least 2GB in size. Performance of the boot device itself is of almost no concern because FreeNAS runs from memory after boot.

After booting, the FreeNAS console, seen in Figure 4, will display options to configure networking, reset the system, access a root shell, reboot, or shut down. Once the system has received an IP address from a DHCP server, the address will be displayed on this screen and can be used to access the web interface. The rest of the configuration must be performed through the web interface. The FreeNAS documentation [6] contains both a Quick Start Guide and comprehensive instructions for configuring a FreeNAS system.

As the system is configured, a copy of the configuration can be saved at any point in the configuration to the administrator's system. This lets you easily restore a previously working configuration. Similarly, if the disks are encrypted, make sure to save a copy of the passphrase and keys.

FreeNAS also provides a "GUI Upgrade" option through the web interface. The upgrade process reminds you to download a copy of the configuration database before performing the upgrade. If the upgrade fails, simply reboot and select the previous version of the operating system from the boot menu.

Beyond NAS: Plugins

Out of the box, FreeNAS contains many of the most popular services for NAS. Although they cover the vast majority of use cases, additional customization may be required. To that end, FreeNAS includes a plugin system with support for a wide variety of add-on software. Each plugin runs in its own FreeBSD jail, which allows it to be run isolated from the host operating system.

In FreeNAS, jails are installed on individually segregated ZFS datasets. The admin creates transparent mount points to storage outside of the jail's dataset, all of which can be done from the web UI. The plugin API also allows a plugin to communicate through an IPC interface to the web UI, allowing it to be configured and enabled/disabled as a service without resorting to the command line. Further details of the FreeNAS plugin system can be found in the FreeNAS documentation.

Final Thoughts

FreeNAS provides powerful administrative tools. As with any power tool, the more knowledge and resources you have on using it properly, the less likely you are to suffer an accident. Unlike with a "boxed solution" like TrueNAS, a FreeNAS deployment is completely up to you to design from start to finish. Thus, you have incredible freedom in designing the solution that's right for you or your organization's needs; however, the responsibility for doing it right, with the right type of hardware, is also yours!

We hope that this article has given you some of the basic information you need to get started creating a high-quality FreeNAS system, all based on open technologies that you can be proud of.