Package Insurance

Many distributions develop, test, build, and distribute their software via a heterogeneous zoo of servers, mirrors, and workstations that make central management and protection of the end product almost impossible. In terms of personnel, distributions also depend on the collaboration of a severely limited number of international helpers. This technical and human diversity creates a massive door for external and internal attackers who seek to infect popular distribution packages with malware. During updates, then, hundreds of thousands of Linux machines download and install poisoned software with root privileges. The damage could hardly be greater.

The danger is less abstract than some might think. Repeatedly in the past, projects have had to take down one or more servers after hacker attacks. The motivation of (at least) all the major distributions to protect themselves from planted packages is correspondingly large and boils down to two actions: one simple and one cryptographic.

Advanced Mathematics

Armed with a checksum, users can determine whether a package has passed through the Internet without error. The MD5, SHA1, and SHA256 hash methods are popular ways to calculate a checksum for a package.

Because checksums provide no protection against intentional tampering, Arch Linux and Debian and its derivatives also sign their packages and repositories. They naturally use public key cryptography to do so. The basis is a key pair. Using the private key, which is kept safely, the project team signs the new packages or the repository. With the public key, which is accessible to everyone through the distribution sites and installation media, users can check whether the signature originates from the owner of the private key and thus comes from the project.

Secrecy

To improve security, some distributions rely on a chain of signatures and checksums, although the project websites reveal little detail about how exactly they do this or what the technical implementation looks like, with sparse information spread over several subpages and obsolete pages. Therefore, you have no way to learn about the internal organizational processes concerning which team member signed which packages, with which key, and when or whether it even happens automatically. The processes at least can be reconstructed to a certain extent for Arch Linux, Debian, Fedora, openSUSE, and Ubuntu distributions.

Arch Linux

The makers of the Slackware-style rolling release distro with Debian-style package management rely on GnuPG and the web of trust concept [1]. A developer first puts together a suitable package for their software and self-signs it using GnuPG. Then, the developer uploads the package to the Arch User Repository (AUR) and initially retains responsibility for the package. If the package proves itself in the repository, it moves into the community repository.

The AUR and community repository are managed by selected Arch Linux users referred to as trusted users [2], each of which must sign the packages they manage with their own GnuPG key. If the Arch Linux team happens to classify one of the packages as particularly important, the package is promoted to one of the other official repositories, such as core or extra. The packages in these repositories are managed in turn by the Core Arch Linux developers.

The project has five official GnuPG keys, known as the master signing keys (Figure 1). Each key belongs to one of five Arch Linux developers [3]. At least three of these developers always sign a key belonging to the package developers and the trusted users, who in turn sign the packages themselves with their certified keys. The signature for a package is kept in a separate file with a .sig suffix; the repository serves this up along with the package.

Because the web of trust is created in this way, the Pacman package manager can now check the signatures: As soon as a package is signed with a key, which in turn is signed with at least one valid master signing key, it is therefore assumed to come from the Arch Linux project.

Each of the master signing keys has a revocation certificate that lies in the trusty hands of another independent Arch Linux developer. This setup prevents one of the master key holders obtaining sole power over the certification process. The packages that reside in a repository are revealed by a small database. It contains the corresponding MD5 and SHA256 checksums for each package.

Debian

To discover which packages are offered by a Debian repository, you need to check out the files distributed across several subdirectories with the name packages. The files also provide the MD5, SHA1, and SHA256 checksums determined for each package. The locations of the packages files are in turn revealed by a file named release, which tells you the associated checksum for each package file.

The Debian team signs release files with an automatically generated key, which the documentation alternatively refers to as the automatic signing key or "ftp-master" key [4]. Currently, each Debian release has only one FTP master key (Figure 2); formerly, the project team produced a new FTP master key every year.

The team signs the release files of every stable Debian version with the automatic signing key in use at the time of release. Additionally, the Debian team signs stable releases with a stable key that is generated for the release. In any case, the signatures end up in a separate file: Release.gpg.

Debian only signs the release files for special releases (proposed-updates, testing, testing-proposed-updates, unstable, experimental) and the security archive with the FTP master key. The keys can be viewed and downloaded [4] for all Debian releases (Figure 2).

Oh, Really!

The FTP Master team, which is responsible for operating the Debian archive [5], is responsible for signing and key management. You can determine the authenticity of a package by inspecting the chain of signatures and checksums: If the signature of the release file is okay, the package manager can trust the content. In turn, the release file contains the checksums of the packages files with the actual packages. If the checksums match, the content of the packages files is also unchanged.

The checksums contained in the packages let the user track the integrity of individual packages. Also, every package comes with a text file, which stores the MD5 checksums of all included files.

Before a package lands in an official Debian repository, the developers need to find an approved maintainer. As the sponsor, the maintainer checks the package and then uploads it to a repository after assessment. The Debian project distinguishes between two types of package maintainers: A Debian maintainer (DM) can upload only their own packages, whereas a Debian developer (DD) as a Debian member can upload any package.

Fedora

As soon as work starts on a new version of Fedora, the developers create a new GnuPG key [6] and use it to sign each RPM package published by the project. The key is used for the public test versions and for the final version.

Each architecture has its own key. In any case, the signatures are included in the metadata of the RPM package. Only some bleeding edge packages in the Rawhide distro do not have a signature.

It is the task of the package manager – dnf, rpm, and the corresponding GUI tools – to check the signatures [7]. Fedora stores the required public key in the /etc/pki/rpm-gpg directory after installing (Figure 3). Alternatively, users can find the public key online [8].

The Fedora team uses the Sigul [6] tool to manage and produce the keys. All hand-picked members of the project who need to sign packages (aka the release engineers [9]) also have access to the key.

openSUSE

Each individual SUSE RPM package is signed with a GnuPG signature, which can be found in the metadata of the RPM package. Typically, SUSE distributions are created automatically in the Open Build Service [10], the public development platform operated by openSUSE. The GPL software of the system automatically signs all the packages and media that pass through it.

When asked, Ludwig Nussel, the release manager of openSUSE Leap, reported that a private signing host is connected to the build service via a separate cable; the private portion of the official key pair resides on this host. Only employees with special security clearance have access to the server room, not even release chief Nussel is allowed in. He assured us that nothing runs on this computer (not even an SSH server) except the signing service. Logins are possible only via a serial console.

SUSE Key as a Backup

In any openSUSE installation, the public parts of two keys are preinstalled in the content.key file: The openSUSE project signing key opensuse@opensuse.org and the SUSE package signing key build@suse.de. SUSE has so far seen no reason to renew the keys, but they do have an expiry date, which authorized persons regularly extend.

Should the private openSUSE key be compromised, the project would use the SUSE key to distribute an update that deletes the compromised keys and rolls out a new openSUSE project key. SUSE's private key is stored in a separate infrastructure, like that described above.

Project Weaves Chains of Trust

Multi-level security similar to the Debian project is founded on the keys in the repositories and on installation media [11]: The core of this structure is the content file, which reveals the directory containing the packages. The Open Build Service signs the file before it is delivered, and the signature ends up in the content.asc file. The content file references other files containing the file names of all packages. For each of these files, content provides the appropriate SHA256 checksum.

The content file also references files with other public keys, which Yast imports without complaint, because content already has a trusted signature. The keys are designed to help users check the signatures in the RPM packages. Next, Yast uses the matching packages.gz file, as designated by the content file. It contains all of the packages in the repository with their corresponding SHA1 checksums.

In this way, a chain of trust is built up as in Debian. Packages that cannot prove their origin at install time are rejected by Yast; rpm also notices attempted fraud, but it is more forgiving in its response (Figure 4).

Beyond RPM

The openSUSE project relies on the same principle as RPM to secures its repositories in the repomd and Yum formats. A signature from the separate repomd.xml.asc file underpins the central repomd.xml file. The public key required for verification is either identical to the one provided by the installation medium (e.g., an update repository), or it is located in the repomd.xml.asc file. If the key is unknown to the system, the user has to trust it explicitly.

The repomd.xml file contains a list with other files in addition to their corresponding SHA256 checksums. These additional files are then given, among other things, a list of all the packages contained in the repository. Again, a chain of trust is built: If the signature of the repomd.xml file is okay, the checksums it contains are trustworthy. In turn, you can use these checksums to discover whether the associated files in the repository have been tampered with.

Ubuntu

Ubuntu secures its packages on the same principle as its Debian roots, but Canonical signs the release file. Because Ubuntu takes over most of the packages from the Debian repositories, Canonical even recommends that Ubuntu developers submit their programs [12] to Debian first.

The personal package archives (PPAs) are a minor exception. Each developer can set up their own repository on the Launchpad platform and deliver their software for Ubuntu users. Launchpad creates its own cryptographic key for each PPA, which the platform uses to sign each package offered by the PPA automatically [13]. In the background, the same system as in Debian is in place here: A PPA comprises a small repository with a release file, which Launchpad signs with the PPA key.

In contrast, developers need to upload the new Snaps [14] to the Canonical store much like smartphone apps. There, the software goes through an automatic – and on a case-by-case basis, manual – review process.

Principle of Hope

Foisting a poisoned backdoor package onto a distribution would be the perfect attack vector for a hacker. The motivation of each system to prevent such a scenario should be pretty high. In their wikis, all distributions discussed here describe in detail how users should check the origins of a package, but they provide surprisingly little information about their own organizational and technical background processes. This kind of obfuscation is a poor match with the kind of transparency that open source seeks to achieve and might possibly originate from a desperate attempt at achieving security through obscurity.

Debian, Ubuntu, and openSUSE use only a single GnuPG key to sign a central information file in the repository. Starting with this file, the package manager then checks the individual packages, which makes the central key the most attractive point of attack.

Only Arch Linux tries to defuse the problem with five master keys. However, it remains unclear in many cases where and how the project keeps the private key it uses for signing.