Features Container Security Lead image: Lead Image © Kae Horng Mau, 123RF.com

Security issues when dealing with Docker images

The Crux with Leaks

Although developers appreciate Docker's ease of use and flexibility, many admins are worried about vulnerabilities. We look at various approaches to securing container images and the price to be paid. By Nils Magnus

Docker Hub is easy for users, and the docker command-line tool can directly tap into it. You can easily pick up prebuilt images for CMS, databases, or distributions and import them into your local infrastructure. But what guarantees do users have that the software running in the container is also free of vulnerabilities?

Threat Modeling

To start, you need to distinguish between threats; security experts refer to this as a threat model. In this case, there are three threat scenarios:

The manufacturer embeds malicious code and offers infected images.
Attackers tamper with the software en route from the manufacturer to the user.
The manufacturer fails to eliminate known security vulnerabilities in its images.

Users need to select software vendors they trust for effective protection against the first case. Well-known and reputable companies would be reluctant to compromise their reputations, but a distant dubious download service should inspire some skepticism. Finding out who actually offers an image on the Docker Hub is important, because potentially anyone could upload it. Docker, Inc. [1] does not check uploads and typically leaves this responsibility to the user.

A good image usually contains a note on its build instructions – the docker file. Repository sites such as GitHub typically host these descriptions and let you download them. Thus, every user can reconstruct how an image was created. Of course, a review of this kind takes time, but it is worthwhile if the image in question will be playing a central role in your own infrastructure. Examples of this would be, for example, basic images for a Java application server or a preconfigured CMS in a container.

Official Images

The name of the image on Docker Hub consists of two components: The username and the actual name of the image, such as johndoe/mysql. If the first part is missing up to the forward slash, it is an official image. Official images play a very prominent role on Docker Hub, they are usually from the upstream projects themselves.

The official images help to ensure quality. Docker employees examine the related content and collaborate with upstream developers, security experts, and the community, according to company statements. Docker, Inc. defines some best practices for construction images with a view to security and other aspects [2]. This includes, for example, making sure that you only download the necessary Docker files via encrypted TLS or correctly validate Open PGP signatures.

Ultimately, Docker, Inc. accepts no liability whatsoever for the image. The projects themselves push their build instructions into a GitHub repository, from which Docker's service then assembles the images. It is very instructive to take a look at the help texts, scripts, and patches in the form of comments and pull requests hosted there. Currently, the figure is just over 100 official images, only a fraction compared to the tens of thousands of images to operator specs that Docker Hub offers.

But if you do not trust Docker Hub, official images or not, you need to look around for alternatives. Some providers of container platforms offer their own registries, Red Hat and Amazon, for example. Conceptually, this changes nothing in the threat model.

For production use in the enterprise, admins would do well to set up their own registries. This is something that you can do amazingly quickly, because there are also Docker images for this. The system administrator then only uploads in-house images or personally tested and verified images from other registries, to this registry. At the same time, you would exclusively allow the run-time environment access to this registry instead of the default.

Damaged in Transit

The second main threat can be quite easily contained with current Docker tools, because the communication between Docker and the client, the Docker host, and the registry is usually TLS encrypted and protected by SSL certificates without requiring too much work from the user.

The Docker Notary [3] project offers an approach for verifying end-to-end whether an image actually comes from the provider who the image claims created it. To describe this in a greatly simplified way, The Update Framework (TUF, [4]) on which the project is based computes a checksum on the manufacturer's side, and users then verify the image. The problem that this project faces is the immense complexity of some of the details, which makes it difficult to understand and thus difficult to use for many users.

Obsolete Software

If you manage computers, you will be aware that the flood of updates and fixes for bugs and vulnerabilities shows no sign of slowing down. Thus, it's an essential task to find out faster than attackers whether an image contains vulnerable libraries or executables.

This task proves to be all the more complex in a container environment because each container has its own set of libraries and tools. At the same time, patching is considered to be anti-pattern, because each instance actually needs to be ephemeral. You want to be able to use the newer instance of an image to update an older one at any time.

When you learn of a vulnerability, you need to build a new image based on the template on Docker Hub. But, how can you ensure that the template is not vulnerable? To do this, Docker, Inc. operates a supplementary service, known as Docker Security Scanning. It screens official images for paying customers, so far, only when specifically hired to do so, and more recently as part of the Advanced Enterprise Edition (see the "Docker's New Licensing Model" box).

Docker's New Licensing Model

Thus far, the Docker software has been released to the public under an open source license (Apache 2.0) in an irregular series of versions every few months. In early 2017, version 1.13 saw the 13th release appear since version 1.0 was published in June 2014.

In early March, Docker, Inc. – the company behind Docker – changed its licensing terms, release frequency, and its naming scheme. The Community Edition (CE) is similar to the previous versions. Monthly Edge releases will appear in the future, containing new features. Each quarter, the company will also publish stable releases. These are maintained for four months, by which time users will need to upgrade to the next release. Docker is also introducing a new versioning scheme with figures for the year and month, starting with Docker CE 17.03., which corresponds to Docker 1.13.1.

The commercial Enterprise Edition (EE) is also released four times a year, but it receives a full year's support. There are three versions of this edition: The Basic Edition is similar to the Community Edition to a large extent but is given access to certified plugins, for example, for overlay networks or storage drivers.

The Standard Edition of Docker EE contains what the provider has marketed as the Docker Datacenter. This includes your own registries, connections to directory services, and centralized management of credentials. The Standard Edition will cost $1,500 per year and node (virtual machine or dedicated server). The Advanced Edition also includes the Scanning Service, which will cost another $500 per year and node.

To allow this to happen, Docker operates a database of files with known security vulnerabilities. During the building on the Docker Hub, the image maker matches the files generated in any filesystem layer against this database. A web front end of the hub shows the results, and every visitor can view these – provided they log in with a free Docker ID.

The Docker Hub website [5] offers a search box in which users can enter the container name (e.g., mongo for the NoSQL database). If the attribute official appears in the results list, you can click on the image name and select the Tags tab. Now, a color-coded map of the image's security status (Figure 1) appears for each version of the image. Critical errors are marked red, major vulnerabilities in orange, and minor vulnerabilities yellow.

Figure 1: Users who log onto Docker Hub with their Docker ID to search for official images see a little more than anonymous users: The results of the security scan appear for each version.

Clicking on the Tag name shows more details, such as the name of the affected package, a classification, and the CVE number that provides more details of the discovered vulnerability (Figure 2).

Figure 2: In each image layer, the Docker Security Scanning service uses static analysis to check whether it can find a file with a known vulnerability.

Independent software vendors can also offer certified containers in the new Docker Store, which Docker continuously scans for security weaknesses.

Sobering

The results of the scans are sobering. An analysis I performed some time ago revealed that just seven percent of the 114 official images at the time were completely free of warnings. The worst offender had no less than 69 safety warnings. Current sampling does not substantially change these findings. Either manufacturers are not taking the tests seriously, and not focusing on the messages quickly enough, or the scanner is so sensitive that it returns too many false positives. Both presumptions do little to justify the user's trust in the production maturity of the service.

Other manufacturers offer a very similar approach. At CoreOS, the security scanner goes by the name of Clair [6]; it scans local and remote container images, including Docker images. In Red Hat's Project Atomic has an Atomic scan, which uses a plugin system [7] and the OpenSCAP local policy scanner [8] for the scan.

Security Included

No single tool automatically secures container environments, not even if you wave money around to entice it. The recommended approach is to see what your own environment offers in terms of specific tools for enabling the described mechanisms.

Before starting an instance of the image, administrators should check – manually or preferably in an automated process – whether it comes with known vulnerabilities, and only continue if these are patched. Unproblematic containers today can easily fall foul of effective attacks tomorrow. It helps to keep track of the installed components, scan your own system regularly (see box "There are Scanners, and Then There Are Scanners"), and use a container management system that supports your efforts to build and deploy new images quickly in an automated workflow.

There Are Scanners, and Then There Are Scanners

Several methods are available for determining whether the software installed on a computer contains vulnerabilities, and they differ significantly. Some manufacturers store information on CVE numbers in the software management package database, CVE being a vendor-independent description of security vulnerabilities, maintained by the MITRE Corporation. Enterprise versions from Red Hat and CentOS contain such information.

Tools more reminiscent of scanners in terms of functionality specifically look for files that are known to contain a vulnerability. If the scanner finds such a file, it points out the risks involved. However, there is no guarantee that the vulnerability is actually active.

To implement this procedure, this kind of software creates hashes of all the files on a computer and compares them with a centrally maintained database of vulnerable files. These can basically be executables or libraries. The list also contains signatures for known trojans or viruses.

The principle is thus closely related to primitive virus scanners and suffers from the same flaw: If the signature proffered by a malicious file differs in one byte from the signature in the list, this method will fail. Despite this, it still has some success in discovering the versions of all OpenSSH libraries affected by the Heartbleed vulnerability and distributed by popular Linux distributions. The effectiveness of this approach depends on very good and continuous maintenance of what is often referred to as the "known bad" list. Docker follows this path with its Docker Security Scanning offering. Where the lists came from, and how Docker maintains them, is something that the company does not reveal.

Network and system security scanners check whether dangerous services or configurations exist. Originally launched as a port mapper, Nmap is one of the most prominent examples. With its Nmap Scripting Engine (NSEs), admins can use Lua to write small programs that check for attack patterns on a case-by-case basis [9].

The proprietary tool Nessus [10] as well as OpenVAS [11], which was forked from it and is still available under a free license, explicitly seek for known and vulnerabilities accessible via the network. Both internally rely on Nmap for some tasks.

Local tools like Lynis [12] or the OpenSCAP policy test suite check a host for configuration vulnerabilities.

Good Sources, Quick Scans

Many system administrators retrieve software from a variety of sources. It is always advisable to carefully consider the providers of these sources – this also applies to classic software without containers. In this regard, Docker and others are no less or more secure than traditional installations.

If you provide images for Docker, but also offer rkt or Project Atomic, you should disclose how you handle updates and known vulnerabilities. This is the only way to ensure permanently secure operation of such software. Whether a commercial product can be linked with higher quality has been the subject of much discussion since the early days of open source, because some promises relating to security quality can genuinely be purchased with commercial applications.

The different static security scanning offerings usually test signatures. The results of the current implementations are mediocre and contain many false positives. The question arises as to whether the additional overhead of reworking actually justifies the amount of money you pay.