Vulnerability scanning inside and outside the container

With the wide acceptance of the concept of containerized applications due to the benefits they bring, one should not overlook the security in container environments. Docker containers (and possibly others) by default come with several Linux security protections such as namespaces, cgroups, seccomp profiles, limited capabilities etc. So, the container environments may look secure by default but these seemingly harmless containers can have vulnerabilities and misconfigurations that are not covered by these default protections.

For instance, vulnerable code deployed as part of the container can provide a pathway to an attacker to land in our infrastructure. This article provides an overview of some of the vulnerability scanning options (various components that should be scanned) one should consider when using containers.

Learn Container Security

Build your skills around Docker and Kubernetes security, including key technologies, creating and running a secure cluster, and more.

Start Learning

Inside the container

Let us begin by discussing some of the vulnerability scanning options and areas to be considered inside a container. 

Traditional vulnerability scanning

Docker containers are instances of Docker images and these images are built from configuration files known as Dockerfiles. A docker image consists of several dependencies that are required by the application being run in the container. In addition to it, it also includes a base image such as Ubuntu, CentOS or Alpine. All of these components can be prone to vulnerabilities especially when the component versions are outdated with known vulnerabilities.

Such vulnerabilities can lead to privilege escalation and/or container escape. It is recommended to perform regular security audits within the containers to identify such security issues. Lynis is one such security auditing tool for UNIX derivatives like Linux, macOS, BSD, Solaris, AIX, and others that can perform security scans. 

Container security audits

Regular container security audits should be performed to ensure that containers are being run with all the best practices. These audits on the running containers can be performed from outside of the container. Let us see some examples of how this can be done.

Checking for readonly filesystem:

The following excerpt shows how we can check if a running container is started with a readonly filesystem using docker inspect command.

8ab9dee1a9dfa66c933566d9b7b3778ba101d77d5b34a1c09167c3597972dc8c

$ docker inspect --format '{{ .HostConfig.ReadonlyRootfs }}' 8ab9dee1a9d

false

As we can observe in the preceding excerpt, we are reading the container property {{ .HostConfig.ReadonlyRootfs}} by providing a container identifier to check if the container is running with a readonly filesystem. The response shows that the container is not started with a readonly filesystem.

Checking for privileged containers:

The following excerpt shows how we can check if a running container is started with --privileged flag.  

f907068aaef77b0cd6162f71296df6b7333e5f3ce74238b93a11d02f3c31a28c

$ docker inspect --format '{{ .HostConfig.Privileged }}' f907068aaef7

false

As we can observe in the preceding excerpt, we are reading the container property {{ .HostConfig.Privileged}} by providing a container identifier to check if the container is running as a privileged container. The response shows that the container is not started with a --privileged flag. Running a container with --privileged flag will override all protections docker enforces on containers and such a compromised container can allow an attacker to escape the container and gain access to the underlying host.

As a general best practice, we should consider running scanners to check for all these missing best practices on running containers. CIS Benchmarks can be used as general guideline to start with.

Docker image scanning

Containers are instances of images built from configuration files. A running container can have vulnerabilities originating from an insecure component built into the image. To detect such issues, it is recommended to scan docker images for security vulnerabilities. While this is not an activity done inside the container, it will help secure the environment inside a container. Docker image scanning can be done in different places.

1. Standalone process on the client system where images are built and tested.
2. A build server used in CI/CD pipelines
3. Image registry as and when an image is uploaded.

Trivy is an open source option to perform docker image scanning. 

Outside the container

Let us now discuss some of the vulnerability scanning options we should consider outside the containers.

Traditional vulnerability scanning

Once docker images are built with security in mind and containers are spun up with appropriate hardening, another step in ensuring that the work loads running in the containers are safe is to perform live scanning on the running containers. This can uncover vulnerabilities that weren’t found in early stages such as when writing application code.

In addition to it, monitoring the container traffic can also detect attacks attempted against the application running as well as the container itself. Application layer vulnerabilities such as SQL Injection, Cross Site Scripting, Remote Command Execution may be identified by using traditional web vulnerability scanners. The following figure shows automated scan results from OWASP ZAP run against an application running in a container.

Similarly, scanning container logs and analysing container network traffic can also give visibility into various attacks attempted. The following log entry from the container shows that a path traversal attack was attempted on the application running in the container.

Even though the example discussed here was shown using a traditional web vulnerability scanner run from a different machine in the same network as the host, where the container is running, there are other approaches that can be used to perform vulnerability scanning. Some of the examples include:

• Embedding an agent into the container that can perform the scans
• Side-car containers for log analysis
• Scanning by running the scanner as another container on the same host
• Scanning from the docker host (the host, where Docker engine is running)
• Scanning from a host in the same network as the docker host

Depending on certain conditions and choices, some of these options may or may not be feasible to implement in some environments.

If these containers are built and run in CI/CD environments, security scans (DAST) can be performed immediately after a container is started right on the CI server. This will help detect the vulnerabilities even before the containers hosting the applications are in production. So, running scans from CI server is another option.

In addition to performing vulnerability scans on the exposed applications, we should also consider network scans from the local network segment of the docker host, internal LAN as well as external network where services are exposed.

Scan the code

Docker containers are meant to run applications. If the application code has vulnerabilities, it can expose the container even if the container environment is well hardened. These vulnerabilities can often be found by performing source code reviews. Automated scanners can also be used to identify vulnerabilities in source code.

Security hygiene of the host 

The host where containers are running should be well hardened. We can reduce the attack surface by limiting the exposure of services running on the host. One example would be restricting access to Docker Remote API to a limited number of hosts that require this access. It is also a good idea to use SELinux or AppArmor on host OS. Another area of concern on Docker hosts is who can run docker commands. The ability to run Docker commands can let someone get root privileges. Needless to mention all the services on the docker host including docker runtime should be updated. Running regular scans using CIS benchmarks for docker as well as host OS would be a good starting point to find the gaps.

Isolate container networks

When multiple containers are created on a host, all of them will be in the same network. Let us assume that there are multiple applications deployed as containers on the same host. Even though these application containers do not require communication between each other, they all will be in the same network. This lack of network isolation can be risky especially when one of the applications is compromised. It is recommended to use per application network. 

Images being pulled

Images are pulled from image repositories to spin up containers. Leaving old images without updating them can lead to unknown vulnerabilities in the images as new vulnerabilities are regularly found. So, it is recommended to perform updates every now and then. Also check for updates in the upstream repositories. For instance, we pulled python3.9.1-alpine when building our image. But, python3.9.2-alpine is the latest image now.  It is also recommended to implement access controls on who can push images to the private repositories.

Learn Container Security

Build your skills around Docker and Kubernetes security, including key technologies, creating and running a secure cluster, and more.

Start Learning

Conclusion

This article has provided an overview of some of the vulnerability scanning options and various components that should be scanned when using containers.  

Sources

https://offensivekubernetes.com/dockerizing-php-flask-web-application/

https://www.stackrox.com/post/2020/05/kubernetes-security-101/

https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/