Docker Controller Security

The Docker controller manages Docker on the host, so it has the access of the Docker engine itself.

The controller opens every connection to Distr itself, over HTTPS, so the host needs no inbound ports.

The Compose file that installs the controller runs two containers as root, the controller and the autoheal service. Autoheal only mounts the Docker socket, the controller mounts:

A deployment target editor is a user with the read_write or admin role in the vendor organization or in the customer organization the deployment target belongs to. The API only enables metrics when a request sets metricsEnabled.

The controller runs in the host’s network namespace. The autoheal container has no network and only talks to the Docker socket.

Distr renders the Compose file when the controller connects and again for every self-update. A change to the deployment target, such as enabling metrics, reaches the host with the next controller update, and local edits to the Compose file are overwritten then.

The controller talks to the Docker engine only through the mounted socket. It runs Compose operations through the Compose library built into the controller, and the self-update through the Docker CLI of the controller image. The commands below are what the controller runs, written as the equivalent Docker CLI calls, with these placeholders:

Distr sets <project> as the name of the Compose file it sends. The controller writes that file to <dir>/docker-compose.yaml and, when the deployment has an env file, that file to <dir>/.env. Both are readable by root only. A deployment without an env file has no .env: the controller deletes one left by an earlier revision, because Compose reads a .env in the project directory on its own.

The controller applies a deployment when Distr sends a revision it has not applied yet, and again on every cycle while the last apply failed or did not finish. A running deployment is not applied again.

First it logs in to the Distr registry with its current login token and to the registries of the deployment’s application entitlements with their credentials, the equivalent of docker login. It stores these credentials in <docker-config> and uses that directory instead of the host’s Docker config for every command of the deployment that needs one of these logins.

It then runs:

docker compose --project-directory <dir> --file <dir>/docker-compose.yaml \
  up --detach --remove-orphans --no-build

This creates the networks and volumes of the project, pulls the images that are missing or whose pull_policy asks for it, creates and starts the containers, runs their post_start hooks and removes the containers of services the project no longer defines. Images are never built on the host. An apply that has not finished after 10 minutes is canceled and reported as an error.

When a revision is created with Force restart, the controller restarts the deployment after a successful apply:

docker compose --project-directory <dir> --file <dir>/docker-compose.yaml restart

With image cleanup enabled, the controller collects the images of all containers of the previous revision with docker compose --project-name <project> ps --all before an apply and, once the apply succeeded, removes each of them:

docker image rm <image>

It does not force the removal, so an image the new revision or another container still uses stays on the host.

When a deployment is no longer part of the deployment target, the controller removes it:

docker compose --project-directory <dir> --file <dir>/docker-compose.yaml down --volumes --remove-orphans
docker compose --project-name <project> down --volumes --remove-orphans
docker image rm <image>

The first down loads the project so that Compose runs the pre_stop hooks of its services. The second one removes whatever is left with the label of the project, including volumes and networks a previous revision declared. The images are those of the deployment’s containers and are removed whether or not image cleanup is enabled. Finally the controller deletes <dir>.

When the controller version of the deployment target changes, the controller fetches its new Compose file from Distr, adds its target secret to it and writes it to /scratch/distr-update.yaml. It removes a stopped container named distr-controller-update left over from an earlier update, and then starts the update from a container of the new image:

docker run --detach --rm --name distr-controller-update \
  --entrypoint /usr/local/bin/docker-entrypoint.sh \
  --env HOST_DOCKER_CONFIG_DIR=<host-docker-config-dir> \
  --env DOCKER_HOST=unix:///var/run/docker.sock \
  --volumes-from <controller-container-id> \
  <controller-image>:<new-version> \
  docker compose -f /scratch/distr-update.yaml up -d

--volumes-from gives the update container the mounts of the controller, including the Docker socket. The update recreates the controller and autoheal containers of the distr project and removes itself when it is done.

These calls only read. They run for the applied revision of every deployment:

With metrics enabled the controller also reads the usage of its own containers through docker ps --filter label=com.docker.compose.project=distr, docker stats --no-stream and docker inspect, and the size of the image store through the equivalent of docker system df every 5 minutes. It reads the CPU and memory usage of the host from /proc, the mounted devices from /hostfs/proc/mounts, the usage of each mounted /dev/ device and the size of the container log files that docker inspect names under /hostfs.

The autoheal container checks every 5 seconds for unhealthy containers and restarts them, through the Docker API, as the equivalent of:

docker ps --filter health=unhealthy --filter status=running --filter label=sh.distr.autoheal=true
docker restart --time 10 <container>

It skips containers labeled autoheal=False and takes the timeout from a container’s autoheal.stop.timeout label where one is set. With autoheal enabled for all containers, the label filter is left out, so it restarts every unhealthy container on the host. See Autoheal.

Anything else that runs comes from the Compose file of a deployment, such as its lifecycle hooks.

• The target secret is part of the controller’s environment, so anyone who can inspect the controller container can read it. During a self-update it is also written to /scratch/distr-update.yaml. Authentication explains how to replace it.
• The env files of the deployments, with their secrets filled in, are kept in <dir>/.env in the scratch volume, readable by root only.
• The registry credentials of every deployment, including the controller’s login token for the Distr registry, are kept in <docker-config>/config.json in the temporary directory of the controller container.