Deployment Agents

Deployment Agents are a core component of the Distr platform, designed to run within customer environments alongside vendor applications. These lightweight agents bridge the gap between the Distr Hub and deployment targets, enabling comprehensive lifecycle management while requiring minimal access to customer infrastructure.

Note

Distr deployment agents are the key components that enable the Assisted Self-Managed Customer and BYOC use cases.

Distr offers specialized agents for different deployment tools, each designed to handle specific deployment scenarios while maintaining consistent core functionality.

Docker Agent

The Docker Compose agent is specifically designed for container-based deployments, providing comprehensive management of both single- and multi-container applications. It handles application updates with minimal downtime, collects container health metrics and logs, and supports environment variable templating for flexible configuration.

Note

Unless you’re using Distr to deploy your own applications internally, the Distr agent will always be configured and deployed by the end customer. Head over to the end-customer agent guide to learn how it should be installed.

In the following, we want to dive deep into how the Docker agent works.

Requirements

The host system must have Docker and Docker Compose installed.

Installation and Environment

The installation of the agent is done by the end customer (or the vendor, if it should be vendor-managed) by fetching a Docker Compose file and starting it with docker compose up. This is why the installation command looks like this: curl "https://<distr-hub>/api/v1/connect?targetId=<targetId>&targetSecret=<targetSecret>" | docker compose -f - up -d, where <distr-hub> is the hostname of the Distr Hub, <targetId> is the ID of the deployment target, and <targetSecret> is the secret of the deployment target.

The authentication via targetId and targetSecret ensures that the agent can only fetch the deployments of one specific deployment target. Although the secret is generated by the Distr Hub, it is only shown once to the user. The Distr Hub does not store the secret but can verify it.

Let’s have a closer look at the Docker Compose file for running the agent:

name: distr
services:
  agent:
    network_mode: host
    restart: unless-stopped
    image: 'ghcr.io/glasskube/distr/docker-agent'
    environment:
      # Authentication with Distr Hub, see above
      DISTR_TARGET_ID: '<targetId>'
      DISTR_TARGET_SECRET: '<targetSecret>'
      # various endpoints of the Distr Hub:
      DISTR_LOGIN_ENDPOINT: 'https://<distr-hub>/api/v1/agent/login'
      DISTR_MANIFEST_ENDPOINT: 'https://<distr-hub>/api/v1/agent/manifest'
      DISTR_RESOURCE_ENDPOINT: 'https://<distr-hub>/api/v1/agent/resources'
      DISTR_STATUS_ENDPOINT: 'https://<distr-hub>/api/v1/agent/status'
      DISTR_METRICS_ENDPOINT: 'https://<distr-hub>/api/v1/agent/metrics'
      # how often the agent fetches deployments
      DISTR_INTERVAL: '5s'
      # agent versioning information, needed for agent self-updates:
      DISTR_AGENT_VERSION_ID: '<db-id-of-agent-version>'
      DISTR_AGENT_SCRATCH_DIR: /scratch
      DISTR_REGISTRY_HOST: 'localhost:8585'
      HOST_DOCKER_CONFIG_DIR: ${HOST_DOCKER_CONFIG_DIR-${HOME}/.docker}
    volumes:
      # the host's docker socket is mounted into the agent container
      - /var/run/docker.sock:/var/run/docker.sock
      - scratch:/scratch
      - ${HOST_DOCKER_CONFIG_DIR-${HOME}/.docker}:/root/.docker:ro
volumes:
  scratch:

Core Logic

The agent executes the following core logic at the interval defined by the DISTR_INTERVAL (default 5 seconds) environment variable:

1. Fetch the deployments from DISTR_RESOURCE_ENDPOINT
2. Delete Docker Compose projects that are no longer in the fetched list of deployments (i.e., have been undeployed), with docker compose down -v
3. Start Docker Compose projects that are in the fetched list of deployments, with docker compose up -d. The environment variables of the deployment are also passed to this command. As this could take a while when images have to be pulled, the agent will send the status PROGRESSING to the backend, to indicate that work is still in progress.
4. Send each Docker Compose output to the Distr Hub via DISTR_STATUS_ENDPOINT, thereby reporting the deployment’s status.

All requests to the Distr Hub are authenticated with a JWT token, which is obtained by the agent by calling the DISTR_LOGIN_ENDPOINT with the DISTR_TARGET_ID and DISTR_TARGET_SECRET credentials.

The Docker Compose project names of the deployments have the structure distr-<short-deployment-id>, so one deployment in Distr matches one Docker Compose project on the host.

To be able to use the docker binary of the host system, the agent needs to have the Docker socket mounted into the container.

Authentication with OCI registries

If a deployment uses images from the Distr registry, the agent automatically authenticates with the registry, as it is already authenticated with the Distr Hub anyway. That way, no additional steps are required for the customer. This authentication secret is stored inside a temporary directory of the Docker container.

For images from any other authenticated OCI registry, the agent mounts the Docker config directory of the host system into the container, such that authentication secrets can be reused. This means that any registries that are accessible from the host system will also be accessible to the agent container.

Another option is to provide a username and a personal access token via an application license (PRO Feature). If a deployment is then created with this license, the agent will log in to the registry given in the license using the provided credentials.

Agent Self Updates

If the agent version of a deployment target is updated in the Distr Hub, the agent container will update itself in the next cycle. Since with a version update, environment variables can also change, the agent will fetch the manifest from DISTR_MANIFEST_ENDPOINT and then restart itself. For that to work, the DISTR_AGENT_SCRATCH_DIR and scratch volume mount are needed.

Docker Metrics

Metrics are optional but enabled by default. They can be switched on or off via the Distr Hub when editing the deployment target. The agent scrapes and reports the metrics to the DISTR_METRICS_ENDPOINT every 30 seconds.

The reported metrics are:

• number of CPU cores of the host system
• current overall CPU usage in percent
• amount of memory of the host system
• current overall memory usage in percent

Docker Swarm

A deployment can be run in swarm mode. For that to work, the host running the Distr Docker agent must have Docker Swarm enabled and initialized. Instead of docker compose up, the agent will use docker stack deploy to deploy the deployment. Unlike regular compose mode, docker stack deploy is not called in every cycle, but only whenever the deployment has changed.

Helm Agent

The Helm agent specializes in Kubernetes deployments, managing the complete lifecycle of Helm charts in customer environments. It handles chart installation and upgrades, monitors Kubernetes resource health, supports customer-specific value overrides, and provides rollback capabilities for failed deployments.

In the following, we want to dive deep into how the Helm agent works.

Installation and Environment

The installation of the agent is done by the end customer (or the vendor, if it should be vendor-managed) by executing a command looking like this: kubectl apply -n <namespace> -f "https://<distr-hub>/api/v1/connect?targetId=<targetId>&targetSecret=<targetSecret>", where <distr-hub> is the hostname of the Distr Hub, <targetId> is the ID of the deployment target, and <targetSecret> is the secret of the deployment target. This fetches the kubernetes manifests from the Distr Hub and applies it in the selected namespace. That namespace must exist already.

The authentication via targetId and targetSecret ensures that the agent can only fetch the deployments of one specific deployment target. Although the secret is generated by the Distr Hub, it is only shown once to the user. The Distr Hub does not store the secret but can verify it.

The fetched manifests most prominently contain the distr-agent deployment, which follows the following structure:

Danger

Scaling the same agent to more than one replica will lead to an undefined behavior and is not recommended. The deployment manifests should not be altered.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: distr-agent
spec:
  selector:
    matchLabels:
      app: distr-agent
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: distr-agent
    spec:
      serviceAccountName: distr-agent
      securityContext:
        runAsNonRoot: true
      containers:
        - name: distr-agent
          image: 'ghcr.io/glasskube/distr/kubernetes-agent'
          imagePullPolicy: IfNotPresent
          env:
            - name: DOCKER_CONFIG
              value: /opt/config/.docker/
            - name: DISTR_AGENT_CONFIG_DIRS
              value: |
                /opt/config/.agent/env
                /opt/config/.agent/auth
          envFrom:
            - secretRef:
                name: distr-agent-auth
            - configMapRef:
                name: distr-agent-env
          volumeMounts:
            - name: dockerconfig
              mountPath: /opt/config/.docker/
            - name: env
              mountPath: /opt/config/.agent/env
            - name: auth
              mountPath: /opt/config/.agent/auth
            - name: cache
              mountPath: /.cache/
      volumes:
        - name: env
          configMap:
            name: distr-agent-env
        - name: auth
          secret:
            secretName: distr-agent-auth
        - name: cache
          emptyDir: {}

The agent can be either cluster-scoped or namespace-scoped. This can not be changed after creating the agent via the Distr Hub. Based on that, the distr-agent service account will either have a ClusterRoleBinding to the built-in cluster-admin or only a regular RoleBinding to a new namespace-admin role, that permits to do everything in this namespace.

The distr-agent-env config map contains all the environment variables needed for the agent to run. They are the same as in the docker agent.

The distr-agent-auth secret contains the agents targetId and targetSecret, that are also used the same as in the docker agent.

Core Logic

The agent executes the following core logic at the interval defined by the DISTR_INTERVAL (default 5 seconds) environment variable:

1. Fetch the deployments from DISTR_RESOURCE_ENDPOINT
2. Uninstall the helm releases that are no longer in the fetched list of deployments (i.e., have been undeployed)
3. Install or upgrade helm releases that are in the fetched list of deployments. As this could take a while when images have to be pulled, the agent will send the status PROGRESSING to the backend, to indicate that work is still in progress.
4. Send the helm output to the Distr Hub via DISTR_STATUS_ENDPOINT, thereby reporting the deployment’s status.

All requests to the Distr Hub are authenticated with a JWT token, which is obtained by the agent by calling the DISTR_LOGIN_ENDPOINT with the DISTR_TARGET_ID and DISTR_TARGET_SECRET credentials.

Altering the Helm Release

The Distr Helm agent uses the Helm API to perform Helm releases in the Kubernetes cluster similar to how the Flux Helm Controller works.

This allows you to also inspect the status of the Helm release with helm ls -a -n <namespace>.

Danger

Manually altering the Helm release with helm upgrade or helm rollback will stop the Distr agent from performing any future changes. It is currently not easily possible to “resync” the revision id to let the Distr agent pick up the release again.

Authentication with OCI registries

If a deployment uses images from the Distr registry, the agent automatically authenticates with the registry, as it is already authenticated with the Distr Hub anyway. That way, no additional steps are required for the customer. This authentication secret is stored inside a temporary directory of the Docker container.

To reference to images of your Distr registry in your helm chart, please check: Configuring a Helm Chart to Support Distr Artifacts.

Another option is to provide a username and a personal access token via an application license (PRO Feature). If a deployment is then created with this license, the agent will log in to the registry given in the license using the provided credentials.

Agent Self Updates

If the agent version of a deployment target is updated in the Distr Hub, the agent container will update itself in the next cycle. Since with a version update, environment variables can also change, the agent will fetch the manifest from DISTR_MANIFEST_ENDPOINT and then restart itself.

Kubernetes Metrics

Metrics are optional but enabled by default for cluster-scoped agents. Metrics can be switched on or off via the Distr Hub when editing the deployment target. The agent scrapes and reports the metrics to the DISTR_METRICS_ENDPOINT every 30 seconds.

The reported metrics are:

• number of CPU cores of all nodes of the cluster
• current overall CPU usage in percent
• amount of memory of all nodes of the cluster
• current overall memory usage in percent

Requirements

• metrics-server needs to be installed in the cluster
• The agent must be cluster-scoped, since it needs access to the node metrics resources.

Terraform Agent (Coming Soon)

Currently in development

The Terraform agent is designed for Bring Your Own Cloud (BYOC) deployments and is currently in development.

Configuration Steps

Follow the configuration steps in the Deployment Agent Quickstart Guide.

Agent Section in the Vendor Portal

The Deployment Agents section of Distr consists of two main components that work together to provide a complete deployment management solution.

Applications

The Applications section provides vendors with a dedicated space to configure and manage their software offerings. Using either Docker Compose files or Helm charts, vendors can define their application templates, specify which environment variables are configurable by customers, and manage different versions of their applications. This section serves as the foundation for all subsequent deployment activities.

For detailed instructions on adding applications to Distr, see our platform management guides:

• Onboarding a Docker Application
• Onboarding a Helm Application

Deployments

Deployments Overview

The Deployments section acts as the central management and health dashboard for all deployments.

Use this section to:

• Monitor the real-time health and status of all deployments
• View deployment metadata and logs
• Track deployment states across all customer environments

Deployment Health

The Deployments page shows status information at both deployment target and deployment level.

The Distr agent, which is deployed once per deployment target, reconciles the state of the deployment target every 5 seconds, thereby also reporting the health of the deployment target and the contained deployments.

Deployment Target Status List

• ⚫ Not connected: the dot to the left of the deployment target name is grey.
• 🟢 Connected: the dot is green.
• 🟡 Stale, i.e., the agent has been connected before but did not report its health in the last 60 seconds: the dot is yellow. Only in this case, you can reconnect the agent by creating new credentials for the agent. Please note that this will invalidate the previous credentials.

Deployment Status List

• ⚫ No status: The agent has not reported anything regarding the deployment yet.
• 🔴 Error: There is an error regarding the deployment. Consider investigating the errors via the Status logs.
• 🟡 Stale: The agent has not reported the deployment health in the last 60 seconds.
• 🔵 Progressing: The agent is currently installing or upgrading the deployment, due to your previous request.
• 🟢 OK: The deployment is healthy and running.

Agent Capabilities

All Distr agents share a common set of core capabilities designed to ensure reliable and secure deployments. These include guided installation processes, automated application updates, comprehensive health monitoring, log aggregation for troubleshooting, and secure, outbound-only communication with the Distr Hub.

Note

We are also working on a read-only version of the Distr agent that will not be able to alter a deployment, but will just monitor the deployment status and report logs and metrics.