Maintenance Jobs

Distr includes built-in CLI tasks for database and storage pruning to prevent unbounded growth of resources such as metrics, logs and status entries.

The cleanup routine can be executed manually. One or more targets can be specified in a single invocation.

distr cleanup $CLEANUP_TARGET...

Available cleanup targets:

For production deployments we recommend scheduling these jobs automatically, either using the built-in job scheduler for single instance deployments or using Kubernetes CronJobs for high-availability deployments.

If you only have one instance of Distr running (e.g., using Docker Compose), you can use the integrated job scheduling.

The internal scheduling can be configured via environment variables.

An example configuration file can be found on github.com/distr-sh/distr/deploy/docker/quickstart:

# Cron interval for cleaning metrics older than METRICS_ENTRIES_MAX_AGE
CLEANUP_DEPLOYMENT_TARGET_METRICS_CRON="0 * * * *"
CLEANUP_DEPLOYMENT_TARGET_METRICS_TIMEOUT="10m"
# Cron interval for cleaning expired OIDC state entries
CLEANUP_OIDC_STATE_CRON="0 * * * *"
CLEANUP_OIDC_STATE_CRON_TIMEOUT="10m"
# Cron interval for cleaning unreferenced artifact blobs from S3 (requires registry)
CLEANUP_ARTIFACT_BLOB_CRON="0 * * * *"
CLEANUP_ARTIFACT_BLOB_TIMEOUT="10m"
CLEANUP_ARTIFACT_BLOB_MIN_AGE="24h"
# Cron interval for permanently deleting soft-deleted organizations older than CLEANUP_ORGANIZATION_MIN_AGE
# CLEANUP_ORGANIZATION_CRON="0 0 * * *"
# CLEANUP_ORGANIZATION_TIMEOUT="10m"
# CLEANUP_ORGANIZATION_MIN_AGE="720h"
# Cron interval for permanently deleting user accounts without an organization older than CLEANUP_USER_ACCOUNT_MIN_AGE
# CLEANUP_USER_ACCOUNT_CRON="0 0 * * *"
# CLEANUP_USER_ACCOUNT_TIMEOUT="10m"
# CLEANUP_USER_ACCOUNT_MIN_AGE="720h"
# Cron interval for deleting unreferenced files older than CLEANUP_FILE_MIN_AGE
CLEANUP_FILE_CRON="0 * * * *"
CLEANUP_FILE_TIMEOUT="10m"
CLEANUP_FILE_MIN_AGE="24h"

If these variables are not set, no cleanup jobs are scheduled.

For high-availability deployments with multiple instances of Distr (e.g., using Kubernetes), the built-in job scheduling is not suitable, as it would lead to multiple instances trying to perform the same cleanup tasks concurrently.

Therefore, we recommend using CronJobs in Kubernetes to handle the cleanup tasks. But the concept can apply to any HA setup where the cleanup jobs are triggered externally.

These jobs can also be configured via our Helm Chart.

An example configuration can be found in github.com/distr-sh/distr/deploy/charts/distr:

cronJobs:
  - name: deployment-target-metrics-cleanup
    labels:
      distr.sh/job: deployment-target-metrics-cleanup
    args: [cleanup, DeploymentTargetMetrics, --timeout, 10m]
  - name: oidcstate-cleanup
    labels:
      distr.sh/job: oidcstate-cleanup
    args: [cleanup, OIDCState, --timeout, 10m]
  - name: artifact-blob-cleanup
    labels:
      distr.sh/job: artifact-blob-cleanup
    args: [cleanup, ArtifactBlob, --timeout, 10m]
  - name: file-cleanup
    labels:
      distr.sh/job: file-cleanup
    args: [cleanup, File, --timeout, 10m]
  # - name: organization-cleanup
  #   labels:
  #     distr.sh/job: organization-cleanup
  #   args: [cleanup, Organization, --timeout, 10m]
  # - name: user-account-cleanup
  #   labels:
  #     distr.sh/job: user-account-cleanup
  #   args: [cleanup, UserAccount, --timeout, 10m]

Some cleanup targets allow for additional customization using the following environment variables:

METRICS_ENTRIES_MAX_AGE="24h"
CLEANUP_ARTIFACT_BLOB_MIN_AGE="24h"
CLEANUP_ORGANIZATION_MIN_AGE="720h"
CLEANUP_USER_ACCOUNT_MIN_AGE="720h"
CLEANUP_FILE_MIN_AGE="24h"

When the pull-through cache is in use, Distr can automatically sync tags from upstream registries on a schedule. This is configured via a single environment variable:

# Cron schedule for syncing tags from upstream registries
REGISTRY_UPSTREAM_SYNC_CRON="15,45 * * * *"

If REGISTRY_UPSTREAM_SYNC_CRON is not set, no automatic sync is scheduled. Tags are still fetched on demand when a client pulls an image, but the tag list will not be proactively refreshed.

When custom domains are enabled, a job checks that each registered domain still points at your CNAME target. A domain is only used for links, mails, controller manifests and registry URLs while that check succeeds.

distr maintenance verify-custom-domains --timeout 10m

For single instance deployments, schedule it with the built-in scheduler:

CUSTOM_DOMAIN_VERIFICATION_CRON="*/15 * * * *"
CUSTOM_DOMAIN_VERIFICATION_TIMEOUT="4m"

For high-availability deployments, use a CronJob instead, as our Helm Chart does:

cronJobs:
  - name: custom-domain-verification
    labels:
      distr.sh/job: custom-domain-verification
    schedule: '*/15 * * * *'
    args: [maintenance, verify-custom-domains, --timeout, 10m]

Distr encrypts the sensitive columns of its database. Values written before you upgraded to a version that does so are still in plaintext, and after a key rotation some values are still encrypted with the previous key. This command moves both to the active key:

distr maintenance encrypt-database --timeout 60m

Unlike the jobs above this is not recurring work, so do not schedule it. There are exactly two occasions on which it has anything to do, the upgrade to a version that encrypts and a rotation of DATABASE_ENCRYPTION_KEY, and both are things you do deliberately. Run it once afterwards. Nothing in Distr writes a plaintext value, so once it reports that there is nothing left, that stays true.

It is safe to run while Distr is serving traffic, it can be interrupted and resumed, and it is a no-op once everything is on the active key. Until then, Distr logs a warning on every start naming the columns that are left.

On Kubernetes, run it in a Distr pod, which already has the database URL and the encryption key in its environment:

kubectl exec deploy/distr -- /distr maintenance encrypt-database --timeout 60m

The command dies with the pod if it is replaced while running, and a long run can outlive the kubectl exec connection. Neither loses work: every batch is committed on its own, so run it again and it continues where it stopped.

For single instance deployments you can instead have it run during startup:

DATABASE_ENCRYPTION_MIGRATE_ON_BOOT=true

Leave this off if you run more than one replica, for the reason above: the first replica of the new version would start encrypting while replicas of the previous one are still serving.

Prior to v4.2.0, every status a controller reported was appended to a DeploymentRevisionStatus table, which the portal showed as a status history per deployment. Since controllers report once per CONTROLLER_INTERVAL that table grew continuously, and the DeploymentRevisionStatus cleanup target pruned entries older than STATUS_ENTRIES_MAX_AGE, always keeping the newest one per deployment.

Distr now stores only the latest status, on the deployment revision itself, so there is nothing left to prune. The cleanup target, its cron environment variables (CLEANUP_DEPLOYMENT_REVISION_STATUS_CRON, CLEANUP_DEPLOYMENT_REVISION_STATUS_TIMEOUT) and STATUS_ENTRIES_MAX_AGE have been removed, along with the GET /api/v1/deployments/{id}/status and GET /api/v1/deployments/{id}/status/export endpoints. Remove the deployment-revision-status-cleanup CronJob from your Helm values and the variables from your environment; Distr ignores them.

What a controller reports is now written to the controller logs instead, which you reach from the menu of a deployment target in the portal. Their retention is the Loki retention described below.

Prior to v3.0.0, deployment and deployment target logs were stored in PostgreSQL and pruned by the DeploymentLogRecord and DeploymentTargetLogRecord cleanup targets, keeping the newest LOG_RECORD_ENTRIES_MAX_COUNT entries per deployment resource or deployment target (count-based retention).

Logs are now stored in Grafana Loki and retention is time-based, managed entirely by the Loki configuration (the shipped default is 30 days via limits_config.retention_period). The cleanup targets, their cron environment variables (CLEANUP_DEPLOYMENT_LOG_RECORD_*, CLEANUP_DEPLOYMENT_TARGET_LOG_RECORD_*), and LOG_RECORD_ENTRIES_MAX_COUNT have been removed.

Prior to v2.20.0, Distr tracked a separate DeploymentTargetStatus per deployment target. This concept has been removed and is no longer available as a cleanup target.