Maintenance Jobs

Database Cleanup Jobs

Distr includes built-in CLI tasks for database and storage pruning to prevent unbounded growth of resources such as metrics, logs, and status entries.

The cleanup routine can be executed manually. One or more targets can be specified in a single invocation.

distr cleanup $CLEANUP_TARGET...

Available cleanup targets:

Target	Description
DeploymentRevisionStatus	Deployment revision status entries
DeploymentTargetMetrics	Deployment target metrics entries
OIDCState	Expired OIDC state entries
ArtifactBlob	Unreferenced registry blobs from S3 (requires registry to be enabled)
Organization	Permanently delete soft-deleted organizations past the retention period

For production deployments we recommend scheduling these jobs automatically, either using the built-in job scheduler for single instance deployments or using Kubernetes CronJobs for high-availability deployments.

Automated job scheduling for Single Instance deployments

If you only have one instance of Distr running (e.g., using Docker Compose), you can use the integrated job scheduling.

The internal scheduling can be configured via environment variables.

An example configuration file can be found on github.com/distr-sh/distr/deploy/docker:

# Cron interval for cleaning deployment revision statuses older than STATUS_ENTRIES_MAX_AGE
CLEANUP_DEPLOYMENT_REVISION_STATUS_CRON="0 * * * *"
CLEANUP_DEPLOYMENT_REVISION_STATUS_TIMEOUT="10m"
# Cron interval for cleaning metrics older than METRICS_ENTRIES_MAX_AGE
CLEANUP_DEPLOYMENT_TARGET_METRICS_CRON="0 * * * *"
CLEANUP_DEPLOYMENT_TARGET_METRICS_TIMEOUT="10m"
# Cron interval for cleaning expired OIDC state entries
CLEANUP_OIDC_STATE_CRON="0 * * * *"
CLEANUP_OIDC_STATE_CRON_TIMEOUT="10m"
# Cron interval for cleaning unreferenced artifact blobs from S3 (requires registry)
CLEANUP_ARTIFACT_BLOB_CRON="0 * * * *"
CLEANUP_ARTIFACT_BLOB_TIMEOUT="10m"
CLEANUP_ARTIFACT_BLOB_MIN_AGE="24h"
# Cron interval for permanently deleting soft-deleted organizations older than CLEANUP_ORGANIZATION_MIN_AGE
# CLEANUP_ORGANIZATION_CRON="0 0 * * *"
# CLEANUP_ORGANIZATION_TIMEOUT="10m"
# CLEANUP_ORGANIZATION_MIN_AGE="720h"

If these variables are not set, no cleanup jobs are scheduled.

Automated job scheduling for High-Availability deployments

For high-availability deployments with multiple instances of Distr (e.g., using Kubernetes), the built-in job scheduling is not suitable, as it would lead to multiple instances trying to perform the same cleanup tasks concurrently.

Therefore, we recommend using CronJobs in Kubernetes to handle the cleanup tasks. But the concept can apply to any HA setup where the cleanup jobs are triggered externally.

These jobs can also be configured via our Helm Chart.

An example configuration can be found in github.com/distr-sh/distr/deploy/charts/distr:

cronJobs:
  - name: deployment-revision-status-cleanup
    labels:
      distr.sh/job: deployment-revision-status-cleanup
    args: [cleanup, DeploymentRevisionStatus, --timeout, 10m]
  - name: deployment-target-metrics-cleanup
    labels:
      distr.sh/job: deployment-target-metrics-cleanup
    args: [cleanup, DeploymentTargetMetrics, --timeout, 10m]
  - name: oidcstate-cleanup
    labels:
      distr.sh/job: oidcstate-cleanup
    args: [cleanup, OIDCState, --timeout, 10m]
  - name: artifact-blob-cleanup
    labels:
      distr.sh/job: artifact-blob-cleanup
    args: [cleanup, ArtifactBlob, --timeout, 10m]
  # - name: organization-cleanup
  #   labels:
  #     distr.sh/job: organization-cleanup
  #   args: [cleanup, Organization, --timeout, 10m]

Additional configuration

Some cleanup targets allow for additional customization using the following environment variables:

STATUS_ENTRIES_MAX_AGE="24h"
METRICS_ENTRIES_MAX_AGE="24h"
CLEANUP_ARTIFACT_BLOB_MIN_AGE="24h"
CLEANUP_ORGANIZATION_MIN_AGE="720h"

Caution

STATUS_ENTRIES_MAX_AGE and METRICS_ENTRIES_MAX_AGE do not have a default value. If unset, the cleanup routine for these targets will have no effect.

Caution

The Organization cleanup target permanently and irreversibly deletes organizations and all their associated data. The default retention period is 30 days (720h). Only enable this job if you are sure you want to purge soft-deleted organizations.

Upstream Tag Synchronization

When the pull-through cache is in use, Distr can automatically sync tags from upstream registries on a schedule. This is configured via a single environment variable:

# Cron schedule for syncing tags from upstream registries
REGISTRY_UPSTREAM_SYNC_CRON="15,45 * * * *"

If REGISTRY_UPSTREAM_SYNC_CRON is not set, no automatic sync is scheduled. Tags are still fetched on demand when a client pulls an image, but the tag list will not be proactively refreshed.

Note

In high-availability deployments, the built-in scheduler runs on every instance. To avoid redundant upstream requests, consider using a Kubernetes CronJob to trigger syncs externally instead.

Removed features

Deployment & Deployment Target Log Cleanup (removed in v3.0.0)

Prior to v3.0.0, deployment and deployment target logs were stored in PostgreSQL and pruned by the DeploymentLogRecord and DeploymentTargetLogRecord cleanup targets, keeping the newest LOG_RECORD_ENTRIES_MAX_COUNT entries per deployment resource or deployment target (count-based retention).

Logs are now stored in Grafana Loki and retention is time-based, managed entirely by the Loki configuration (the shipped default is 30 days via limits_config.retention_period). The cleanup targets, their cron environment variables (CLEANUP_DEPLOYMENT_LOG_RECORD_*, CLEANUP_DEPLOYMENT_TARGET_LOG_RECORD_*), and LOG_RECORD_ENTRIES_MAX_COUNT have been removed.

Deployment Target Status (removed in v2.20.0)

Prior to v2.20.0, Distr tracked a separate DeploymentTargetStatus per deployment target. This concept has been removed and is no longer available as a cleanup target.