Introducing Distr v3
After 26 minor feature releases in the 2.x series, Distr v3 replaces the log storage backend with Grafana Loki for longer log retention.
I am Philip, co-founder of Distr, which helps software and AI companies distribute their applications to self-hosted environments.
Our Open Source Software Distribution platform is available on GitHub (github.com/distr-sh/distr).
Distr 3.0 is out, our first major release after 26 minor feature releases in the 2.x series. The headline change is a new log storage backend that increases log retention.
We also just passed 1,200 stars on GitHub. Thank you to everyone who has starred, filed issues, and contributed. If Distr helps you ship software to your customers, a star helps us reach more vendors.
If you use Distr, there is nothing you need to do; your log viewer already runs on the new backend. If you self-host, this major release changes your deployment setup, so read the upgrade notes below.
Using Loki as the Logging Backend
When we shipped container log collection initially, log records went into PostgreSQL next to everything else, and retention was count-based: cleanup jobs kept the newest 10,000 entries per deployment resource and dropped the rest.
We needed to fine-tune the cleanup job multiple times over the past weeks as some of our customers streamed thousands of log records every minute, which led to constant log record churn, and PostgreSQL needed to constantly either store or delete logs. Also, some of our customers didn’t even have a couple of days of logs available, which made debugging past issues particularly challenging.
As a lot of our self-hosted and open-source community users also value our simple architecture with only PostgreSQL and an object storage as services with a persistence requirement, we were hesitant to introduce another database to store time series data (like InfluxDB) or even Elasticsearch for full-text search.
We have already been using Grafana Loki internally for years, and after a short PoC we decided to use it as the logging backend for Distr instead of further tuning PostgreSQL (e.g. with BRIN indexes, table partitioning, or a PostgreSQL extension like TimescaleDB).
Loki was a great fit, as it requires no persistent volume of its own and only uses an object storage as the backend (it even creates different paths for tenants, so each tenant’s logs are stored in a separate path). For smaller self-hosted deployments, it is a single container running the Go application in monolithic mode, while we run Loki in distributed mode, which scales to hundreds of GBs of log ingestion per day.
CPU and memory usage after the rollout
We rolled the change out to our own infrastructure first. Here is what it looked like on deployment day; the rollout happened at around 12:45:
| Memory usage | CPU usage |
|---|---|
 |  |
Memory went up a bit, as the Loki ingesters, queriers, and distributors now run alongside the Distr backend and buffer recent log chunks in memory before flushing them to object storage. That is a fixed, predictable cost we gladly pay.
CPU usage dropped noticeably. Before v3, cleanup jobs rotated thousands of log records in PostgreSQL every 10 minutes to enforce the count-based retention, and those constant delete cycles caused the tall CPU spikes on the left of the chart. With Loki, expired chunks simply age out of object storage, so the spikes are gone and PostgreSQL spends its cycles on actual application queries.
Log retention
All Pro users now get 7 days of log retention, and the Business plan extends this to 30 days.
Looking Back at the Distr 2.x Series
Distr follows Conventional Commits, and our release versions are derived from them: new features bump the minor version, and only a breaking change triggers a major one. After 26 minor feature releases since Distr 2.0, the Loki migration is the first breaking change, which makes this release v3.
A lot has shipped in the 2.x series. Some highlights, linked to their release:
- 2.4: Secrets Management and an improved deployment flow
- 2.7: Health Checks for Docker Compose deployments with distinct “running” and “healthy” states, and Agent Log Collection
- 2.8: TOTP Multi-Factor Authentication
- 2.11: Pre-Flight Checks and custom Helm v4 upgrade options
- 2.12: License Keys, signed JWT tokens your application can verify to enforce usage limits, seat counts, and feature access
- 2.14: Support Bundles, collecting logs and diagnostics from a customer deployment in a single archive, without SSH access or screen sharing
- 2.15: metric-based Alerts, plus Docker Agent improvements like container health checks with autoheal, disk metrics, and pruning of previous container images
- 2.16: the new Log Viewer
- 2.18: Vendor Billing and License Templates
- 2.21: a Pull-Through Cache for the registry
- 2.22: Partner Organizations* and role scopes for Personal Access Tokens
- 2.25: the Deployment History timeline and referencing env files for Docker Agents
- 2.26: Branding improvements, including custom page titles, favicons, and showing your logo to all users and on the login page
* Partner Organizations are available on the Business plan. A partner is a reseller that sells the vendor’s software to its own customers and handles their onboarding, support, and management in Distr, while receiving applications, release notes, and other resources from the vendor.
Upgrading to Distr v3
This is a breaking change for self-hosted installations: Distr now requires a Loki instance and will not start without LOKI_URL set.
Both shipped deployment methods include a preconfigured Loki, and its configuration values are managed by the Distr team with reasonable defaults for self-hosters. For most setups the upgrade just means picking up the new manifests:
- The Docker Compose distribution adds a
lokiservice backed by the bundled object storage. Note that provisioning its bucket uses apre_startlifecycle hook, which requires Docker Compose 5.3.0 or newer. The Docker self-hosting docs show a workaround for older versions. - The Helm chart bundles a Loki instance (enabled by default) that persists to the configured object storage. Details are in the Kubernetes self-hosting docs. For the upgrade itself, we recommend a two-step rollout: first deploy the v3 chart with the Distr image tag pinned to your current v2 version, so Loki gets deployed and becomes ready, then remove the pin to upgrade Distr to v3. Otherwise the Distr backend can start before its Loki dependency exists. We used the same procedure for our own infrastructure.
If you run your own Loki, we recommend configuring it with authentication and a time-based retention policy.
The following environment variables have been removed:
CLEANUP_DEPLOYMENT_LOG_RECORD_*CLEANUP_DEPLOYMENT_TARGET_LOG_RECORD_*LOG_RECORD_ENTRIES_MAX_COUNT
If you configured the DeploymentLogRecord and DeploymentTargetLogRecord cleanup cron jobs (e.g. as custom entries in the Helm chart’s cronJobs list), remove them, as the cleanup command no longer knows these targets. The full list of new environment variables is in the configuration reference.
One thing to be aware of: log records collected before the upgrade are not migrated to Loki. The log viewer starts fresh with logs collected after the upgrade. The old records remain in your PostgreSQL database, as the log tables will only be dropped in a future minor release, so a rollback to v2 is possible without losing any log history.
Join the Conversation
The whole Distr team wants to say thank you, and we hope Distr 3 helps you manage your self-hosted deployments even better.
We’d love to hear your thoughts on this release, especially if you self-host and hit anything unexpected during the upgrade.
