Skip to content
Daniel Gietmann
← Blog
Coding

Cleaning up old Docker tags automatically without deleting the wrong thing

docker-cleanup keeps the newest tags per repository in a self-hosted Docker registry and deletes the rest on a schedule. How it sorts, why a delete can hit more tags than expected, and why no space is free afterwards yet.

Every CI run that pushes an image to a self-hosted registry creates a new tag there. After a few months a lot of versions are lying around, and usually only the last few are needed. The registry does not clear any of that away on its own.

That is why I wrote docker-cleanup. The tool talks to the registry through the HTTP API v2 that Docker and OCI registries have in common. Per repository it keeps the newest tags and deletes the rest on a schedule. It runs as a container, and it is configured through environment variables only.

A dry run first

docker run --rm \
  -e REGISTRY_URL=https://registry.example.com \
  -e REGISTRY_USERNAME=user \
  -e REGISTRY_PASSWORD=pass \
  -e RETENTION_COUNT=10 \
  -e IGNORE_TAG_PATTERNS='latest,stable' \
  -e RUN_ONCE=true \
  ghcr.io/danielgtmn/docker-cleanup:latest

DELETE_ENABLED defaults to false. In that case the container writes a line Dry-run: would delete tag manifest with repository, tag and digest to the log for every tag it would delete. Per repository there is also a summary: how many tags there are, how many of them stay and how many should go. With RUN_ONCE=true the container exits after one pass.

Instead of a username and password you can also pass a token through REGISTRY_TOKEN. If the registry has its tokens fetched from a separate token server, it first answers with 401. The container then fetches the token itself and logs in there with username and password.

Take your time looking at the list from the dry run. With DELETE_ENABLED=true exactly these tags disappear, plus every other tag that points to the same digest. Why is explained further down.

How the container sorts

Per repository the container fetches all tags and reads the manifest for each one. For multi-arch images it takes the manifest for linux/amd64, and if that does not exist, the first one in the list. It reads the age from the created field in the image's config blob.

Tags that match a pattern in IGNORE_TAG_PATTERNS always stay and are not counted. Of the remaining ones, the RETENTION_COUNT newest stay. If the container finds no manifest for a tag, it leaves that one in place as well and writes a warning to the log. The patterns are globs, separated by commas. release-*, for example, protects all tags that start with release-. With IGNORE_REPOSITORY_PATTERNS you exclude whole repositories.

How well that works depends on the date. Some build tools write a fixed date into created for reproducible builds, such as January 1, 1970. If all images have the same date, the sorting no longer says anything about which tag is new. Tags without any date end up behind all tags with a date, so they are thrown out first. Among themselves the container sorts them by name, ascending as text. The alphabetically first name then counts as the newest. With v1, v2, v9 and v10 and RETENTION_COUNT=2, v1 and v10 stay, v2 and v9 are deleted.

Deletion goes through the digest

docker-cleanup deletes the manifest a tag points to, and sends its digest to the registry to do so. If several tags point to the same image, all of them are gone afterwards. In the reference registry distribution, the delete handler first looks up all tags for this digest and then removes each of them.

A pattern in IGNORE_TAG_PATTERNS therefore only protects the name of the tag. An example: stable is in IGNORE_TAG_PATTERNS and points to the same image as v1.4.0, which has already dropped out of the ten newest. The container deletes the manifest of v1.4.0, and stable is gone afterwards too. The same happens to a tag that should actually stay among the newest if it shares its digest with a deleted tag. docker-cleanup does not check that at the moment.

If your CI always sets latest together with the newest release, nothing happens, because that release stays anyway. You have to be careful when you add a protected tag to an older image after the fact. The digest of a tag is shown, for example, by:

docker buildx imagetools inspect registry.example.com/my/app:stable

If the same digest shows up in the dry run for a tag that is to be deleted, stable is affected too.

The registry has to allow deletion

distribution rejects deletion through the API by default. It only accepts it with this entry in the configuration:

storage:
  delete:
    enabled: true

As an environment variable that is REGISTRY_STORAGE_DELETE_ENABLED=true. If the setting is missing, the registry answers with 405, and the container's log shows a Delete failed with this status for every tag.

Once the manifest is gone, the layers are still on disk. Only the registry's garbage collection frees the storage. With distribution that is a separate command:

registry garbage-collect --dry-run /pfad/zur/config.yml

With --dry-run it only shows what it would remove, without the option it really deletes. According to the documentation, the registry should be read-only or not running at all in the meantime, because otherwise an upload during garbage collection can lose layers. Garbage collection therefore belongs, separate from the cleanup, in a time window in which nobody uploads images. Harbor, GitLab and other registries have their own ways of doing this.

Schedule and restarts

Without RUN_ONCE the container runs permanently and cleans up according to CRON_EXPRESSION, by default 0 3 * * *. The time applies in the container's time zone, and without TZ that is UTC. With TZ=Europe/Berlin, 3 o'clock is also 3 o'clock German time.

In addition, one pass runs right at startup, before the schedule kicks in at all. So with DELETE_ENABLED=true every restart and every deployment deletes immediately and does not wait until the night. As long as you are still tweaking the configuration, you are better off leaving DELETE_ENABLED on false.

Without REPOSITORIES the container fetches the list of all repositories through /v2/_catalog. Not every registry exposes this endpoint. If it answers with 403 or 404, the run aborts with a hint. Then you enter the repositories yourself, for example REPOSITORIES=my/api,my/worker.

Once the dry run looks right, it looks roughly like this in Docker Compose:

services:
  registry-cleanup:
    image: ghcr.io/danielgtmn/docker-cleanup:latest
    environment:
      - REGISTRY_URL=https://registry.example.com
      - REGISTRY_USERNAME=user
      - REGISTRY_PASSWORD=pass
      - RETENTION_COUNT=10
      - IGNORE_TAG_PATTERNS=latest,stable
      - DELETE_ENABLED=true
      - CRON_EXPRESSION=0 4 * * *
      - TZ=Europe/Berlin
    restart: unless-stopped

Common questions

How do I delete old tags in a self-hosted Docker registry automatically?
docker-cleanup runs as a container, keeps the number of newest tags per repository set with RETENTION_COUNT and deletes the rest on the schedule in CRON_EXPRESSION. DELETE_ENABLED defaults to false, in which case the container only writes to the log which tags it would delete.
Why does docker-cleanup delete a tag that is listed in IGNORE_TAG_PATTERNS?
docker-cleanup deletes the manifest through its digest, and after that all tags that point to the same image are gone. A pattern in IGNORE_TAG_PATTERNS only protects the name of the tag. If a protected tag like stable points to the same image as a tag that gets deleted, it is gone afterwards too.
Why is no storage free yet after deleting Docker tags?
Once the manifest is gone, the layers are still on disk. Only the registry's garbage collection frees the storage, with distribution that is the command registry garbage-collect. It belongs in a time window in which nobody uploads images.
Why does the Docker registry answer with 405 when deleting?
distribution rejects deletion through the API by default. It is only accepted with storage.delete.enabled: true in the configuration or with the environment variable REGISTRY_STORAGE_DELETE_ENABLED=true. If the setting is missing, the docker-cleanup log shows a Delete failed with this status for every tag.