Michael Krausch-Hofmann
DevOpsSRE person and engineer at Red Hat leading the pipeline and reliability work for building "modern" container images in Project Hummingbird
Sessions
Update one package in your container image and watch your users re-download 500MB of unchanged content. Traditional Dockerfile layers are instruction-based—a single package update invalidates an entire layer.
This talk introduces chunkah, a tool that post-processes container images into content-based layers. Files are grouped by package, not Dockerfile structure. Update one package, users download only that layer.
We'll cover:
- Why instruction-based layers hurt pull performance (with numbers)
- How content-based splitting works under the hood
- Live demo: chunkah in action, before/after comparison
- How
podman historyreveals package-to-layer mapping
From Project Hummingbird, where we run 70+ production container images with chunkah, we'll show real metrics on bandwidth savings and how to adopt this in your own builds.
Audience: Container image builders, CI/CD engineers, registry operators, anyone who's wondered why their image pulls are so slow.
Rebuild the same Containerfile with the same packages and you'll get a different image hash. This isn't a bug—it's the default behavior of container builds, and it breaks verification, caching, and supply chain security.
This talk explores the surprisingly hard problem of reproducible container builds. We'll dissect exactly what breaks reproducibility—timestamps, SQLite journal modes, machine-id files, transaction logs—and show practical techniques to fix each one.
We'll cover:
- Live demo: same Containerfile, different hash—why?
- SOURCE_DATE_EPOCH and timestamp normalization
- The SQLite WAL surprise—and the one-line fix
- A checklist of artifacts to remove for reproducibility
- Verification: rebuilding from SLSA provenance attestations
From Project Hummingbird (70+ container images, SLSA Level 3), we'll show how to achieve bit-for-bit identical rebuilds in your CI/CD pipelines.
Audience: CI/CD engineers, security teams, anyone wondering why their image rebuilds don't match.