Back to System Design Index

Free Course15 lessonsLive diagramsMukul

Docker Containers in Production: 15 Lessons From the Layer Ledger

Nine years of containers under deploy pressure taught me one discipline: layers cached, bytes billed, and verdicts per push. Each lesson starts from a real production postmortem, states its assumptions up front, and ends in math you can re-run. The bill hides in the boundary.

TL;DR: Fifteen Docker lessons with live diagrams, from layers to registries and scans and the Kubernetes handoff. Built from production postmortems with registry math.

By Mukul Kumar Mishra · Containers course · Updated October 23, 2026

Diagram: a three-layer image stack feeding a push that bills bytes, then a per-push scan that returns a CVE verdict.
Figure 1. Layers are cached and reused; the push bills bytes and the scan runs per push. Illustrative course diagram.

What you will be able to do

How interviews test this course: layer math out loud first, security discipline second, handoff planning third. Lesson 15 ships the proof.

Lesson 01 · Foundations · Images are ledgers

Image Layers

Assumption first: an image is a stack of immutable layers, not a box. Each instruction adds a layer. Identical layers share storage and download once. Bloated layers download on every node forever. Read layer history before trusting any image. Bytes without ledgers compound silently.

base set layers added shared cached bytes billed
  • Inspect history and sizes per layer. Mystery gigabytes usually hide in one RUN step.
  • Prefer slim bases and explicit versions. Latest and full toolchains tax every pull.
  • Deep dive: Replit Agent Runtime, sandbox images with install bills.
Interview room

Q1. Image pulls take 4 minutes per node. Which layer dominates? Seen at: backend and platform loops.

Q2. Two services share 80 percent of layers. What does the registry charge? Seen at: SRE and platform loops.

Ledger read. Now order the file. Lesson 02: Dockerfile →

Lesson 02 · Caching · Order prints money

Dockerfile Order

Layer cache reuses every unchanged step and rebuilds everything below the first change. Stable steps belong early. Churning source belongs last. Dependency manifests before code cuts rebuilds tenfold. Order without cache math rebuilds the world per commit.

stable first deps early code last rebuilds rare
  • Count invalidated layers per typical commit. One early COPY can void a gigabyte of cache.
  • Split dependency install from source copy. Manifests change monthly, code changes hourly.
  • Deep dive: Lovable Preview, repeated installs without caches.
Interview room

Q1. Builds rerun npm install on every commit. What two lines move? Seen at: backend and full-stack loops.

Q2. Cache hit ratio sits at 20 percent. What reorder lifts it past 80? Seen at: platform and SRE loops.

Order set. Now shrink the ship. Lesson 03: Multi-Stage →

Lesson 03 · Size · Ship runtime, not toolchain

Multi-Stage Builds

Build stages compile with full toolchains. Final stages carry binaries and runtime only. A Go service drops from 900MB to 25MB when the compiler stays behind. Smaller images pull faster and scan faster and patch faster. Toolchains in production are freight, not features.

build stage binary kept runtime slim pulls fast
  • Compare final size against binary size. Gaps above 5x mean toolchain rode along.
  • Use distroless or slim final stages for static binaries. Shells in prod widen exploits.
  • Deep dive: Cursor Token Postmortem, payloads priced per byte moved.
Interview room

Q1. Node image ships at 1.2GB with a 60MB app. What stages fix it? Seen at: backend and full-stack loops.

Q2. Distroless breaks debugging at 3 AM. What compromise holds? Seen at: SRE and platform loops.

Ship shrunk. Now price the push. Lesson 04: Registry →

Lesson 04 · Economics · Pushes bill twice

Registry Bills

Registries charge storage and data transfer per pull. Every node pulls every image on deploy and scale and restart. A 1GB image across 200 nodes moves 200GB per rollout. Retention without lifecycle rules stores every experiment forever. Pull math decides image budgets.

100 stored bytes 62 pulled bytes 34 nodes counted rules set · modeled, not invoices
  • Multiply image size by node count by deploys per month. That product is the invoice.
  • Set lifecycle rules for dev tags and keep release tags pinned. Infinite retention is slow debt.
  • Deep dive: Vercel Fluid Bill, waiting billed by the same math.
Interview room

Q1. Two hundred nodes pull 800MB ten times daily. What is the monthly transfer? Seen at: platform and SRE loops.

Q2. Registry stores 4,000 untagged dev images. What policy clears them? Seen at: SRE and backend loops.

Pushes priced. Now pin the tag. Lesson 05: Tags →

Lesson 05 · Releases · Tags lie, digests hold

Tags Versus Pins

Mutable tags move under deploys while digests freeze bytes forever. Latest in production means unrepeatable incidents and undebuggable rollbacks. Pin digests in manifests and promote tags through environments. Repeatability without pins is luck with YAML.

tag moved digest froze promote staged rollback exact
  • Count deploys that cannot name their bytes. Each is an incident without a suspect.
  • Resolve tags to digests at deploy time and record them. Promotion carries hashes, not hopes.
  • Deep dive: Lean Proof Check, pin versions and recheck outside.
Interview room

Q1. Latest redeploys a broken build at midnight. What pinning prevents repeat? Seen at: SRE and platform loops.

Q2. Rollback must restore exact bytes. What artifact names them? Seen at: backend and SRE loops.

Bytes frozen. Now scan them. Lesson 06: Scans →

Lesson 06 · Security · Verdicts per push

CVE Scans

Scanners grade every layer against vulnerability feeds on each push. Critical CVEs block promotion while high ones ticket with dates. Unscanned bases accumulate forbidden packages silently. Scan gates without owners become ornamental red. Verdicts need humans with dates.

push scanned critical blocked high ticketed owner named
  • Track mean time from CVE publish to patched deploy. Weeks of lag are exposure with dashboards.
  • Rebuild bases monthly and fail closed on scanner outage. Silent skips ship holes.
  • Deep dive: ChatGPT Channel, boundaries priced per leak.
Interview room

Q1. Base carries 12 critical CVEs. What rebuilds first? Seen at: platform and security loops.

Q2. Scanner goes dark Friday. Do pushes proceed? Seen at: SRE and security loops.

Verdicts filed. Now guard secrets. Lesson 07: Secrets →

Lesson 07 · Secrets · Never bake the key

Secrets Discipline

Baked secrets live in layers forever, readable by anyone with pull access. Build args and env files leak through history and inspect. Inject secrets at runtime from vaults and mount short-lived tokens. Rotation without rebuilds proves the design.

vault holds runtime mounts tokens expire layers clean
  • Audit history and inspect output for key shapes. One match means rebuild and rotate.
  • Scope tokens per service and expire hourly. Broad forever-keys are breach multipliers.
  • Deep dive: Agentflayer Heist, one document out and keys with it.
Interview room

Q1. Key appears in layer history. What are the first three moves? Seen at: security and backend loops.

Q2. Service needs DB passwords at boot. What delivers them? Seen at: platform and SRE loops.

Keys safe. Now persist data. Lesson 08: Volumes →

Lesson 08 · State · Boxes forget by design

Volumes

Containers lose writable layers on restart by design. Volumes persist databases and uploads and caches outside the box lifecycle. Binds suit dev files. Named volumes suit prod data. Backups cover volumes, not containers. State without volumes is amnesia with uptime.

box forgets volume keeps backup covers restore timed
  • Size volumes from growth rate times retention. Full disks halt writes before alerts wake you.
  • Separate data volumes from log volumes. Shared fate fills both together.
  • Deep dive: Doltgres Data Loss, backups must hold data.
Interview room

Q1. Database container restarts nightly and loses rows. What persists them? Seen at: backend and SRE loops.

Q2. Volume fills at 2 AM. What alerts before writes halt? Seen at: SRE and platform loops.

Data kept. Now wire the network. Lesson 09: Networks →

Lesson 09 · Networks · Names before addresses

Networks Are DNS

Compose and overlay networks give each service a DNS name. Hardcoded IPs rot on every reschedule. Publish only edge ports and keep east-west traffic on internal networks. Exposed databases invite the internet to dinner. Names scale, addresses expire.

names set ports edged east internal internet out
  • Count published ports per host. Each is an attack surface with a number.
  • Isolate tiers on separate networks. Flat networks let breaches wander.
  • Deep dive: etcd Handshake, silent connections billed per socket.
Interview room

Q1. Service reaches DB by container IP. What breaks on reschedule? Seen at: backend and platform loops.

Q2. Which ports stay published on a three-tier app? Seen at: SRE and security loops.

Names wired. Now free the logs. Lesson 10: Logs →

Lesson 10 · Observability · Logs leave the box

Log Discipline

Stdout and stderr stream to collectors while files inside boxes die with restarts. JSON lines with trace IDs join logs to requests. Unbounded logs fill disks and drown signals. Rotation and sampling keep the stream affordable. Debugging without centralized logs is archaeology.

stdout streams IDs joined rotated capped signals kept
  • Budget log bytes per request times RPS. Verbose debug at peak is a second bill.
  • Sample health and debug logs at 1 percent. Keep 100 percent of errors.
  • Deep dive: MoEngage Millisecond, write-heavy discipline.
Interview room

Q1. Logs fill the node disk weekly. What three knobs stop it? Seen at: SRE and backend loops.

Q2. Trace a failed checkout across six services. What joins the lines? Seen at: backend and platform loops.

Stream flowing. Now gate rollouts. Lesson 11: Health →

Lesson 11 · Rollouts · Probes decide truth

Healthchecks Gate Rollouts

Health endpoints tell orchestrators when to send traffic and when to restart. Liveness restarts the stuck. Readiness gates the unready. Startup guards slow boots. Missing probes promote broken boxes with full traffic. Truth needs endpoints, not hopes.

live restarts ready gates startup guards traffic earned
  • Time p99 startup and add headroom for thresholds. Hair triggers flap healthy boxes.
  • Separate liveness from readiness strictly. Shared probes restart the merely busy.
  • Deep dive: Discord Voice Herd, checks outran grace.
Interview room

Q1. Deploys send traffic before warmup ends. Which probe fixes it? Seen at: SRE and backend loops.

Q2. Liveness kills busy workers mid-request. What separates the signals? Seen at: platform and SRE loops.

Truth gated. Now sign contracts. Lesson 12: Limits →

Lesson 12 · Contracts · Requests and limits

Resource Limits Are Contracts

CPU shares and memory limits turn noisy neighbors into budgeted tenants. Requests reserve scheduling. Limits cap bursts. OOM kills punish the limitless first. Unbounded boxes tax every neighbor at 3 AM. Contracts keep the host honest.

before requests reserved limits capped after OOM avoided neighbors calm
  • Size requests from p50 and limits from p99 with headroom. Guesses become evictions.
  • Set JVM and runtime heaps below container limits. Unaware runtimes invite the killer.
  • Deep dive: NATS Flight-Data Incident, a preliminary report with distinct restriction and passenger-recovery windows.
Interview room

Q1. One box starves twelve neighbors at peak. What contract fixes it? Seen at: SRE and platform loops.

Q2. OOM kills the payment worker nightly. Requests or limits move? Seen at: backend and SRE loops.

Contracts signed. Now match environments. Lesson 13: Compose →

Lesson 13 · Parity · Dev mirrors prod

Compose for Dev Parity

Compose files declare the same services and networks and volumes developers run locally. Parity kills works-on-my-machine before review. Override files carry dev-only ports and volumes. Drift without linting returns within a quarter. Mirrors need wiping schedules.

services declared overrides dev-only parity linted drift wiped
  • Diff dev versus prod configs monthly. Each unexplained delta is a future incident.
  • Pin dev image digests weekly. Floating dev tags hide integration breaks.
  • Deep dive: Stripe's Idempotency and Modeled Scale Teardown, documented retry semantics and explicit workload assumptions.
Interview room

Q1. Bug reproduces in prod but never locally. What differs first? Seen at: backend and full-stack loops.

Q2. Dev compose drifts quarterly. What automation pins it? Seen at: platform and SRE loops.

Mirrors clean. Now hand off. Lesson 14: Handoff →

Lesson 14 · Orchestration · Docker packs, Kubernetes runs

Kubernetes Handoff

Docker packages and Kubernetes schedules: registries feed clusters, probes become pod checks, limits become requests, Compose services become Deployments. Handoff checklists carry digests and secrets paths and health endpoints. Rework without a list repeats every lesson expensively.

images pinned probes mapped limits carried deploys green
  • Verify every Compose service has a Deployment and Service and limit before migration week.
  • Migrate stateless first and stateful last. Data moves slower than boxes.
  • Deep dive: Kubernetes Course, fifteen on-call lessons.
Interview room

Q1. Move twelve Compose services to a cluster. What migrates first? Seen at: platform and SRE loops.

Q2. Probes differ between Compose and pods. What maps to what? Seen at: SRE and backend loops.

Handoff mapped. Now ship it. Lesson 15: Capstone →

Lesson 15 · Capstone · Ship one container

Ship One Container

Capstone: containerize one service in four weeks. Week one shrinks the image and pins digests. Week two wires probes and limits and logs. Week three scans and rotates secrets and rehearses rollback. Week four ships with dashboards and runbooks. Graduates carry pulls, not slides.

image slim gates wired scans clean ship live
  • Exit criteria: pulls under 30 seconds and zero critical CVEs and rollback timed and bill shown.
  • Present registry cost before and after. Savings fund the next migration.
  • Deep dive: Kubernetes Course, the orchestration companion.
Interview room

Q1. Cut our image pulls from 4 minutes to 30 seconds. Show the plan. Seen at: platform and SRE loops.

Q2. Rollback drill missed its window. What ships Monday? Seen at: SRE and platform loops.

Course complete. Start the companion. Kubernetes Course →