Netflix in Production: 25 Lessons From the Stream
A streaming system makes one promise: press play and deliver the next frame. Edge delivery, encoding, gateways, data pipelines and operations all contribute to that experience and its cost. These lessons trace the path from playback to bill using published engineering practice and arithmetic readers can rerun.
TL;DR: Twenty-five Netflix lessons with live diagrams, from Open Connect to encodes, breakers, recommendations, and stream economics. The complete stream in one place.
By Mukul Kumar Mishra · Streaming course · Updated October 23, 2026
What you will be able to do
- Trace a play from button through steering to edge cache to encoded ladder, pricing each hop in bandwidth and latency.
- Run gateway and discovery and breakers and chaos drills that keep hundreds of services alive through zone failures.
- Serve taste from EVCache and Cassandra, pipe trillions of events through Keystone and Flink, and lake history by the petabyte.
- Rank rows with recommendations, price every experiment, watch telemetry separately, schedule millions of containers, and ship without fear.
- Total cost per streaming hour monthly, ace the design-Netflix interview round, and ship a miniature streamer with proof.
How interviews test this course: play-flow narration first, steering and encode math second, resilience stories third. Lesson 24 rehearses the round.
Play Is a Promise
Assumption first: pressing play starts a contract measured in rebuffers, not requests. The client fetches manifests and picks bitrates and refills buffers ahead of stalls. Startup under two seconds keeps the promise. Every rebuffer spends trust the catalog earned. Design playback for the worst network the member owns, not the demo wifi.
- Budget startup as DNS and manifest and first segment. Seconds here decide churn.
- Adapt bitrate down before the buffer empties. Late adaptation rebuffers loudly.
- Deep dive: ElevenLabs Voice, 75ms audio budgets with the same math.
Q1. Startup must stay under 2 seconds on 3G. What loads first?
Q2. Rebuffers spike on one ISP nightly. What steering changes?
Promise priced. Now own the last mile. Lesson 02: Open Connect →
Open Connect Owns the Last Mile
Netflix serves most bytes from appliances embedded inside ISP and IXP networks, not from distant clouds. Short paths cut transit bills and latency and peak-hour congestion together. Thousands of appliances form a private CDN tuned for video. The last mile is rented space run like owned infrastructure.
- Measure offload ratio: bytes from appliances over total bytes. High nineties is the target.
- Place appliances where evening peaks concentrate. Geography decides cache hit reality.
- Deep dive: Cloudflare's 2025 Outage, where one feature file produced different failure modes by proxy version.
Q1. Peak traffic triples at 8 PM in one metro. Where do bytes come from?
Q2. ISP link saturates during finals. What absorbs the spike?
Mile owned. Now fill before demand. Lesson 03: Fill →
Fill Windows Fill Before Demand
New titles deploy to appliances during configured fill windows before release, not after the crowd arrives. Popularity prediction decides which titles land where and how many copies each earns. Faulty encodes get replaced through the same rails. Proactive caching turns launch night from stampede into routine.
- Size fill bandwidth from title bytes times appliance count over window hours. Missed windows mean launch pain.
- Prioritize by predicted popularity per region. Global hits fill everywhere, niches fill near fans.
- Deep dive: ShareChat Assignment, predicted placement with receipts.
Q1. A global launch lands Friday 8 PM. When does the title reach edge?
Q2. Faulty encode needs global replacement in hours. What rails carry it?
Filled early. Now steer each play. Lesson 04: Steering →
Steering Picks the Path
Each play session gets steered to the best source for its network moment: embedded appliance first, peers next, origin last. Health and load and congestion feed the choice continuously. Static DNS assignments cannot react to evening peaks. Steering turns topology into a live decision.
- Track served-from-edge percentage per metro per hour. Dips reveal congestion before complaints.
- Fail over mid-session without rebuffer. Sticky assignments die with sick servers.
- Deep dive: GCP us-central1, maintenance sequencing and verification constrain physical blast radius.
Q1. Appliance sickens mid-evening. Where do its sessions go?
Q2. Two paths tie on latency. What breaks the tie?
Path picked. Now encode smarter. Lesson 05: Encode →
Per-Title Encode Ladders
Fixed bitrate ladders waste bits on cartoons and starve action scenes. Per-title encoding shapes each ladder to content complexity, saving roughly a fifth of bandwidth at equal quality. Per-shot goes finer still. Codecs march forward with AV1 cutting further. Encoding is the cheapest bandwidth Netflix buys.
- Compare bits per quality point across titles. Flat ladders overspend simple scenes.
- Re-encode the head catalog on codec jumps. Back catalog pays the march forward.
- Deep dive: LLM Inference Cost, per-task shaping with the same math.
Q1. Animation and action share one ladder. What changes per title?
Q2. AV1 promises 30 percent savings. What migrates first?
Bits shaped. Now guard the door. Lesson 06: Zuul →
Zuul Guards the Door
Zuul fronts every request into the cloud with routing and filters and throttling and brownouts. Dozens of clusters pass more than a million requests per second to a hundred backend clusters. Filters shed CPU-heavy features under overload instead of dying. The door decides what the house survives.
- Size gateway capacity from peak RPS times filter cost. Brownouts trade features for survival.
- Throttle device retries at the edge. Client storms die at the door, not inside.
- Deep dive: Coinbase's May 2026 outage, a provider event with separate matching-engine and Kafka recovery failures.
Q1. Retry storm triples inbound RPS. What sheds first?
Q2. CPU-heavy personalization strains the door. What degrades gracefully?
Door held. Now find services. Lesson 07: Eureka →
Eureka Names the Fleet
Hundreds of services with thousands of instances churn hourly through deploys and autoscaling. Eureka keeps the registry so callers find live instances without frozen configs. Heartbeats and evictions prune the dead within seconds. Discovery without health is a phone book of ghosts.
- Size heartbeat intervals against eviction lag. Slow pruning routes to corpses.
- Cache registry reads client-side with refresh. Registry outages must not halt calls.
- Deep dive: etcd Handshake, silent sockets billed per connection.
Q1. Deploy replaces 500 instances in minutes. How do callers track them?
Q2. Registry blips for 30 seconds. What keeps traffic flowing?
Fleet named. Now isolate failure. Lesson 08: Hystrix →
Hystrix Breaks Cascades
Tens of billions of calls daily run wrapped in breakers that isolate latency and stop cascades and serve fallbacks. Thread pools bulkhead slow dependencies. Open breakers fail fast instead of queueing doom. Dashboards stream metrics in seconds for recovery. Resilience is a library and a habit.
- Tune timeouts from p99 and headroom. Generous timeouts queue, tight ones flap.
- Serve degraded fallbacks per dependency. Static rows beat blank screens.
- Deep dive: Salesforce's Dreamforce login stall; the public record does not establish its queue or breaker settings.
Q1. Dependency p99 triples at peak. What contains the blast?
Q2. Breaker flaps all afternoon. Timeout or threshold moves?
Cascades broken. Now budget retries. Lesson 09: Retries →
Retry Budgets Cap Storms
Retries can multiply offered load when clients retry blindly. Set a bounded budget per dependency, use backoff with jitter to spread attempts, and throttle excess retries before they reach an overloaded service. Choose limits from measured baseline load, dependency capacity, and timeout behavior rather than a universal retry percentage.
- Test each policy at its configured maximum attempt count and combined downstream fan-out before approval.
- Run the retry-storm calculator with your RPS. Storms modeled beat storms survived.
- Deep dive: the modeled Kafka rebalance scenario: one original and two retries make three attempts per affected call.
Q1. Three services retry one slow API. What caps the product?
Q2. Budget exhausts nightly at 8 PM. What moves: budget or dependency?
Storms capped. Now rehearse fire. Lesson 10: Chaos →
Chaos as a Discipline
Chaos Monkey kills instances in production so weaknesses surface on schedule, not launch night. Chaos Kong evacuates whole regions to prove multi-region claims. Game days extend the drill to people and process. Systems never killed in practice die in public. Failure rehearsed is failure discounted.
- Measure detection and recovery minutes per drill. Shrinking means the drills work.
- Start with staging, then prod off-peak, then peak with blast limits. Courage needs guardrails.
- Deep dive: AWS Thermal, cooling failed without rehearsal.
Q1. Leadership fears prod chaos. What drill runs first?
Q2. Region evacuation stalls halfway. What proved missing?
Fire rehearsed. Now cache hot. Lesson 11: EVCache →
EVCache Serves at 1ms
EVCache holds tier-0 working sets with millisecond SLAs at hundreds of thousands of requests per second. Trillions of daily operations ride memcached and sidecars and global replication. SSD tiers cut personalization storage cost with microsecond penalties. Hot paths without EVCache-equivalents pay database prices per read.
- Size working sets from hot keys, not totals. Ten percent of keys earn the RAM.
- Replicate personalization cross-region async. Stale taste beats unavailable taste.
- Deep dive: MoEngage Millisecond, owned NVMe at 1ms.
Q1. Personalization reads spike 10x at 8 PM. What serves them?
Q2. RAM bill doubles yearly. What tier absorbs cold taste?
Hot cached. Now persist globals. Lesson 12: Cassandra →
Cassandra Holds the Catalog
Catalog and viewing state ride Cassandra across regions with AP semantics: available under partitions, reconciled after. Chaos runs against production clusters to prove node loss is routine. Monitoring and auto-remediation keep thousands of nodes honest. Shared-nothing scales where shared-everything stalls.
- Size replication factor against quorum latency. Factor 3 with local quorum balances well.
- Repair and compaction schedules are production work, not chores. Skipped maintenance compounds.
- Deep dive: Discord Messages Scale, 177 nodes humbled.
Q1. Region partitions for an hour. Reads or writes bend?
Q2. Node loss during peak goes unnoticed. What architecture earns that?
Catalog held. Now stream events. Lesson 13: Keystone →
Keystone Pipes Trillions
Viewing and diagnostic and playback events flow through a Kafka-fronted pipeline into S3 and search and secondary streams. Routing services fan each event to the sinks that need it. Backpressure and QoS keep floods orderly. Pipelines without routing become swamps with retention.
- Size brokers from peak events per second times retention days. Peaks decide, averages lie.
- Separate operational metrics into Atlas-style telemetry. Mixed pipes drown signals.
- Deep dive: Roblox Kafka, 18 trillion daily with tiers.
Q1. Launch night triples events. What absorbs the burst?
Q2. One sink lags hours. What isolates the slow reader?
Events piped. Now compute nearline. Lesson 14: Flink →
Flink Computes Nearline
Stream processors turn raw events into trending rows and quality signals within minutes. Chaos-tested jobs survive instance kills mid-window. Exactly-once semantics protect counts that pay. Batch stays for history, streams decide tonight. Freshness without correctness misleads faster.
- Size state backends from window bytes times key cardinality. State decides cost.
- Checkpoint to durable stores on short intervals. Long gaps replay expensively.
- Deep dive: Razorpay Kafka, queue math across systems.
Q1. Trending needs five-minute freshness. Batch or stream?
Q2. Job dies mid-window at peak. What restores exact counts?
Nearline fresh. Now lake the history. Lesson 15: Lake →
The Lake Keeps History
S3 and Spark and open table formats hold viewing history by the petabyte for training and finance and content bets. Nightly jobs load terabytes per dataset into serving stores. Partitioning and file sizing decide query bills. Lakes without lifecycle rules store every experiment at full price.
- Price scans per query from bytes read. Partition pruning cuts most bills tenfold.
- Compact small files weekly. File count taxes every query like rows do.
- Deep dive: Notion Lake Cost, lake bills with the same math.
Q1. Analyst scans cost 10x this quarter. What prunes them?
Q2. Nightly load misses its window. What parallelizes safely?
History kept. Now taste by math. Lesson 16: Taste →
Recommendations Earn the Play
Personalization ranks rows and artwork per member from viewing history and context. Offline training feeds nearline serving through cached feature stores. SSD tiers cut the storage bill for taste data. Rows without relevance waste the most expensive pixels on screen.
- Measure take-rate per row position. Top rows carry multiples of lower ones.
- Precompute nightly and adjust nearline. Stale models bore, live models cost.
- Deep dive: Perplexity Engine, retrieval bills per question.
Q1. Row take-rate halves this month. Model or presentation fault?
Q2. Feature store bill doubles. What tier absorbs cold taste?
Taste ranked. Now test everything. Lesson 17: Tests →
Experimentation Prices Bets
Hundreds of concurrent experiments test artwork and algorithms and flows with statistical guardrails. Canary analysis promotes winners and rolls back regressions automatically. Untested launches are opinions with deploy rights. Bets without measurements are expenses.
- Size samples from minimum detectable effect. Small wins need big crowds.
- Gate launches on north-star and guardrail metrics together. Single-metric wins lie.
- Deep dive: Swappie Revenue, ten points bought with milliseconds.
Q1. Artwork test lifts clicks but cuts plays. Ship or kill?
Q2. Canary regresses latencies 5 percent. What decides rollback?
Bets priced. Now watch everything. Lesson 18: Atlas →
Atlas Watches Everything
Operational metrics flow through dedicated telemetry, never the event pipeline. Dimensional time series with fast queries catch incidents in seconds. Dashboards and alerts share one signal source. Mixed pipes drown the signals that page. Telemetry is the nervous system, budgeted like one.
- Budget series cardinality per service. Unbounded labels bankrupt stores.
- Alert on symptoms and burn rates, not raw CPU. Paged humans need meaning.
- Deep dive: NATS Flight-Data Incident, a preliminary technical account and recovery timeline.
Q1. Cardinality explodes after a deploy. What caps it?
Q2. Alerts fire on CPU while users hurt elsewhere. What re-aims them?
Signals separate. Now schedule work. Lesson 19: Titus →
Titus Schedules Millions
Millions of containers weekly ride Titus across EC2 fleets with bin-packing and autoscaling. Batch and service and GPU workloads share one scheduler with different promises. Autoscaling releases idle capacity back to encoding jobs. Schedulers without defrag strand cores while capacity idles.
- Measure bin-pack efficiency weekly. Stranded cores are rent without work.
- Separate latency-sensitive from batch-tolerant queues. Mixed promises break both.
- Deep dive: Kubernetes Course, fifteen on-call lessons.
Q1. GPU jobs starve behind batch queues. What separates them?
Q2. Evening peak needs 30 percent more cores. What frees them?
Work scheduled. Now ship safely. Lesson 20: Spinnaker →
Spinnaker Ships Safely
Continuous delivery pipelines bake and canary and promote across regions with automatic rollback. Hundreds of deploys daily stay boring through gates. Canary analysis compares metrics before full rollout. Deploys without canaries are leaps with changelogs.
- Size canary traffic and duration from error budget burn. Thin canaries miss regressions.
- Halt pipelines on canary regression automatically. Human gates sleep at 3 AM.
- Deep dive: Discord Voice Herd, safety checks and grace.
Q1. Canary passes but full rollout burns. What stage missed it?
Q2. Deploy velocity stalls on manual gates. What automates safely?
Deploys boring. Now predict capacity. Lesson 21: Capacity →
Predictive Capacity Beats Panic
Evening peaks arrive on schedule, so capacity gets predicted, not panicked. Forecasts from history and launches and marketing set fleets before demand. Predictive scaling buys steady state cheap and bursts on time. Reactive scaling buys the peak at premium and misses the start.
- Forecast peak cores from last four same-weekday evenings and launch calendar. History predicts most peaks.
- Pre-warm caches and fleets before tentpoles. Cold starts during premieres rebuffer loudly.
- Deep dive: Azure WAN, five hours unprepared.
Q1. Tentpole launches Friday. When does capacity land?
Q2. Forecast misses by 20 percent upward. What absorbs it?
Capacity preset. Now total the month. Lesson 22: Economics →
Stream Economics Totaled
Streaming bills in transit and compute and storage and licensing per viewing hour. Owned edge and shaped encodes cut the dearest line first. Regional clouds price the same play differently. Cost per streaming hour, tracked monthly, is the north-star efficiency metric. Bills without per-hour math hide in averages.
- Attribute cost per streaming hour per region. Averages hide the dearest metro.
- Negotiate transit with offload ratios as leverage. Edge share is bargaining power.
- Deep dive: Figma Multiplayer Cost, per-unit bills that close.
Q1. One metro costs 3x per hour. What lever moves first?
Q2. Codec migration promises 25 percent savings. What proves it?
Month totaled. Now feed the machine. Lesson 23: Studio →
Studio Supply Chain Feeds Play
Titles travel from cameras through ingest and QC and mezzanine masters and encodes and metadata before any play. Faulty masters get caught by automated checks and human eyes. Supply delays become launch delays with marketing attached. Content ops is the factory upstream of every lesson above.
- Track hours from ingest to playable per title. Bottlenecks live in QC queues.
- Version masters immutably. Re-encodes from lost masters cost quarters.
- Deep dive: Doltgres Data Loss, masters matter everywhere.
Q1. Launch slips a week on QC backlog. What parallelizes safely?
Q2. Master corrupts after encodes ship. What restores the chain?
Factory fed. Now ace the round. Lesson 24: Interview →
Design Netflix Interview Round
The iconic round compresses this course into forty-five minutes: clarify scale, sketch play flow, price bandwidth, pick stores, cache hot, queue async, break boxes, close with bills. Examiners probe steering and encodes and breakers hardest. Candidates with per-hour math stand apart. Speak the stream aloud.
- Rehearse the full flow in forty minutes with five reserved for questions. Pacing is scored.
- Open with numbers, close with cost. Middles vary, bookends decide.
- Deep dive: Interview Bootcamp, fifteen rounds rehearsed.
Q1. Design video sharing for 50M daily viewers. Sketch and price it.
Q2. Steering and encodes draw the hardest probes. What numbers answer them?
Round rehearsed. Now ship proof. Lesson 25: Capstone →
Ship One Stream
Capstone: build a miniature streamer in six weeks. Weeks one and two serve static ladders with adaptive switching. Weeks three and four add edge caching and steering and breakers. Weeks five and six load-test and chaos-drill and price per streaming hour. Graduates carry plays, not slides.
- Exit criteria: sub-2s startup and zero rebuffers at 5x load and bill per hour shown.
- Present the design as the interview round from lesson 24. Proof doubles as prep.
- Deep dive: System Design Course, the fifteen-lesson companion.
Q1. Demo your streamer under injected zone failure. What stays smooth?
Q2. Price your streamer per thousand hours. What dominates?
Course complete. Start the companion. System Design Course →
Sources and further viewing
Architecture facts above follow the Netflix Technology Blog series linked here. Modeled numbers in lessons are labeled estimates, not Netflix financials.
- Netflix TechBlog: Content Popularity for Open Connect
- Netflix TechBlog: Netflix and Fill on proactive caching
- Netflix TechBlog: Open Sourcing Zuul 2 at 1M RPS
- Netflix TechBlog: Introducing Hystrix for resilience
- Netflix TechBlog: Caching for a Global Netflix
- Netflix TechBlog: Caching from RAM to SSD
- Netflix TechBlog: Evolution of the Data Pipeline to Keystone
- Netflix TechBlog: Can Spark Streaming survive Chaos Monkey
- Netflix TechBlog: Titus containers at 3M per week
- Netflix TechBlog: Auto Scaling Production Services on Titus

