System Design Course: 15 Lessons From Real Outages
Forget hundred-link reading lists. Each lesson starts from a real production failure, teaches three ideas that prevent the next one, and points at one calculator to feel the math. Work top to bottom. Every lesson ends where the next begins.
TL;DR: Fifteen story-driven system design lessons with diagrams, from request paths to interview playbooks. Built from production postmortems, with calculators for the math.
By Mukul Kumar Mishra · System design course · Updated October 29, 2026
Map the Request Path
A browser request may cross DNS, a CDN, a load balancer, application servers, caches, queues, and a database. The diagram below is a simplified reference path, not a trace of a specific company's service. Map the actual dependencies and timing for the system you are designing.
- Every dependency can fail. Draw the systems in your real request path before choosing where to add redundancy.
- For a serial path, fourteen 20 ms hops would total 280 ms before application work; parallel work, cache hits, and local processing change the result. Measure the actual path.
- NATS's preliminary report describes a specific flight-code process defect and restrictions for about six hours. It does not disclose a web-request hop count. Read the incident review.
- Deep dive: Cloudflare's 2025 Outage: How a Bot-Feature File Broke the Proxy, a published example of shared infrastructure impact.
Draw this map for every system you touch this year. Interviews reward candidates who start at the edges. Production rewards engineers who already know where the red boxes hide.
Q1. Design TinyURL end to end, from API to storage. Seen at: Amazon-style and Google-style loops.
Q2. What happens, packet by packet, when you type a URL and press enter? Seen at: Meta-style loops and early-stage startups.
The next lesson looks at how an API makes retries safe. Lesson 02: APIs that survive →
Charge Once, Even When Asked Twice
Stripe reported $1.4 trillion in 2024 total payment volume. That dollar figure is not a transaction count. Stripe's API documentation describes idempotency keys that let clients safely retry eligible requests. The design lesson is to make retries explicit in the API contract.
- Idempotency keys let eligible retries reuse an earlier result; they do not make a distributed workflow globally exactly once.
- Store the key and the result together, atomically, or the check lies.
- Deep dive: Stripe's $1.4T 2024 Volume: Idempotency and a Modeled Scale Teardown.
Notice what the diagram never shows. No step asks the network to behave. Reliability comes from assuming retries, duplicates and reordering, then designing so none of them matter.
Q1. Design a payments charge API that survives client retries. Seen at: Stripe-style and Amazon-style loops.
Q2. A webhook fires twice. Who pays for the duplicate? Seen at: Stripe-style payments loops.
Safe retries assume the retrier is who they claim. Next lesson steals the keys. Lesson 03: Secrets that leak →
Keys Are Load-Bearing Walls
A user uploads a document. The summary comes back with their Google Drive API keys inside a rendered image. No clicks. No malware. One poisoned document and every permission the key carried.
- Scope every key to one surface and one permission set. Least privilege is a blast-radius budget.
- Short TTLs and rotation beat long-lived secrets. Stolen keys should die alone.
- Deep dives: You Uploaded a Document. It Downloaded Your API Keys. and Railway's build pipeline as a decryption path.
Lesson 2 taught safe retries. Keys decide who retries as whom. A leaked admin token turns every retry endpoint into an attacker tool, which is why signing and verification belong beside idempotency in every API design.
Q1. Where do you store third-party API keys, and what happens the hour one leaks? Seen at: fintech and developer-platform loops.
Q2. Design scoped tokens for a developer API from scratch. Seen at: Stripe-style and Google-style loops.
Secrets locked. Now the database itself dies. Lesson 04: Databases that live →
Backups Lie. Restores Testify.
Doltgres copied every row and lost customer data anyway. The backup skipped encoding metadata, then garbage collection deleted what the backup forgot. The backup succeeded. The restore testified against it.
- Replicas protect uptime. Only verified restores protect data.
- Garbage collection must never delete what it cannot prove dead.
- Deep dive: The Backup That Forgot the Data.
Replicas lag. That lag is a promise with fine print. Reads from a replica can return yesterday, which is fine for dashboards and fatal for balances. Know which reads need the primary.
Q1. Design backup and restore for a 10 TB Postgres database. Seen at: AWS-style and fintech loops.
Q2. Your replica lags 30 seconds. Which reads break first? Seen at: Meta-style data loops.
Data survives. Now connections run out at 3 AM. Lesson 05: Postgres at scale →
Ten Thousand Friends, One Door
Supabase's GA page reports 44M+ databases managed behind one pooling promise. Razorpay nearly died on a 3,000-connection cap nobody had read. Postgres forks a process per connection, so every pool is a negotiation with physics.
- Pool size is math, not vibes. Try the pool sizer with your own RPS.
- Serverless does not remove caps. It relocates them into docs nobody reads.
- Deep dives: Serverless Postgres and Razorpay Kafka.
Watch for the pooler becoming the bottleneck itself. One proxy in front of Postgres is a single point with a nice dashboard. Pair it, or shard past it, before traffic does.
Q1. Size the connection pool for 50k RPS at 20ms p99. Seen at: Google-style and Uber-style loops.
Q2. Design a rate limiter for a public API gateway. Seen at: Google-style and Lyft-style loops.
Connections tamed. Now traffic arrives in avalanches. Lesson 06: Queues that absorb →
Let the Queue Take the Punch
Razorpay aimed at 10,000 UPI transactions per second. Roblox routes 18 trillion messages a day. Neither lets traffic touch workers directly. A queue stands in the middle and absorbs what humans cannot schedule.
- Separate urgent from patient traffic or one surge starves everything.
- Partitions buy concurrency but tax brokers. Count them like money.
- Deep dives: Razorpay Kafka and Roblox tiers.
Dead letters deserve readers. Every message in the red box is a bug report from production. Teams that review them weekly find failures before customers do.
Q1. Design a notification system for 100 million users. Seen at: Meta-style and WhatsApp-style loops.
Q2. How do you process one million events per second without losing any? Seen at: Uber-style and LinkedIn-style loops.
Queues smooth traffic. Nothing smooths a traffic spike aimed at your GPUs. Lesson 07: Compute that scales →
Count Requests, Then Count Dollars
Vercel bills the wait between your function and someone else's API. One million requests per second for a month is 2.6 trillion events. At that scale the envelope math matters more than the code.
- Desired replicas equal current times utilization over target. Try the HPA sizer.
- Serverless bills wait time. Keep dependencies fast or pay for standing still.
- Deep dive: Vercel Fluid Compute.
Cold starts punish spiky traffic most. Provisioned minimums cost steady money but remove the worst tail. Model both shapes before launch, not after the bill.
Q1. YouTube upload traffic triples at 6 PM. Walk through the scaling. Seen at: Google-style loops.
Q2. When does autoscaling make things worse instead of better? Seen at: AWS-style and startup loops.
Scaled compute still fails. The killer is how it retries. Lesson 08: Retries that spare →
Retry Like Water, Not Like Fire
Replit agents retried failed deploys until retries cost more than features. Every retry is a second request arriving exactly when the system is weakest. Unbounded retries are a self-inflicted DDoS with a deploy log.
- Exponential backoff and jitter and a hard stop. Miss any of the three and retries amplify.
- Circuit breakers convert repeated failure into fast failure. Fast failure is a feature.
- Deep dives: Replit retries, GitHub's flow-table retry storm, and the retry storm calculator.
Idempotency from Lesson 2 is what makes retries safe to attempt. Without it, backoff only spaces out the damage. The two lessons compose. Design them together.
Q1. Convince me your retries cannot DDoS your own database. Seen at: Amazon-style loops.
Q2. Design a circuit breaker from scratch, states included. Seen at: Netflix-style and fintech loops.
Retries handled. Now the whole building loses power. Lesson 09: Surviving the blast →
Zones Are the Unit of Failure
AWS lost a zone to heat. NATS lost a sky to one fault. Four AI labs went dark in one morning. Different causes, one lesson. The blast radius you plan for is the blast radius you get.
- Deploy across zones with traffic live in all of them. Cold standby rots.
- Cells bound failure. One dark cell is an incident. One dark region is a disaster.
- Deep dives: AWS thermal, NATS flight-data incident and cell design.
Test the dark cell on purpose. Until a drill takes a zone away, the claim that the other two carry the load is a paragraph in a runbook.
Q1. One cloud region dies during checkout peak. Now what? Seen at: Amazon-style loops.
Q2. Design failover that customers never notice. Seen at: Google-style and Netflix-style loops.
Surviving is step one. Agreeing on what happened is step two. Lesson 10: Consistency calmly →
Everyone Agrees, Eventually
Figma lets ten cursors fight over one canvas without visible war. Stripe settles money the same way bankers did in 1494. Both solve agreement without asking every participant to stop moving.
- Pick one convergence point. Distributed agreement without a referee is a research project.
- Money uses double-entry ledgers. Collaboration uses operation order. Same instinct.
- Deep dives: Stripe's idempotency and modeled scale and Figma multiplayer.
Strong consistency costs latency. Every synchronous replica adds round trips to the critical path. Spend it only where money or safety demands it.
Q1. Two users edit the same document offline. Who wins and why? Seen at: Google-style and Figma-style loops.
Q2. Explain Raft leader election as if to a smart intern. Seen at: Every infrastructure loop.
Agreement settled. Now the data outgrows the machine. Lesson 11: Caching that pays →
Never Pay Twice for Hello
The best cache answers before the model wakes up. A voice agent spends 75 milliseconds of GPU on a greeting; the same greeting served from cache returns in 5, which is 15× less latency for identical output. The ElevenLabs teardown linked below prices that at 40% repeats, and its 1M RPS scenario assumes a 60% hit rate.
- Cache repeated work first. Greetings, prefills and hot keys pay for the whole layer.
- Every entry needs a TTL and an invalidation story. Stale money spends like real money.
- Deep dives: The Voice That Bills Per Hello, Vercel Fluid Compute and Cloudflare's Bot-Feature File Outage.
Lesson 10 taught convergence. Caches are replicas that converge lazily by design. TTL is a consistency budget measured in seconds, which is why cache design belongs right after consistency, not before it.
Q1. Design a CDN cache with instant invalidation for news pages. Seen at: Meta-style and Cloudflare-style loops.
Q2. Hit rate falls from 95 to 40 percent overnight. Debug it live. Seen at: SRE and platform loops.
Hot data served cheap. Now the cold data outgrows the machine. Lesson 11: Caching that pays →
Shard by the Query, Not the Table
Notion holds 200 billion blocks across 480 logical shards. Nobody queries all of them at once. Every query touches one workspace, so every shard holds whole workspaces. The shard key follows the access pattern.
- Shard by what queries ask together. Resharding later costs quarters.
- Hot shards are normal. Plan the split before the fire.
- Deep dive: Notion warehouse.
Count twice before sharding. Sharding too early trades simple problems for distributed ones. Shard when one machine groans, not when a blog post excites.
Q1. Design photo storage for two billion pictures with a global feed. Seen at: Meta-style loops.
Q2. How do celebrity posts break naive sharding? Seen at: X-style and Discord-style loops.
Storage solved. Then finance reads the bill. Lesson 13: Cost is architecture →
The Bill Is a Design Document
Perplexity pays for every question twice. Cursor users hit a paywall at 11:47 AM. Voice AI bills per hello. Each bill traces back to an architecture choice nobody priced at design time.
- Price one request before scaling to millions. Try the token bill and RPS envelope calculators.
- Caching is margin engineering. Every repeated hello should be nearly free.
- Deep dives: Perplexity and Cursor tokens.
Revisit the bill quarterly. Traffic shapes drift, prices change, and yesterday's optimization rots. The waterfall from the diagram deserves a standing meeting.
Q1. Price this architecture monthly, line by line, out loud. Seen at: Startup and CTO-round loops.
Q2. Cut the bill 40 percent without touching reliability. Seen at: FinOps-flavored platform loops.
You can now read any architecture. Time to perform one live. Lesson 14: Seeing production →
If Nobody Paged, Did It Happen
A model exfiltrated data for months before a 481-million-transcript review caught it. The signals existed the whole time. Nobody had drawn the dashboard that would have shown them. Monitoring is not paperwork. It is the only sense organ production has.
- Alert on symptoms users feel, not causes you guess. Latency and error rate first, CPU last.
- Sample smart. Keep every error, sample the boring middle. Try the sampling fitter.
- Deep dive: NATS Flight-Data Incident, with a reported defect and distinct recovery windows.
Lesson 8 taught surviving blasts. Blasts you cannot see, you cannot survive. Game days and failover drills are blind without signals, which is why observability closes the resilience arc, not the cost chapter.
Q1. p99 latency triples at 2 AM with no deploy. Walk through it live. Seen at: Google-style and Meta-style SRE loops.
Q2. Design the dashboards for a payments API before it launches. Seen at: Stripe-style and fintech loops.
You can now see anything. Time to perform everything live. Lesson 14: Seeing production →
The 45-Minute Performance
Pastebin. Forty-five minutes. The interviewer is not testing knowledge. They are testing whether you ask about scale before drawing boxes, name failure domains before features, and price the design before praising it.
- Clarify scope first. Pastebin for 100 users is a weekend project. For 100 million it is lessons 1 through 13.
- Estimate out loud. Writes per second, storage per year, bandwidth per day. Interviewers score the math.
- Close by breaking it. Name three failures and their fixes before they ask. That is the hire signal.
Course complete. The shelf keeps growing: all case studies and AI Slop Watch. New failures arrive weekly. So will new lessons.
Bring one failure story of your own. Interviewers remember candidates who broke production and fixed it. This course gave you fifteen borrowed ones. Earn one.