Event Streaming in Production: 15 Lessons From Published Cases
A resilient event-streaming design treats the log as a source of truth, expects delivery to repeat, and prices backlog recovery. These lessons draw on published incidents and clearly labeled scenarios, with assumptions and math readers can rerun.
TL;DR: Fifteen streaming lessons with diagrams, from partitions and exactly-once budgets to rebalance drills. Published incident findings and modeled inputs are identified separately.
By Mukul Kumar Mishra · Event streaming course · Updated October 31, 2026
What you will be able to do
- Size partitions and consumer groups from throughput math, then plan for the rebalances deployments trigger.
- Preserve per-key ordering under hot keys, route dead letters to readers, and evolve schemas without breaking consumers.
- Price retention and exactly-once guarantees as budgets, then defend them with modeled numbers.
- Cap retries with jitter and circuit breakers, separate urgent from patient traffic, and estimate the monthly backlog.
- Practice streaming designs by clarifying scope, estimating out loud and naming failure modes.
Practice in order: clarify scope, estimate the workload, then test your design against failure cases. Lesson 14 walks through that sequence.
The Log Is the Truth
Assumption first: every downstream system disagrees about what happened. The log settles it. One gaming platform routes 18 trillion messages a day through one append-only backbone, then replays any of them on demand. Databases forget. Logs remember. Design the memory first.
- Append, never overwrite. Immutable history makes every downstream rebuild a replay, not a rescue.
- One backbone, many readers. Matchmaking and analytics read the same events at their own pace.
- Deep dive: 18 Trillion Messages a Day, one platform for urgent and patient traffic.
Draw this shape for every pipeline you inherit this year. Interviews reward candidates who start from the log. Production rewards engineers who can replay the Tuesday nobody understood.
Q1. Design event ingestion for a game with 18 trillion daily messages.
Q2. A downstream team corrupts its state. How does the log save them?
Truth stored. Now split it for speed. Lesson 02: Partitions →
Partitions Buy Concurrency
A partition can be assigned to one consumer in a group at a time. In this modeled example, six partitions allow up to six group members to own a partition concurrently; adding more consumers cannot raise that partition-limited ceiling. Real throughput also depends on partition service rate, ordering needs and downstream capacity.
- Partitions set the concurrency ceiling. More consumers than partitions means paid idleness.
- Every partition taxes the broker. Assignment math and metadata grow with the count, not with the data.
- Deep dives: Razorpay Kafka and Roblox tiers.
Size partitions from peak throughput divided by per-partition headroom, with a stated growth factor. Lesson 4 shows what happens when the count meets a rolling deploy. The ceiling is load-bearing.
Q1. Size partitions for 60,000 events per second with 3x headroom.
Q2. Consumers idle while lag grows. What do you check first?
Concurrency bought. Now share it across a group. Lesson 03: Groups →
Groups Share the Load
Modeled setup: 120 consumers share 40,000 partitions, an average of about 333 partitions per member. A consumer group is a contract: the coordinator assigns, members consume, generations mark agreement. Assignment count alone does not specify rebalance duration, and these group-size inputs do not describe a reported incident.
- Under the classic eager protocol, every member revokes its current assignment before a new assignment is installed; group-wide record processing can pause during that handoff.
- In that modeled protocol, repeated membership changes can keep restarting the handoff. Other group protocols have different assignment behavior.
- Deep dive: the modeled Kafka rebalance cascade, with its timing inputs stated separately.
Sketch your groups with member counts, partition counts and generation dashboards before the next deploy. The map from Lesson 1 now has owners. Owners restart. Restarts rejoin. Rejoins are Lesson 4.
Q1. Explain consumer groups and generations to a smart intern.
Q2. Two teams share one group id by accident. What breaks?
Load shared. Now trace the classic-eager cascade. Lesson 04: Rebalance →
Classic Eager Rebalances Can Pause the Group
Modeled classic-eager scenario: a rolling deploy changes membership in a 120-member group with 40,000 partitions. If each new membership change arrives before the prior rebalance completes, group-wide consumption can remain paused. The companion model assumes an 11-minute pause; Kafka does not derive that duration from partition count alone.
- Classic eager rebalancing requires every member to revoke current assignments before the new assignment is installed; cooperative assignors use a different handoff.
- A consumer that misses its configured poll or session limit can create another membership change. Check those timers before treating them as recovery controls.
- Deep dives: Kafka Consumer Rebalance Explained and the modeled cascade scenario.
When asked how a deploy can pause a classic-eager group, describe the overlap: a membership change arrives before the prior rebalance completes. The companion example assumes an 11-minute pause; partition count alone does not provide that duration. Fix the trigger race, not the brokers.
Q1. In a modeled classic-eager scenario, a rolling deploy leaves a group paused for an assumed 11 minutes. Explain the overlap condition and what the number does—and does not—establish.
Q2. Brokers look healthy while consumption stalls. Where do you look?
Storm modeled. Now remove the trigger. Lesson 05: Static ids →
Static Ids Can Avoid Some Rebalances
A stable group.instance.id with a suitable session timeout can prevent removal of a member during brief unavailability, such as a short process restart. If the member returns after timeout, Kafka can remove it and reassign its partitions. Cooperative assignment can retain partitions that keep their owner, but it does not make group changes rebalance-free.
- A stable instance ID may avoid removal during transient unavailability when the member returns before its session timeout.
- A cooperative assignor may retain partitions that stay with their current owner and transfer revoked partitions in a later round.
- Deep dive: Kafka Consumer Rebalance Explained for protocol mechanics and conditional mitigations.
Review protocol defaults alongside capacity. The modeled example combines several individually reasonable choices; it is not evidence of a customer outage. Review static membership, assignment protocol, rollout pacing, timeout behavior and retry budgets together, then validate them against measured group behavior.
Q1. Compare classic eager and cooperative assignment. What can a 40,000-partition count tell you about rebalance duration?
Q2. What breaks when two members share one static id?
Storms ended. Now keep events in order. Lesson 06: Key order →
Order Lives in the Key
Assumption: global order is purchasable. It is not. Ordering holds within a partition, so the key decides the queue. Hash by order id and one order never overtakes itself. Hash randomly and refunds precede charges. A chat platform learned the hot-key corollary: one popular channel concentrates 5,000 reads per second onto a single partition.
- Key by what must stay ordered. Payments by order id, messages by channel, sessions by user.
- Plan the hot-key split before the fire. Celebrity keys break naive hashing on event night.
- Deep dive: Discord message storage, hot partitions with receipts.
List your top ten keys by volume before launch, with a split plan for each. Capacity follows the hottest partition, never the average. The average is a rumor the hot key never heard.
Q1. Keep per-user order for a chat app at 5,000 reads per second per channel.
Q2. One key takes 80 percent of traffic. Now what?
Order kept. Now read the mail nobody wants. Lesson 07: Dead letters →
Dead Letters Deserve Readers
A consumer can repeatedly fail on the same poison event, using worker time that healthy messages need. A dead-letter topic quarantines the event after a bounded number of attempts. Review recurring failures to find issues before they affect more messages.
- Bound attempts before routing to dead letters. Unbounded poison retries are a self-inflicted stall.
- Alert on dead-letter growth rate, not just depth. A rising slope is an incident writing its first draft.
- Deep dive: 18 Trillion Messages a Day, patient lanes for patient problems.
Staff the dead-letter review like on-call, because it is one. Quarantine without readers is a morgue with extra steps. Readers turn poison into patches.
Q1. Design poison-message handling for a payments pipeline.
Q2. Dead letters grow 10x overnight. What happened?
Poison quarantined. Now sign contracts with producers. Lesson 08: Schemas →
Schemas Are Contracts
Producers and consumers do not always upgrade together. A renamed field can break older consumers. A schema registry makes evolution a reviewed contract: check backward and forward compatibility at publish time, and reject breaking changes before they reach the log.
- Gate publishes on compatibility. Add optional fields freely. Renames and type changes need a migration plan.
- Version every event. Consumers read the version first, the payload second.
- Deep dive: Notion's 200 Billion Blocks and the Warehouse Bill Behind a Blank Page, change capture across 480 shards.
Register your three highest-volume event types this week with compatibility rules written down. Ungated schemas are handshake deals with teams you have never met. The log remembers what the meeting forgot.
Q1. Rename a field consumed by forty services without downtime.
Q2. A producer ships a breaking change at midnight. Who stops it?
Contracts signed. Now price keeping history. Lesson 09: Retention →
Retention Is Money
Retention has different storage and access costs. Hot NVMe can cost more per gigabyte-month than object storage, while retained data also brings read, replication and rebalancing work. Separate urgent matchmaking from patient analytics so they do not share the same service level or storage tier.
- Tier by access age, not by sentiment. Yesterday serves from NVMe. Last quarter serves from object storage.
- Price retention per topic per tier monthly. Unpriced history grows until the invoice explains it.
- Deep dive: 18 Trillion Messages a Day, urgent versus patient by design.
Follow the packet above from hot to cold, then tier your own oldest topic. Retention without tiers is a subscription to urgency. The bill hides in the boundary between hot and cold.
Q1. Keep a year of events queryable without paying hot prices all year.
Q2. Replay demand spikes on six-month-old data. What breaks?
History tiered. Now promise exactly-once honestly. Lesson 10: Exactly-once →
Exactly-Once Is a Budget
Assumption: the network delivers once. In practice, queues and clients can retry, so consumers need a duplicate-handling strategy. Stripe reported $1.4 trillion in 2024 total payment volume; that is dollar volume, not a transaction count. Its API documentation describes idempotency keys for eligible retries, not a global exactly-once guarantee.
- Key each operation and commit its deduplication record with the effect. That can provide once-per-key behavior for a defined operation, not global exactly-once processing across every service.
- Store the key beside the result atomically, or the check lies under concurrency.
- Deep dive: Stripe's $1.4T 2024 Volume: Idempotency and a Modeled Scale Teardown, which separates reported payment volume from modeled event rates.
Look at what the diagram leaves out. Nothing in it asks the network to behave. Reliability here is a design stance: expect retries, duplicates and reordering, then make correctness independent of all three. Exactly-once is not a protocol setting; it requires a budget and an idempotency key.
Q1. Guarantee exactly-once charging over an at-least-once queue.
Q2. Dedup storage grows forever. When do keys expire?
Duplicates tamed. Now cap the retry wave itself. Lesson 11: Retry storm →
Retries Without Jitter Are DDoS
Modeled example: if producers continue at 60,000 events per second while consumers make no progress for 660 seconds, the backlog is 60,000 × 660 = 39.6 million events. Separately, two immediate retries mean one original and two retries, or three attempts per affected call. Backlog and retry attempts are different quantities; the example is not Kafka telemetry.
- Use backoff, jitter and a hard stop. Without them, retries amplify instead of healing.
- Price your policy with the retry storm calculator before the incident instead of after.
- Deep dive: the modeled Kafka rebalance cascade, with recovery arithmetic separated from protocol facts.
Retries are only safe to attempt because of the idempotency from Lesson 10. Without it, backoff just spreads the damage out more evenly. Design the two together and review them together.
Q1. Show how your retry policy avoids overwhelming its own brokers.
Q2. Lag is extreme and breakers are open. What consumes first on recovery?
Amplifier capped. Now slow the producer instead. Lesson 12: Backpressure →
Backpressure Beats Buffering
Assumption: bigger buffers absorb bigger spikes. Buffers only postpone the reckoning while hiding the signal. A payments migration aimed at 10,000 transactions per second survived by separating urgent payments from patient analytics into lanes with independent workers. Backpressure tells producers to slow down. Buffering tells operators everything is fine until it is not.
- Lane traffic by urgency before it shares workers. One surge must never starve everything.
- Report drops and delays explicitly. Visible failure is safer than silently losing data.
- Deep dive: Razorpay Kafka, a reported connection cap and a client-migration account whose exact benchmark deltas remain marked for verification.
Separate your own urgent from patient traffic this quarter, then load-test the urgent lane alone. Queues that absorb spikes are designed. Queues that merely buffer are postponed incidents with dashboards.
Q1. Traffic triples at 6 PM with mixed urgency. Walk through the flow.
Q2. When does backpressure hurt more than buffering?
Flow steadied. Now envelope the month. Lesson 13: Envelope →
The Envelope Math
Throughput is also a financial input. In the modeled example, 60,000 events/s × 30 days × 86,400 s/day = 155,520,000,000 events, about 155.5 billion events in a 30-day month. A separate assumed 660 s pause with no consumer progress yields 60,000 × 660 = 39,600,000 backlog events. These are scenario calculations, not Kafka telemetry. Run your own arrival and drain rates through the envelope calculator.
- Multiply before you build. Events per second times seconds per month is the invoice in waiting.
- Run your workload through the RPS envelope calculator and the pool sizer.
- Deep dive: the modeled Kafka rebalance cascade, with backlog inputs and arithmetic labeled.
Present envelopes as ranges with stated assumptions, never as single confident numbers. Precision without assumptions is theater. The bill hides in the boundary between the model and the month.
Q1. Size a pipeline for 155 billion monthly events, out loud.
Q2. Retention doubles while the budget stays flat. What moves tiers?
Month priced. Now rehearse the interview. Lesson 14: Interview bench →
The Interview Bench
For a notification design serving 100 million users, clarify scope, estimate the workload and test the design against failure cases. Broker details matter only when they support the design.
- Clarify scope before drawing. Urgent versus patient, ordered versus not, retained for how long.
- Estimate partitions, consumers and backlog with the RPS envelope calculator. Show the math.
- Test your own design against hot keys, rolling restarts and poison events, with a fix for each.
Rehearse three answers: partition sizing with headroom math, a rebalance scenario with protocol-specific fixes, and an exactly-once design with key-expiry assumptions. Use diagrams to make the assumptions visible.
Q1. Design notifications for 100 million users end to end in 45 minutes.
Q2. Describe a pipeline failure and how you fixed it.
Answers rehearsed. Now run one pipeline for real. Lesson 15: Run pipeline →
Run One Pipeline
Start small on purpose. One topic, three consumers, one dashboard. Produce a million events, kill a consumer mid-run, watch the rebalance, read the dead letters, replay the log, price the month. A pipeline you have broken on purpose is the only kind you understand.
- Run the fourteen drills from lessons 1 through 13 against one real topic. Gaps are visible at a glance.
- Stop one consumer mid-run and measure the rebalance. Untested failover is not a runbook.
- Close with the invoice. Events per month times modeled unit price, labeled as estimates.
Course complete. Continue with the System Design Course, browse all courses, or explore the case studies.
The log remembers. The invoice explains. You just built the study guide. Earn the grade.
Sources and Method
Lessons link to Buildopsy's published incident analyses and explicitly labeled models, including the Kafka rebalance scenario. Reported incident facts are kept separate from modeled inputs. Cost examples use labeled assumptions and public list prices.