Data Pipelines That Pay: 15 Lessons From the Data Desk
I write from the pipeline trenches where the 3 AM page meets the morning invoice. Each lesson starts from a real data bill plus teaches the choice that shrinks the next one. Work top to bottom. Every pipeline here is a budget with a schedule attached.
TL;DR: Fifteen reflective data engineering lessons with diagrams, from batch windows to lakehouse bills plus pipeline cost reviews. Built from production postmortems with calculators for the math.
By Ira · Data desk course · Updated September 18, 2026
What you will be able to do
- Choose batch, stream, or tiered lanes from a freshness contract, then defend the choice with numbers.
- Size Kafka tiers plus partitions, survive rebalances, isolate backfills, plus read lag dashboards like a local.
- Design fat-row schemas plus lakehouse layouts that cut rows read and bytes scanned.
- Price any pipeline monthly with the RPS envelope plus downtime calculators, then cut the bill without touching freshness.
- Answer data engineering interviews with freshness-first designs, priced math, plus one honest failure story.
How interviews test this course: freshness contracts first, priced designs second, failure stories last. Lesson 14 rehearses all three.
Batch Has a Bedtime
I have watched a nightly job miss its bedtime by eleven minutes plus take the morning dashboard with it. Nobody paged at night. Everybody asked questions at nine. Batch fails quietly plus bills politely until the window breaks.
- Batch promises freshness by a clock time, not by an event. Name the hour in the contract before naming the tool.
- Windows shrink as data grows. Measure transform minutes per gigabyte now so growth never surprises the morning.
- Deep dive: The Workspace That Ate the Warehouse, where CDC plus hourly flows keep a calm product honest.
I love batch for what it admits. It says the business can wait until morning. When that sentence stays true, batch is the cheapest correct system on the menu. When it stops being true, no tuning saves it.
Q1. Design nightly revenue reporting for 100 million orders with a 6 AM freshness promise. Seen at: fintech plus marketplace loops.
Q2. The nightly job slips from 4 AM to 9 AM over six months. What do you measure first? Seen at: data platform loops.
Batch keeps its bedtime until the business stops sleeping. Then the stream starts calling. Lesson 02: Stream awake →
Stream Never Sleeps
I have sat beside a personalization team that counted conversions in milliseconds. Their event store carried 250,000 writes a second at one millisecond average because the gap between action plus reaction was the product itself. Sleep was never on the roadmap.
- Stream buys freshness by the millisecond plus pays in always-on compute. Price the awake hours before promising them.
- Write direction picks the hardware. Write firehoses want owned fast disks while read lakes want dense capacity.
- Deep dive: MoEngage Rent Nothing, the published 250K writes story with hardware receipts.
Lesson 1 taught bedtime discipline. This lesson teaches the price of skipping sleep. I choose stream only when the business can point at revenue inside the minute. Everything else stays batch plus stays cheap.
Q1. When does a recommendation feed need stream plus when is hourly batch enough? Seen at: marketplace plus media loops.
Q2. Size a stream store for 250K writes a second at p99 under 10ms. Seen at: infrastructure plus ads loops.
Freshness chosen. Now the pipe itself needs a budget. Lesson 03: Kafka budget →
Kafka Is a Budget With Partitions
I have seen two teams run the same Kafka with opposite invoices. One treated urgent matchmaking plus patient analytics as equals. The other gave each its own lane. Same events. Very different mornings. The tier was the difference.
- Separate urgent from patient traffic or one surge starves both. Tiers are latency insurance with a topic name.
- Partitions buy concurrency plus sell operational load. Count them like money because brokers do.
- Deep dives: Roblox 18 trillion messages plus Razorpay Kafka.
Run the shape through the RPS envelope calculator to feel the stakes. Events per second times seconds per month turns every partition choice into a line item. I learned to tier first plus scale second.
Q1. Design ingestion for one million events per second with urgent plus patient readers. Seen at: gaming plus ads loops.
Q2. How many partitions for 40K partitions worth of pain? Defend fewer. Seen at: platform plus infra loops.
Tiers set. Then a deploy restarts everything at once. Lesson 04: Rebalance tax →
Rebalances Are the Tax
I have watched a routine rolling deploy pause every consumer at once. One hundred twenty restarts plus forty thousand partitions renegotiating ownership for eleven minutes. The brokers stayed healthy. The pipeline did not move. Healthy brokers make the best alibis.
- Rolling deploys plus eager rebalances compose into pauses. Stagger restarts with static membership to keep ownership calm.
- Retries without backoff multiply the storm. Budget retries like capacity before the incident, not after.
- Deep dive: Kafka Rebalance Storm, plus the retry storm calculator.
Lesson 3 taught partition budgets. This lesson collects the tax. I now ask every review where the rebalance graph lives. Teams that cannot show it will meet it at 3 AM.
Q1. A deploy pauses all consumers for ten minutes. Walk through the fix live. Seen at: platform plus SRE loops.
Q2. Prove your retry policy cannot DDoS your own brokers. Seen at: Amazon-style loops.
Movement restored. Now the bytes themselves need rules. Lesson 05: Schema teeth →
Schemas Are Contracts With Teeth
I have lived the review where seventy fetches served one request. Every feature slept in its own row. One ranking call woke them all. Then somebody packed co-requested features into fat rows plus row reads fell a hundredfold. Naming things better beat buying hardware.
- Store what is requested together, together. Audit fan-in on the hottest request before touching hardware.
- Cache locality follows routing. Client-side entity routing plus service splits carried a published 95 percent hit rate.
- Deep dive: ShareChat 10x Cheaper Assignment, from 2B rows down to 18.4M a second.
The published arc still moves me. A feature store climbed from one million to one billion features a second without database growth. Schema craft did what scale could not. I audit fan-in first in every pipeline review since.
Q1. Design a feature store for one billion reads a second at p99 under 20ms. Seen at: ML platform loops.
Q2. Your p99 climbs while throughput stays flat. Schema or hardware? Prove it. Seen at: backend plus data loops.
Rows tamed. Now the warehouse starts billing by the scan. Lesson 06: Lakehouse bill →
The Lakehouse Bill
I have opened a blank page that hid two hundred billion blocks. Four hundred eighty shards plus change streams plus search indexes plus permission checks plus vector jobs. The page looked empty. The lake never slept. Somebody paid per scan.
- CDC turns one edit into six downstream jobs. Count the readers per write before adding another index.
- Partition by query pattern plus compact small files or scans price the mess monthly.
- Deep dive: The Workspace That Ate the Warehouse, the 200B-block lake with modeled cost.
Lesson 5 taught row discipline. This lesson teaches scan discipline. Every unpartitioned query is a full-lake invoice wearing an analytics costume. I partition by workspace plus time first, then argue about engines.
Q1. Design CDC from Postgres to a lake for 200B rows with hourly freshness. Seen at: data platform loops.
Q2. Warehouse spend triples with flat traffic. Which three checks run first? Seen at: FinOps-flavored data loops.
Scans priced. Now the disks underneath send their own invoice. Lesson 07: Slow invoice →
Storage Is the Slow Invoice
I have priced three disk options for one workload plus watched each fail differently. Memory shapes with network disks missed the direction. Premium volumes passed the test plus failed the budget. Affordable volumes failed the tail exactly when success arrived. Local NVMe won by owning the physics.
- Match disks to dominant direction. Write firehoses want local NVMe while read lakes want dense capacity.
- Demand tail promises in writing. Averages soothe dashboards while tails page humans at night.
- Deep dive: MoEngage hardware math, 250K writes at 1ms average over 200TB.
My rule from that file still guides me. When disks belong to somebody optimism, latency belongs to somebody throttle policy. I ask for the hardware receipt beside every architecture sketch now.
Q1. Pick storage for 250K writes a second with 1ms average. Defend the receipt. Seen at: infra plus database loops.
Q2. p99 spikes only during bursts on shared volumes. What changed? Seen at: SRE plus storage loops.
Bytes housed. Now the compute above them needs discipline. Lesson 08: Flink discipline →
Flink Discipline
I have seen managed streaming feel gentle in week one plus become the largest line by month six. One team moved streaming jobs off managed Dataflow onto self-run Flink plus cut streaming cost by 93 percent. Same events. Operations burden accepted on purpose. Freedom with an on-call rota.
- Prototype on managed, reprice at steady state. Stable scale plus managed per-unit prices rarely stay friends.
- Autoscale stream workers to actual load, not peak imagination. Idle parallelism is a subscription to nothing.
- Deep dive: ShareChat streaming cut, where Flink discipline led the tenth-bill march.
Lesson 2 priced awake compute. This lesson prices who operates it. I keep a yearly calendar note now. Reprice every managed pipe with operations burden on the same page. The cheapest engine is the one the team can run hot.
Q1. When do you exit managed streaming for self-run Flink? Show the math. Seen at: platform plus cost-aware loops.
Q2. Flink lags only at peak hours. Autoscaling or backpressure? Seen at: streaming plus backend loops.
Streams leaned. Then history asks for a rerun at noon. Lesson 09: Backfill bite →
Backfills Bite at Noon
I have watched a backfill eat live latency for lunch. History replayed through the same serving path as lunch traffic. Reads spiked. Dinners of dashboards went cold. The fix was not bigger hardware. It was a separate lane for the past.
- Isolate replays from live reads with quotas plus separate queues. History should wait, lunch should not.
- Throttle by live p99, not by replay speed. Backfill progress is vanity when serving bleeds.
- Deep dive: Discord hot partitions, where replay pressure met quorum amplification.
Lesson 4 taught deploy isolation. This lesson teaches time isolation. I schedule replays like roadworks now. Signed, lit, plus never across the only bridge at noon.
Q1. Replay one year of events without moving live p99. Sketch the lanes. Seen at: data plus SRE loops.
Q2. Backfill doubles serving latency. What guard trips first? Seen at: backend plus platform loops.
History contained. Now the present needs eyes. Lesson 10: Seeing pipelines →
Seeing the Pipeline
I have debugged a pipeline that looked green while going stale. Throughput graphs smiled. Lag graphs, the ones nobody had drawn, told the truth. Freshness is a signal you draw on purpose or discover by complaint.
- Alert on staleness users feel, not CPU you guess. Lag plus freshness SLO first, broker CPU last.
- Read dead letters weekly. Every poisoned record is a bug report production filed for you.
- Deep dive: Rebalance lag math, plus the sampling fitter for signal budgets.
Lesson 9 taught lane discipline. Lanes without gauges still crash. I keep three gauges now. Watermark age, dead letter growth, plus freshness burn. Everything else is decoration.
Q1. Freshness SLO burns at 2 AM with no deploy. Walk through it live. Seen at: Google-style plus Meta-style SRE loops.
Q2. Design pipeline dashboards before launch. Which three panels ship day one? Seen at: data platform loops.
Eyes open. Now finance walks into the review. Lesson 11: Cost review →
Cost Is a Pipeline Review
I have sat in the review where congratulations end plus the real assignment begins. One system scaled a thousandfold to a billion features a second. Then leadership asked for the same system at a tenth of the cost. Latency fell from 40ms to 8ms along the way. Craft with a deadline.
- Price one event before scaling to billions. Try the RPS envelope plus downtime cost calculators.
- Profile before purchasing. Half of one published serving fleet deleted itself under a profiler.
- Deep dives: ShareChat tenth bill plus MoEngage same invoice rebuild.
Every optimization in those files was available on day one. What arrived later was permission to care. I give that permission early now. The row is the rupee plus the cache is the only honest employee.
Q1. Price this pipeline monthly, line by line, out loud. Seen at: startup plus CTO-round loops.
Q2. Cut the bill 40 percent without touching freshness. Seen at: FinOps-flavored platform loops.
You can now read any pipeline bill. Time to meet skew up close. Lesson 12: Hot keys →
Hot Keys Burn Loud
I have watched a popular channel aim thousands of reads at one partition within seconds while 177 nodes stood around it. The writes were fine. The reads queued, replicas queued with them, plus quorum consistency spread the pain to cold channels sharing the hardware. The cluster did not run out of disk. It ran out of schema.
- Design schemas for reads first when reads dominate. Write-optimized layouts tax every popular key forever.
- Coalesce fan-in before adding nodes. One request answered once beats a thousand duplicate reads.
- Deep dive: Discord Trillions of Messages, 177 nodes down to 72 at 15ms p99.
Quorum multiplies every hot read across replicas, so the saturated set grows faster than the fleet. Adding nodes dilutes cold load while leaving the hot set saturated. I check the hottest partition first in every review now, because that partition is the cluster wearing a costume.
Q1. One channel draws 5K reads a second to a single partition. Fix it live. Seen at: messaging plus backend loops.
Q2. Reads at quorum spread a hot key to neighbors. Prove it with math. Seen at: database plus infra loops.
Skew tamed. Now the ceiling nobody read about. Lesson 13: Connection ceiling →
The 3,000-Connection Ceiling
I have lived the surge plan that assumed infinite headroom. Pods scaled out confidently ahead of a final match, aiming at 10,000 transactions a second. The broker counted connections to 3,000 plus stopped accepting friendship. Nobody on either side had mapped the limit. The cap lived in documentation nobody reads before an incident.
- Read managed caps before surge day. Serverless removes provisioning, never limits.
- Swap the client before the cluster. One library change cut connections from 1,100 to 300 plus latency from 250ms to 10ms.
- Deep dive: Razorpay Kafka, five years from SQS to provisioned with receipts.
The bottleneck was never Kafka. It was the code talking to Kafka. I keep a caps page for every managed service now: connections, partitions, throughput, plus the authentication tax. Surge plans reference it or they reference hope.
Q1. Design UPI payments for 10K TPS with a surge plan. Name every cap. Seen at: fintech plus payments loops.
Q2. Latency falls 25x from a client swap. Explain the mechanism. Seen at: backend plus infra loops.
Caps mapped. Now turn all of it into interview answers. Lesson 14: Interview pipeline →
The Interview Pipeline
I have watched strong engineers fail data rounds they could have owned. They described tools when the panel wanted budgets. They listed engines when the panel wanted freshness contracts. Interviewers hire engineers who price the pipe plus name its failure modes. This lesson turns fourteen lessons into answers.
- Open with freshness. Batch by morning, stream by minute, or tiered by reader urgency, stated in the first minute.
- Price out loud with the RPS envelope calculator. Panels score the math more than the engine names.
- Close with one failure story plus its fix. Borrow mine until you earn yours: the backfill that ate lunch, tamed by Lesson 09.
Rehearse three answers this week: a nightly reporting design, a stream sizing with hardware receipts, plus a bill cut of 40 percent without touching freshness. Say row counts, partition counts, plus rupee counts. Tools are adjectives. Numbers are nouns.
Q1. Design clickstream analytics for one billion events a day. Price it monthly. Seen at: data platform plus marketplace loops.
Q2. Tell me about a pipeline you broke plus fixed. What changed forever? Seen at: behavioral plus data loops.
Answers rehearsed. Now price a pipeline of your own. Lesson 15: Price yours →
Price Your Own Pipeline
Bring one pipeline from your own work. Mine was a recommendation feed that outgrew its nightly window. Yours might be logs, billing events, or search signals. We will give it lanes, a schema, a lake plan, plus a bill that survives review.
- State freshness first. Batch by morning, stream by minute, or tiered by reader urgency.
- Name the partition key, the fat-row grouping, plus the lake layout in one page. Vague plans bill vaguely.
- Close with the invoice. Events per month times modeled unit price plus storage growth, labeled as estimates.
Course complete. Keep reading the shelf: all case studies plus the system design course. New pipelines arrive weekly. So will new bills.
Bring one failure story of your own. Interviewers remember engineers who priced the pipe plus fixed it. This course gave you twelve borrowed ones. Earn one.
Sources plus method
This course teaches from published Buildopsy postmortems linked under each lesson: ShareChat feature store economics, MoEngage event store hardware, Notion lake architecture, Razorpay Kafka, Roblox tiers, Kafka rebalance math, plus Discord scale. Throughput figures follow those accounts. Cost framing is modeled from public list prices.