Data Pipelines That Pay: 15 Lessons From the Data Desk
Each lesson connects a data-engineering choice to its reliability and cost consequences. Work top to bottom: every pipeline is a budget with a schedule attached.
TL;DR: Fifteen data-engineering lessons with diagrams, from batch windows to lakehouse bills and pipeline cost reviews. Published incidents and modeled examples are identified separately.
By Mukul Kumar Mishra · Data desk course · Updated October 24, 2026
What you will be able to do
- Choose batch, stream, or tiered lanes from a freshness contract, then defend the choice with numbers.
- Size Kafka tiers and partitions, survive rebalances, isolate backfills, and read lag dashboards with confidence.
- Design fat-row schemas and lakehouse layouts that cut rows read and bytes scanned.
- Price any pipeline monthly with the RPS envelope and downtime calculators, then cut the bill without touching freshness.
- Answer data engineering interviews with freshness-first designs, priced estimates, and one honest failure analysis.
How interviews test this course: freshness contracts first, priced designs second, failure stories last. Lesson 14 rehearses all three.
Batch Has a Bedtime
Modeled schedule: a nightly job finishes 11 minutes after its freshness target, leaving the morning dashboard stale. The duration is an exercise input, not a reported incident. State the deadline, alert on lateness and measure how long the backlog takes to clear.
- Batch promises freshness by a clock time, not by an event. Name the hour in the contract before naming the tool.
- Windows shrink as data grows. Measure transform minutes per gigabyte now so growth never surprises the morning.
- Deep dive: Notion's 200 Billion Blocks and the Warehouse Bill Behind a Blank Page, where CDC and hourly flows serve a calm product.
Batch fits when the business can wait until morning. While that freshness contract holds, it avoids paying for always-on processing. Once it stops holding, tune the contract or choose a streaming path.
Q1. Design nightly revenue reporting for 100 million orders with a 6 AM freshness promise.
Q2. The nightly job slips from 4 AM to 9 AM over six months. What do you measure first?
Batch keeps its bedtime until the business stops sleeping. Then the stream starts calling. Lesson 02: Stream awake →
Stream Never Sleeps
MoEngage's published Eventstore account describes a workload requirement of at least 250,000 events per second, 1ms average latency, under 10ms p99, and more than 200TB of raw data. Those are company-reported design requirements, not a default target for every stream. Price always-on compute and storage against the product's freshness contract.
- Streaming buys freshness by the millisecond and pays in always-on compute. Price the awake hours before promising them.
- Write direction picks the hardware. Write firehoses want owned fast disks while read lakes want dense capacity.
- Deep dive: MoEngage Rent Nothing, the published 250K writes story with hardware receipts.
Streaming is justified when the business can name the value of fresher data. If users or downstream decisions do not need sub-minute updates, a batch schedule may meet the contract at lower operating cost.
Q1. When does a recommendation feed need streaming, and when is hourly batch enough?
Q2. Size a stream store for 250K writes a second at p99 under 10ms.
Freshness chosen. Now the pipe itself needs a budget. Lesson 03: Kafka budget →
Kafka Is a Budget With Partitions
Composite comparison: two pipelines process the same event stream. One mixes latency-sensitive matchmaking with slower analytics; the other gives each workload a separate lane. The separation changes queueing and recovery behavior, and adds an operational boundary that must be owned.
- Separate urgent from patient traffic or one surge starves both. Tiers are latency insurance with a topic name.
- Partitions buy concurrency and add operational load. Count them like money because brokers do.
- Deep dives: Roblox 18 trillion messages and Razorpay Kafka.
Run the workload through the RPS envelope calculator. Events per second multiplied by seconds per month turns partition and retention choices into explicit cost inputs. Separate latency classes before scaling either lane.
Q1. Design ingestion for one million events per second with urgent and patient readers.
Q2. How many partitions can you support before the broker overhead dominates? Defend a lower count.
Tiers set. Then a deploy restarts everything at once. Lesson 04: Rebalance tax →
Rebalances Are the Tax
Modeled classic-eager scenario: 120 members and 40,000 partitions, with a 660-second pause assumed as an input. The cascade condition is that the next membership change arrives before the prior rebalance completes. Neither the group size nor the partition count establishes that duration; this is not a first-hand incident account.
- In classic eager groups, staggering can prevent overlap only when the group settles between membership changes. Static membership may help with brief restarts before session timeout.
- Retries without backoff multiply the storm. Budget retries like capacity before the incident, not after.
- Deep dive: the modeled Kafka rebalance cascade and the retry storm calculator.
A pipeline review should include the consumer-group rebalance graph alongside lag and throughput. If no one can show the handoff duration, the team cannot estimate how a rollout affects freshness.
Q1. A modeled classic-eager example assumes a 660-second pause. Which protocol behavior creates a group-wide pause, and which measurements would you need before applying that duration to a real system?
Q2. Prove your retry policy cannot DDoS your own brokers.
Movement restored. Now the bytes themselves need rules. Lesson 05: Schema teeth →
Schemas Are Contracts With Teeth
ScyllaDB's 2024 ShareChat case study reports about 70 tile fetches per feature in the initial design and a reduction in required row reads from roughly 2 billion to 200 million per second after co-requested features were stored together. In ShareChat's 2025 engineering account, later tile and range changes reduce average fetches to about seven; a separate consistent-hashing step raises local cache hits from 80% to as high as 99%.
- Store what is requested together, together. Audit fan-in on the hottest request before touching hardware.
- Cache locality follows routing. Client-side entity routing and service splits carried a published 95 percent hit rate.
- Deep dive: ShareChat's 10x-cheaper feature-store teardown, which separates schema changes from later cache-locality gains.
The published case reports that the feature store grew from one million to one billion features per second without growing the underlying database. Measure fetch fan-in before adding capacity, and keep schema, tile and cache improvements as distinct steps.
Q1. Design a feature store for one billion reads a second at p99 under 20ms.
Q2. Your p99 climbs while throughput stays flat. Schema or hardware? Prove it.
Rows tamed. Now the warehouse starts billing by the scan. Lesson 06: Lakehouse bill →
The Lakehouse Bill
Notion's data-lake engineering post reports more than 200 billion block rows and 480 logical Postgres shards. It describes CDC through Kafka into Hudi on S3 for offline, search and AI workloads. The course diagram summarizes that disclosed architecture; traffic and cost figures elsewhere on this site are modeled separately.
- CDC turns one edit into six downstream jobs. Count the readers per write before adding another index.
- Partition by query pattern and compact small files so scans do not inflate the monthly bill.
- Deep dive: Notion's 200 Billion Blocks and the Warehouse Bill Behind a Blank Page, the 200B-block lake with modeled cost.
Partitioning should follow query patterns and retention needs. For a modeled warehouse workload, measure bytes scanned and compaction work before changing engines; a full scan is not automatically an error if the freshness contract permits it.
Q1. Design CDC from Postgres to a lake for 200B rows with hourly freshness.
Q2. Warehouse spend triples with flat traffic. Which three checks run first?
Scans priced. Now the disks underneath send their own invoice. Lesson 07: Slow invoice →
Storage Is the Slow Invoice
MoEngage's Eventstore case says its team evaluated memory-optimized instances with EBS, io2 and gp3 before choosing local NVMe for a write-heavy workload. The authors describe io2 as too expensive at their scale and gp3 as unable to guarantee their required p99 under bursts. Those are MoEngage's reported trade-offs, not a universal ranking of storage products.
- Match disks to dominant direction. Write firehoses want local NVMe while read lakes want dense capacity.
- Demand tail promises in writing. Averages soothe dashboards while tails page humans at night.
- Deep dive: MoEngage hardware math, 250K writes at 1ms average over 200TB.
Design rule: compare the measured read/write profile, tail-latency objective and full operating cost before choosing storage. Put the workload measurements beside the architecture diagram.
Q1. Pick storage for 250K writes a second with 1ms average. Defend the workload fit.
Q2. p99 spikes only during bursts on shared volumes. What changed?
Bytes housed. Now the compute above them needs discipline. Lesson 08: Flink discipline →
Flink Discipline
ShareChat's February 2025 account of its content-deduplication service reports that moving streaming jobs from managed Dataflow to self-run Flink cut their cost by about 93%. This was a separate workload from the feature store. A self-managed estimate also needs to include operations, upgrades, capacity headroom and recovery work.
- Prototype on managed, then reprice at steady state. Stable workloads and per-unit managed-service prices can diverge.
- Autoscale stream workers to actual load, not peak imagination. Idle parallelism is a subscription to nothing.
- Deep dive: ShareChat's feature-store case; its Flink cost report concerns a separate content-deduplication service.
Lesson 2 priced always-on compute. This lesson adds the operator cost. Reprice managed and self-run pipelines at steady state, including on-call coverage and recovery work. The cheapest engine is the one the team can operate reliably.
Q1. When do you exit managed streaming for self-run Flink? Show the math.
Q2. Flink lags only at peak hours. Autoscaling or backpressure?
Streams leaned. Then history asks for a rerun at noon. Lesson 09: Backfill bite →
Backfills Bite at Noon
Composite failure mode: a historical backfill shares a serving path with live reads, and the replay raises queueing delay for user traffic. Isolate or throttle backfills against the live service's latency objective; this is a design scenario, not a reported incident.
- Isolate replays from live reads with quotas and separate queues. History should wait; live traffic should not.
- Throttle by live p99, not by replay speed. Backfill progress is vanity when serving bleeds.
- Deep dive: Discord hot partitions, where replay pressure met quorum amplification.
Lesson 4 covered membership changes. This lesson covers time isolation: schedule or throttle replays so they do not consume the capacity reserved for live reads.
Q1. Replay one year of events without moving live p99. Sketch the lanes.
Q2. Backfill doubles serving latency. What guard trips first?
History contained. Now the present needs eyes. Lesson 10: Seeing pipelines →
Seeing the Pipeline
Composite monitoring failure: throughput remains healthy while the pipeline falls behind its freshness target. Without watermark age and lag alerts, the stale output may be noticed by users before operators.
- Alert on staleness users feel, not CPU you guess. Check lag and the freshness SLO before broker CPU.
- Read dead letters weekly. Every poisoned record is a bug report production filed for you.
- Deep dive: modeled rebalance-lag arithmetic and the sampling fitter for signal budgets.
Lesson 9 covered workload isolation. Monitor watermark age, dead-letter growth and freshness-SLO burn so a healthy throughput graph cannot hide stale output.
Q1. Freshness SLO burns at 2 AM with no deploy. Walk through it live.
Q2. Design pipeline dashboards before launch. Which three panels ship on day one?
Eyes open. Now finance walks into the review. Lesson 11: Cost review →
Cost Is a Pipeline Review
ScyllaDB's 2024 ShareChat case reports feature-store scale from 1 million to 1 billion features per second. ShareChat's 2025 engineering post says its feature-system costs fell tenfold through multiple optimizations, including stream-processing changes. A separate ShareChat account of its content-deduplication service reports a 90% cost reduction and p99 latency falling from 30–40ms to 6–8ms. These are separate workloads and measurements.
- Price one event before scaling to billions. Try the RPS envelope and downtime cost calculators.
- ShareChat's feature-store account attributes more than a 50% compute reduction to continuous-profiling-guided optimizations.
- Deep dives: ShareChat's feature-store stages and MoEngage's event-store rebuild.
Cost reviews are more useful when the owner sets a target before scale makes the existing design expensive. Compare each change against its own workload and measurement period; do not combine outcomes from separate services into one performance claim.
Q1. Price this pipeline monthly, line by line, out loud.
Q2. Cut the bill 40 percent without touching freshness.
You can now read any pipeline bill. Time to meet skew up close. Lesson 12: Hot keys →
Hot Keys Burn Loud
Discord's engineering post describes hot message partitions, quorum-read latency and a migration from 177 Cassandra nodes to 72 ScyllaDB nodes with 15ms p99 reads. The 5,000-reads/s hot-channel input in this course is a modeled example, not a Discord measurement. Use per-partition load and queueing data before adding nodes.
- Design schemas for reads first when reads dominate. Write-optimized layouts tax every popular key forever.
- Coalesce fan-in before adding nodes. One request answered once beats a thousand duplicate reads.
- Deep dive: Discord Trillions of Messages, 177 nodes down to 72 at 15ms p99.
Quorum reads add replica work, so a hot key can saturate a small set of nodes even while the wider fleet has spare capacity. Measure the hottest partitions and coalesce duplicate reads before scaling the whole cluster.
Q1. One channel draws 5K reads a second to a single partition. Fix it live.
Q2. Reads at quorum spread a hot key to neighbors. Prove it with math.
Skew tamed. Now the ceiling nobody read about. Lesson 13: Connection ceiling →
The 3,000-Connection Ceiling
Razorpay Engineering's retrospective describes a 10,000-TPS design target and an MSK IAM limit of roughly 3,000 active connections per broker. The same post marks its before-and-after client connection and latency figures [VERIFY]; those deltas are not treated here as confirmed measurements.
- Read managed caps before surge day. Serverless removes provisioning, never limits.
- Measure connections per broker, DNS re-resolution and publish latency when evaluating a client change. Razorpay's source marks its exact before-and-after deltas
[VERIFY]. - Deep dive: Razorpay Kafka, five years from SQS to provisioned with receipts.
For every managed service, record the connection, partition, throughput and authentication limits in a runbook. Exercise the surge plan against those limits instead of treating autoscaling as unlimited capacity.
Q1. Design UPI payments for 10K TPS with a surge plan. Name every cap.
Q2. A source marks before-and-after latency values [VERIFY]. What production measurements would you need before attributing a change to a client swap?
Caps mapped. Now turn all of it into interview answers. Lesson 14: Interview pipeline →
The Interview Pipeline
A useful data-system answer states the freshness contract, sizes the workload and names the failure modes. The lesson turns the preceding examples into a repeatable interview structure.
- Open with freshness. Batch by morning, stream by minute, or tiered by reader urgency, stated in the first minute.
- Price out loud with the RPS envelope calculator. Panels score the math more than the engine names.
- Close with one cited failure analysis and its fix. If you use a hypothetical, label its assumptions rather than presenting it as personal experience.
Rehearse three answers: a nightly reporting design, stream sizing with hardware assumptions, and a cost reduction that preserves freshness. State row counts, partition counts and dollar costs. Explain where each number comes from.
Q1. Design clickstream analytics for one billion events a day. Price it monthly.
Q2. Describe a pipeline failure and its fix. What changed afterward?
Answers rehearsed. Now price a pipeline of your own. Lesson 15: Price yours →
Price Your Own Pipeline
Choose a pipeline you can discuss without exposing private or customer data: logs, billing events, search signals or recommendations. Give it workload lanes, a schema, a lake plan and a clearly labeled cost model.
- State freshness first. Batch by morning, stream by minute, or tiered by reader urgency.
- Name the partition key, fat-row grouping and lake layout on one page. Vague plans bill vaguely.
- Close with the invoice: monthly events multiplied by a modeled unit price, and storage growth. Label estimates.
Course complete. Continue with the case studies and the system design course.
For an interview, use a failure analysis you can support with evidence, or label a modeled scenario and its assumptions.
Sources and Method
Lessons link to published engineering accounts and Buildopsy analyses. Composite and modeled examples are labeled; company-reported workload figures are attributed to their sources. Cost examples use stated assumptions and public list prices.
- ShareChat: How ShareChat built a scalable, cost-efficient ML feature system (Mar. 6, 2025)
- ScyllaDB: ShareChat feature-store scaling account (Aug. 27, 2024)
- ShareChat Engineering: Cost-efficient candidate deduplication in RecSys (Feb. 18, 2025)
- ScyllaDB: ShareChat content-deduplication service cost account (Mar. 2, 2026)
- MoEngage authors via ScyllaDB: Eventstore workload and hardware choices (Jan. 16, 2026)
- Notion Engineering: Building and scaling Notion's data lake (Jul. 1, 2024)
- Discord Engineering: How Discord Stores Trillions of Messages (Mar. 6, 2023)
- Razorpay Engineering: Five Years of Kafka at Razorpay's UPI Switch (Sep. 7, 2026)