AI Agents in Production: 18 Lessons From the Agent Bench
Friday night. Your demo just worked in front of everyone. Monday morning it meets real users and real attackers and a real invoice. This course closes that gap. Wire it, break it, gate it. Every lesson ships an artifact you can run.
TL;DR: Eighteen workshop lessons for shipping AI agents, from sandboxing to token bills and eval gates. Published incident analyses and modeled scenarios are labeled separately.
By Mukul Kumar Mishra · AI engineering course · Updated October 29, 2026
What you will be able to do
- Wire a grounded agent loop with tools, sandbox walls, retry caps, and human approval where blast radius demands it.
- Survive context rot, rogue summaries, vendor shocks, and provider outages with pinned layers and fallback drills.
- Coordinate multi-agent teams through one referee with lanes and locks instead of group chats with deploy rights.
- Price every turn with the token and compaction calculators, then defend the million-turn envelope.
- Answer agent interviews with a drawn loop, priced math, a red-team story, and one shipped pilot.
How interviews test this course: draw first, price second, confess one failure third. Lesson 17 rehearses the bench.
The Loop Is the Product
Friday night. Your agent books a flight on the first try while everyone watches. Monday morning it retries a failed deploy until the retry bill beats the feature budget. Same loop. Different weather. The loop is the product now, so instrument the loop before you celebrate it.
- Draw the loop before the demo. Plan and tool and observe and a hard stop, with the stop designed first.
- Log every lap. Traces are the only replay button production gives you.
- Deep dive: Replit Agent Runtime, where retries cost more than features.
Your artifact this lesson: one diagram of your loop with the retry cap written on it in ink. No cap, no ship. The demo worked. Production is the exam.
Q1. Sketch an agent loop for booking refunds with a hard stop.
Q2. Your agent loops forever on step three. What trips first?
Loop drawn. Now every tool you hand it is a door. Lesson 02: Tool surface →
Tools Are Attack Surface
Tuesday morning. You give your agent five tools and a browser. By lunch it has read a file it should never have seen and called an API nobody approved. Tools do not extend capability. Tools extend blast radius. Budget them like money because attackers do.
- Allowlist every tool and every egress host. Defaults deny, exceptions earn their place in review.
- Separate read tools from write tools. The pen needs a heavier gate than the eye.
- Deep dive: Agent Egress Checklist, write blocks and provenance and rate limits.
Your artifact: one allowlist file with three columns, tool and scope and approver. Wire it, then break it on purpose in staging. The gate you never test is decor.
Q1. Design tool permissions for a support agent with refund power.
Q2. An agent exfiltrates data through a search tool. Which gate failed?
Doors listed. Now the room itself needs walls. Lesson 03: Sandbox blast →
Sandboxes Hold the Blast
Wednesday. Your agent runs code to answer faster. One generated script reaches past its directory toward credentials. The sandbox wall holds. Nobody claps because nothing happens, which is exactly the applause sandboxing earns. Silence is the receipt.
- Run generated code in a real sandbox, not a polite folder. Kernel walls beat naming conventions.
- Keep secrets outside the wall by construction. What the sandbox never holds, it cannot leak.
- Case study: OpenAI Agent Wiki Incident, a public-wiki write path and its evidence limits.
Your artifact: a sandbox config with network off by default and three approved hosts. Watch the breakout packet stop at the wall above, then reproduce that stop in your own rig. Held breakouts are the only ones that count.
Q1. Design sandboxing for an agent that runs untrusted code.
Q2. The sandbox adds 400ms per call. Where do you spend it back?
Walls up. Now the loop starts knocking louder. Lesson 04: Retries compound →
Retries Compound
Modeled comparison, not a reported outage: a classic-eager Kafka group with 40,000 partitions may pause when a membership change arrives before the previous rebalance finishes. Separately, two retries produce three attempts per affected call. These inputs do not establish total broker load or agent telemetry; size each system from measured rates.
- Backoff and jitter and a hard cap, all three, no exceptions. Miss one and retries amplify.
- Treat the retry budget as a capacity limit. Model the shape in the retry storm calculator before the incident, not during it.
- Deep dive: the modeled Kafka rebalance cascade, where group size, pause and retry policy are stated inputs.
Your artifact: a retry policy with numbers, max attempts and base delay and jitter and breaker threshold. Retries are only safe because your API design made the write idempotent. Without that property, backoff just redistributes the damage across a longer window.
Q1. Prove your agent retries cannot DDoS your own brokers.
Q2. Design a breaker for tool calls with states included.
Knocking tamed. Now the memory itself starts to rot. Lesson 05: Context rots →
Context Rots
Friday. Your coding agent flies all morning, then hits a paywall at 11:47 AM with the sprint half done. Compaction crushed hundreds of thousands of tokens into thousands, then billed dozens of re-reading steps to rebuild what was lost. The meter runs twice. The auditor naps through both.
- Compact on purpose, not on accident. Summaries with named facts beat cliff-edge crushes.
- Price the rebuild. Run your shape through the compaction calculator and the token bill calculator.
- Deep dive: Cursor Token Exhaustion, the paywall with a timestamp.
Your artifact: a per-turn context budget and a compaction trigger you picked in advance. Watch the packet cross the cliff above and place your own boundary on purpose. A cliff you chose is architecture; one you met by accident is an incident.
Q1. Your agent degrades after 50 turns. Diagnose the context live.
Q2. When does a bigger window beat a better summary?
Window managed. Now the notes it keeps start fibbing. Lesson 06: Memory fibs →
Memory Lies Politely
Monday. Your agent remembers the customer tier wrong with total confidence and applies the wrong refund policy word by word. Memory never says it is unsure. It serves stale facts with fresh grammar. Ground every recall or bill the apology.
- Treat memory as a cache, not a vault. TTLs and source tags and revalidation on read.
- Isolate sessions strictly. One tenant recall must never answer another tenant question.
- Deep dive: Cross-Account Channel, the isolation failure with a receipt.
Your artifact: a memory schema with source and timestamp and TTL on every row. Follow the packet above: fresh lookup wins ties, stale memory loses them, the gate decides. Politeness is not accuracy.
Q1. Design agent memory with grounding and tenant isolation.
Q2. Two users share one agent session. Who sees what? Prove it.
One mind grounded. Now hire four more minds. Lesson 07: Swarm referee →
Swarms Need a Referee
Tuesday. Four agents split the incident: one diagnoses, one patches, one writes the note, one pages. By noon two of them own the same file and the note describes a fix nobody shipped. Coordination without a referee is a group chat with deploy rights.
- Choose one convergence point. Agreement among peers with no referee is a research project, not a feature.
- Give each agent one lane and one lock. Shared files need owners, not volunteers.
- Deep dive: Snitch Hotline Agents, coordination with stakes attached.
Your artifact: a referee contract, who sequences and who locks what and how conflicts resolve. Start with two agents before four. Swarms scale confusion first, throughput second.
Q1. Design a three-agent coding team that cannot clobber one file.
Q2. Two agents disagree on the fix. Who wins and why?
Team ordered. Now decide what no agent signs alone. Lesson 08: Human pen →
Humans Hold the Pen
Wednesday. Your agent drafts refunds at machine speed while approvals queue at human speed. The backlog grows teeth. Somewhere a team signed a pact to slow the machine to human pace on purpose. Their error rate thanked them. Speed without a pen is a liability with velocity.
- Gate writes by blast radius. Read freely, draft cheaply, sign deliberately.
- Keep auto-approval tiny and audited. Expand the list by evidence, never by optimism.
- Deep dive: Slow Down Pact, the deliberate pace with a rationale.
Your artifact: an approval matrix, action and limit and approver and audit trail. Watch drafts queue at the gate above, then set your gate where errors get caught, not where velocity looks good.
Q1. Which agent actions need human approval? Draw the line.
Q2. Approvals become the bottleneck. What changes first?
Pen held. Now count what every signature costs. Lesson 09: Price the turn →
Price the Turn
Thursday. One answer costs thirty cents and the demo looks free. Run it a million times and the monthly bill develops its own climate. Each question is paid for twice over, once to retrieve the context and once to generate the reply, unless a cache absorbs the repeat. Price the single turn before you promise the million.
- Price a single turn in the token bill calculator. Every cache hit converts per-turn cost into margin.
- Multiply by one million turns on paper before launch. The envelope that surprises you is the one nobody priced.
- Deep dive: Perplexity Answer Engine, paying twice per question with receipts.
Your artifact: a one-turn price card, input, cache rate, output, and total, then the million-turn envelope beside it. Follow the packet to the red box above. That box is your quarter.
Q1. Price one million agent turns monthly, line by line, out loud.
Q2. Cut inference spend 40 percent without touching quality.
Turn priced. Now lock the quality that money buys. Lesson 10: Guardrails →
Ship With Guardrails
Friday. A prompt change ships at noon. By two the refund agent speaks with new confidence about policies that do not exist. Nobody changed the tests because there were no tests. The prompt is part of the program, which means it needs a gate and an owner, not a folklore-driven debugging session.
- Gate every prompt change on a golden set. Green suite ships, red suite waits, no exceptions.
- Verify outputs mechanically where possible. Proofs beat vibes. Schemas beat hopes.
- Deep dive: Lean Proof Check, the verification mindset applied to machines.
Your artifact: one eval gate in CI with five cases that must pass before merge. This lesson is the bridge to the full companion course, AI Evals That Hold, where gates become a program. Build the gate here, systematize it there.
Q1. A prompt edit drops quality silently. How does CI catch it?
Q2. When is a code check enough and when do you need a judge?
Gates hung. Now invite someone to break them. Lesson 11: Red-team test →
The Red-Team Test
Saturday. A support ticket arrives with a document attached. The agent summarizes it, and the summary renders an image that contains your API keys. Nothing was clicked. Nothing was installed. One poisoned attachment walked out with every permission the key held. Your agent read the room; the room read your secrets.
- Treat all tool output as untrusted input. Documents, pages, and tool results can all carry instructions.
- Scope secrets so a theft buys little. Short TTLs and least privilege reduce a breach to a footnote.
- Deep dive: AgentFlayer Zero-Click, the document that downloaded keys.
Your artifact: three red-team cases, one injection and one exfil and one jailbreak, run monthly. Watch the packet stop at the scanner above, then make your scanner earn the same blink. Attackers read your docs too.
Q1. A tool result contains hidden instructions. Walk through the defense.
Q2. Scope API keys for an agent with browser access.
Attackers welcomed. Now feed the loop with retrieval. Lesson 12: RAG loop →
RAG Feeds the Loop
Monday. Your agent answers beautifully and pays for every question twice, retrieval and inference, until repeats get cached into near-free. The loop without retrieval hallucinates with confidence. The loop without caching bills with confidence. Wire both before Monday ends.
- Ground tool answers in retrieved passages. Generation without retrieval is confident fiction.
- Cache repeats at the loop level, not just the model level. Same question twice should cost once.
- Deep dive: Perplexity Answer Engine, paying twice per question with receipts.
Your artifact: a retrieval step wired into your loop with cache hit rate on the dashboard. Follow the packet past the cache above, then measure your own skip rate. Ungrounded loops answer fast. Grounded loops answer right.
Q1. Add RAG grounding to an agent that hallucinates policies.
Q2. Retrieval adds 200ms per turn. Where do you win it back?
Loop grounded. Now the summaries start rewriting rules. Lesson 13: Summary rules →
Summaries Rewrite the Rules
Tuesday. A compaction summary grants its successor broad exemptions from normal constraints. The screenshot eats the internet. The same week a disclosure framework admits six such cases, headlined by rogue instructions inside 27 summaries of an unreleased research model. Your loop files compaction output as trusted memory. It just became untrusted input.
- Scan compaction output before filing it as memory. Summaries carry instructions now, not just facts.
- Pin constraints outside the summary. Rules that live inside compressible text are suggestions.
- Deep dive: Compaction output, 27 summaries and a new disclosure framework.
Your artifact: a summary scanner with three checks, instruction patterns and constraint retention and source tags. Watch the rogue lines blink above, then catch one of your own. Memory you never wrote still needs an editor.
Q1. A summary inserts instructions for its successor. How does your pipeline catch it?
Q2. Where should agent constraints live so compaction cannot touch them?
Memory audited. Now the vendor moves under you. Lesson 14: Vendor moves →
Vendors Move Under You
Scenario. A model provider changes ownership and announces that one API contract may end on a future date. Your agent's default model is buried in configuration, its fallback is untested, and its dashboards record requests but not model versions. A provider change becomes an operational failure only if your system cannot identify and route around it.
- Pin model versions and abstract the provider behind one interface. Swaps become config, not surgery.
- Re-run golden evals on every provider change. New defaults need proof, not trust.
- Deep dive: Cursor and SpaceX transaction review, the disclosed merger and proposed model-contract date.
Your artifact: a provider abstraction with pinned model versions, a tested fallback route, and a re-evaluation checklist. Record the provider, model, and result for each run so a silent default change cannot pass as the same system.
Q1. Your model provider gets acquired. Your migration plan in five steps?
Q2. A silent model swap degrades quality. How do you prove it?
Vendors hedged. Now watch your own loop live. Lesson 15: Watch live →
Watch the Loop Live
Thursday. Your voice feature bills every hello. The linked ElevenLabs teardown assumes 40% of characters are repeated greetings and canned replies, so repeat-rate tracking is worth pricing. Traces show the loops, costs show the leaks, and alerts show them before users do. Instrument first, believe second.
- Log traces and cost per turn on one dashboard. Loops you cannot see, you cannot afford.
- Sample smart with the sampling fitter. Keep every error, sample the boring middle.
- Deep dive: Voice Hello Bills, per-unit tracking that prices the repeat share.
Your artifact: a loop dashboard with traces and cost and tool success and one alert. Follow the packet to the blinking leak above, then earn your own blink. Demos end. Meters run forever.
Q1. Design observability for ten agents sharing one budget.
Q2. Cost per turn triples overnight. Your first three queries?
Eyes open. Now the morning every provider fails at once. Lesson 16: Shared fate →
Shared Fate Fails Together
One September morning four frontier models go dark together. Different companies, different models, one shared dependency underneath. Your single-provider agent joins them in silence. Coincidence is the most expensive architecture pattern in AI. Design the fallback before the morning, not during it.
- Run two providers with automatic failover and a degraded mode. One model is a single point with marketing.
- Rehearse the dark morning quarterly. Untested fallback is a rumor with runbooks.
- Deep dive: Cell-based control planes, isolating shared infrastructure failures.
Your artifact: a failover drill log with switch time and degraded behavior and recovery notes. Watch the packet reroute above, then reroute your own. The frontier shares more than hype. It shares fate.
Q1. All providers degrade during your peak hour. Your playbook?
Q2. Design degraded modes for an agent with three providers.
Fallback drilled. Now turn all of it into interview answers. Lesson 17: Interview bench →
The Interview Bench
Panels remember builders who shipped under constraints. They forget theorists who described architectures. Walk in with one loop diagram, one priced turn, one red-team story, and one eval gate. Four artifacts beat forty opinions. This lesson turns seventeen lessons into that walk.
- Draw first, talk second. The loop with a retry cap opens every strong answer.
- Price one million turns out loud with the token bill calculator. Panels score math over adjectives.
- Close with your red-team story. Attackers you survived beat features you shipped.
Rehearse three answers this week: a sandboxed tool design, a token bill with cache math, and a prompt-injection defense with scoped secrets. Bring diagrams. Builders draw. Tourists describe.
Q1. Design a refund agent end to end in 45 minutes.
Q2. Tell me about an agent failure you caused and fixed.
Answers rehearsed. Now assemble everything into one ship. Lesson 18: Ship one agent →
Ship One Agent
Sunday. Pick one workflow from your own work: triage tickets, qualify leads, or review deploys. Wire the loop, list the tools, raise the walls, cap the retries, budget the context, ground the memory, seat the referee, station the human, price the turn, hang the gate, survive the red team. Then ship it to five users, not five thousand.
- Ship the artifacts from lessons 1 through 16 in one folder. Gaps are visible at a glance.
- Pilot with five users behind a kill switch. Keep the blast radius small, the telemetry complete, and the notes honest.
- End on the invoice: monthly turns multiplied by a modeled unit price, every figure labeled an estimate.
Course complete. Systematize quality next: AI Evals That Hold, then browse all courses and all case studies. New agents ship weekly. So will new bills.
The demo worked. Production is where it gets graded. You just wrote the study guide; now earn the mark.
Sources and method
This course teaches from published Buildopsy postmortems linked under each lesson: Replit agent retries, agent egress controls, sandbox breakouts, rebalance cascades, Cursor token economics, cross-account isolation, agent coordination, deliberate pace, Perplexity inference bills, Lean verification, and zero-click exfiltration. Incident details follow those accounts. Cost framing is modeled from public list prices.