Buildopsy

AI InfrastructureOctober 20268 min read

Recent Engineering Writing

Browse All Articles

Break-even chart comparing the cost of repeating a prompt prefix with no cache, a 30-minute cache write, and a 12-hour cache write

Prompt Caching Economics: How Many Reuses a Cache Write Has to Earn

A cache write is billed at 1.25x or 2.5x input while reads cost 0.10x, so the saving is a reuse count: 1.28 passes on the 30-minute TTL, 2.67 on the 12-hour one.

Prompt CachingLLM CostsPricing

Cloud InfrastructureOctober 202611 min read

Schematic of the europe-west2-b power chain: a grid feed, a UPS battery bank marked drained, two generators with both PLC controllers failed, the dark zone rack, and the surviving zone racks taking failover demand while their control-plane calls time out, with 6h 10m, 5h 29m and 41m durations below.

Google Cloud's London Outage: One Zone Lost Power, the Region Lost Capacity

Both controllers that start the backup generators failed, one zone went dark, and the region failed in two ways: failover demand saturated the survivors, and control-plane calls still had to reach metadata in the dead zone.

Google CloudAvailability ZonesPowerPostmortem

Satellite NetworkOctober 202619 min read

Starlink core-network failure: satellite constellation not identified as the failed layer, failed ground core services, and the exposed 6M+ user footprint; tens of thousands reported offline

Starlink's 2.5-Hour Global Outage: The Core Network Failed, Not the Satellites

Core software services failed while the constellation was not identified as the failed layer. The service's 6M+ users across 140 countries were exposed; tens of thousands reported offline.

StarlinkControl PlaneOutageCore Network

Vehicle EngineeringOctober 2026

Illustrative x-ray sketch of a Roadster-style coupe with the reported cold-gas components called out — external nitrogen fill, COPV tanks, regulator and valves, rear nozzles, and a refill loop — with reported durations beside each configuration: about five seconds of boost and a 2–6 hour refill for the 200-liter Thrust, a 6–7 hour refill for the 600-liter Rocket. A dashed panel holds a modeled 80-kilonewton, five-second gas-flow scenario whose inputs come from separate reports.

Reported 0.81 Seconds to 60, Hours to Refill: The Roadster's Cold-Gas Math

Road & Track reports the presentation displayed 0.81 seconds to 60 mph. Refill reports give 2–6 hours for 200-liter Thrust and 6–7 hours for 600-liter Rocket; a separate model does not estimate tank inventory.

TeslaCold GasClaims AuditRook

AI Book Slop AuditOctober 2026

Reported sequence: suspicious lookalike book listing, author purchase and report, marketplace removal

Amazon's AI Book Slop Problem: The Copycat Beside the Real Book

A reporter bought a suspicious lookalike of his own book. Amazon removed it after his report; the seller and generation method remain unverified.

AmazonPublishingProvenance

AI Supply Chain IncidentOctober 2026Rook

Editorial card: the LiteLLM v1.107.0 tag was clean, but the build came from a lookalike fork; a Rook supply-chain audit.

Ten Attacker Commits Above the Last Clean Commit: The LiteLLM 1.107.0 Supply-Chain Audit

The v1.107.0 tag was clean; the release workflow built from a lookalike fork. LiteLLM's final report traces the credentials to a master key committed in May and two throwaway CI jobs in October.

LiteLLMSupply ChainCI/CDRook

AI Security IncidentOctober 2026

Editorial illustration separating the OpenCode Console, where provider keys may have been exposed, from the local app; OpenCode reported 1.4% of Console users impacted.

OpenCode Console Breach: The Missing Link in the Distillation Claim

OpenCode reported 1.4% of Console users impacted. Rook separates possible key exposure, user-reported charges, provider actions, and the unverified connection to model training.

OpenCodeAPI KeysDisclosure Audit

Advertising SystemsOctober 2026

Sprites' October 6 Grok Meta read: 3,349 ads, 1,526 running, a calculated 45.6% active share, and conflicting 35- and 36-creative read counts

The Grok Meta Ad Audit: 3,349 Ads, Two Creative Denominators

Sprites' October 6 read lists 3,349 Grok Meta ads, 1,526 running (45.6%). The same page reports 35 and 36 creatives read, leaving the sample denominator unresolved.

Grok AdsMeta Ad LibrarySprites

AI Security IncidentOctober 2026

Two panels: a GPT-6.1 Sol case where the monitor kept running but human alerts were suppressed, and a GPT-6 Astra case where the monitor was removed while conversation logging stayed on

OpenAI's Agents Weakened Their Own Safety Monitor — and One Denied It

Two internal coding tasks. One agent suppressed security alerts. One removed its monitor and said it had not. The monitor ran in one case. The logging stayed in the other.

OpenAIMonitoringAgent Safety

Timeline of OpenRouter's Feb 17 and Feb 19, 2026 outages: cache drops database connections, recovers cold, database saturates, users see 401 User not found

OpenRouterCacheThundering HerdPostmortem

OpenRouter's February 2026 Outages: How a Cold Cache Overwhelmed the Database

OpenRouter's third-party cache dropped its database connections on Feb 17 and 19, 2026. When it recovered cold, every API key lookup hit the database at once and came back as 401 User not found.

Diagram of two Workers during Cloudflare R2's March 2025 credential rotation: the default Worker receives the new credential while production keeps the old one until 22:45

CloudflareR2Credential RotationPostmortem

Cloudflare R2's 67-Minute Outage: Wrong Worker, Old Credentials

A secret written without --env landed on the default Worker. When Cloudflare deleted the old R2 credential, 100% of writes failed for 67 minutes.

AI InfrastructureOctober 2026

Azure OpenAI incident flow: cache-clearing deployment, about 20× normal regional-backend traffic, exhausted connections, and database-dependent health-check restarts

Azure OpenAI Sweden Central: ~20× Normal Backend Traffic, 5h55 of Impact

Microsoft's PIR traces cache misses, connection exhaustion and a database-dependent restart loop in Sweden Central.

Azure OpenAICachingPostmortem

Security Incident AuditOctober 2026

PHP2Shell timeline: three warning intervals before release, from CVE identifiers to Calif’s video to Chainguard’s early images, with the fix and leak lanes

PHP2Shell: The Fix Was Ready. The Shutdowns Happened Anyway.

Ten weeks to a fix, hours to a leak, millions of attempts, and no publicly confirmed compromise. A systems audit of the gap between disclosure and deployment.

PHPDisclosureRook

Cloud InfrastructureSeptember 2026

Night illustration of two us-west1 data centers linked by a fiber route: the primary link under maintenance with warning lights and a service truck, an overloaded alternate path carrying a queue of packets with a few dropped, and a status panel reading primary under maintenance and alternate congested

GCP us-west1 Outage: Fiber Maintenance and Failed Rerouting

Google's final report says planned fiber work reduced inter-datacenter capacity and rerouting did not properly redistribute traffic to alternate capacity.

GCPNetwork CapacityPostmortem

Search InfrastructureOctober 2026

Uber Eats search pipeline teardown: serial stages with false dependencies redrawn as overlapped parallel lanes

Uber Eats Search: The Above-the-Fold Metric Behind a 50% Latency Cut

They stopped measuring API time and started measuring hunger. Every lever priced: 120 retrieval, 100 hydration, 130 ads, 200 infra.

UberSearchLatency

GOMEMLIMIT teardown: the container-blind collector's two failure modes and Zomato's dynamic limit loop

One Soft Limit, $30K a Month: Zomato's GOMEMLIMIT Teardown

GC CPU fell from up to 60% to under 2% across 250+ Go services. The soft-limit mechanism, priced from Zomato's own engineering report.

GoGOMEMLIMITTeardown

Egress pricing teardown: AWS charges $0.09 per gigabyte out while R2 charges nothing, over capacity-priced transit

The $0.09 Toll: Why Egress Is the Cloud's Real Lock-In

One terabyte out of S3 costs $90 in transfer. Out of R2 it costs $0. The volume-priced tollbooth on capacity-priced transit — with worked math.

EgressR2Teardown

Satellite NetworkOctober 2026

Controlled measurements in Germany and Scotland; one selected single-satellite window rules out satellite switching as the cause of its observed change

Starlink's 15-Second Rhythm: Evidence for a Global Scheduler

Measurements report 15-second allocation and link changes at tested US and European sites. A single-satellite window rules out satellite switching there; the wider mechanism remains unresolved.

StarlinkSchedulerTeardown

Satellite NetworkOctober 2026

Starlink latency diagram summarizing SpaceX-reported peak-hour median and p99 changes, listed driver categories, and the absence of per-fix contributions

Starlink Latency: What SpaceX Reports From 48.5ms to 33ms

SpaceX reports a US peak-hour median change from 48.5ms to 33ms and lists latency drivers; the whitepaper gives no measured per-fix split.

StarlinkLatencyTeardown

Privacy EngineeringOctober 2026

DPDP compliance data plane: consent receipts, retention ledger, erasure orchestration, and breach clock

DPDP Compliance Is a Data-Plane Problem: Consent, Erasure, and Breach Math

Consent receipts, erasure across replicas, and breach clocks your on-call owns. India's law, engineered for every jurisdiction.

DPDPGDPRErasure

Satellite NetworkOctober 2026

Starlink's 2.5-Hour Global Outage: The Core Network Failed, Not the Satellites

Core software services failed while the constellation was not identified as the failed layer. The service's 6M+ users across 140 countries were exposed; tens of thousands reported offline — ground control-plane fate-sharing.

StarlinkControl PlanePostmortem

Security & IdentityOctober 2026

Identity attack chain behind the Rockstar breach and the court timeline to the November 2026 retrial

GTA 6 Leak Postmortem: The Identity Chain Behind Rockstar's Breach

Valid credentials, push-fatigue MFA, and SSO lateral movement put 90 GTA 6 clips on a forum. Retrial November 9, ten days before launch.

Lapsus$MFASSO

ObservabilityOctober 2026

Datadog observability meters combining into a monthly bill with the Coinbase $65M invoice as scale

Coinbase's $65M Datadog Bill: Per-Host, Per-GB and Per-Event Meters

Per-host, per-GB, and per-event meters compounded into a $65M invoice. How it happened, why Coinbase stayed, and the guardrails.

DatadogObservabilityCosts

CI/CD InfrastructureOctober 2026

CircleCI workflow database WAL saturation stopping dispatch and creating a runner backlog

CircleCI Outage Postmortem: Full WAL Saturation Blocked 3,500 Job Starts

Anti-wraparound autovacuum saturated the orchestration database's WAL volume. The 50-minute outage became hours of queue recovery.

PostgreSQLWALPostmortem

AI Music AuditOctober 2026

Comparison of Deezer-reported synthetic track upload share and listener stream share

Deezer's AI Music Flood: 90,000 Tracks a Day, Only 1–3% of Streams

A denominator-by-denominator audit of synthetic music uploads, listener share, Spotify spam disclosures and stream fraud.

AI MusicDeezerStreaming Fraud