OpenRouter's February 2026 Outages: How a Cold Cache Overwhelmed the Database

1. The normal dependency, and what is known about it

Primary source: OpenRouter's engineering postmortem, published February 20, 2026 and updated June 24, 2026. The facts below are quoted or paraphrased from that document.

OpenRouter's authentication path depends on an external caching layer for fast API key lookups. Under normal operation, the cache serves most lookups and only a small fraction reach the database directly. That makes the database's ability to absorb a sudden increase in lookup traffic a critical part of the system's failure behavior.

That design is common and usually sound. A lookup served from cache avoids a database round trip, and at API-gateway scale that difference is large. The cost is that the fallback now has to handle the full request rate if the cache empties. The postmortem says the cache served most API-key lookups during normal operation. After its entries were invalidated, incoming requests required fresh database lookups, and the database could not handle the resulting volume. The postmortem does not disclose the normal cache-hit ratio, the database's share of authentication traffic, lookup rates, connection counts, or saturation thresholds. The quantitative relationship below is therefore illustrative, not a reconstruction of OpenRouter's actual load.

2. State transition and load amplification

The sequence shows how a recoverable dependency failure became a much broader outage during recovery.

The report establishes that the database could not handle the sudden lookup volume after the cache recovered with its entries invalidated. It does not disclose whether the limiting factor was connection capacity, query throughput, lock contention, or another resource constraint, and it publishes no load curve, sustainable-throughput figure, or recovery-phase database measurements. The actual increase in load and the database's remaining headroom therefore remain unknown.

A labeled hypothetical model shows the amplification. Let R be incoming authentication lookups per second, assumed constant, h the cache hit ratio between 0 and 1, and D the database lookups per second, assuming one database lookup per cache miss. Under normal operation, D = R(1 - h). If the cache hit ratio falls to zero after complete invalidation, D = R. For a hypothetical system with 10,000 lookups per second and a 99% hit ratio, that is 100 database lookups per second under normal conditions versus 10,000 after the cache empties, a 100-fold increase.

These are hypothetical inputs, not OpenRouter's actual telemetry. The model assumes the incoming lookup rate stays constant and that every cache miss produces exactly one database lookup. It deliberately excludes concurrent retries, multiple lookups per request, request coalescing, queueing, and database saturation effects. It demonstrates potential amplification, not the actual incident's load. A cache hit ratio alone cannot tell you where the database's sustainable throughput threshold lies; that requires capacity and latency measurements OpenRouter did not publish.

3. Timeline: two outages, one root cause

All times are UTC. The two incidents share a pattern: occasional errors from the caching layer before customer-visible failures, followed by a loss of database connections, recovery with an empty cache, database overload, and restoration after the cache repopulated. The difference is what surrounded the incidents.

February 17

February 19

Impact summary from the postmortem: on February 17, approximately 20% of API requests failed between 5:27 and 5:40 UTC, followed by 80-90% failure rates between 5:40 and 6:05 UTC. February 19 followed a similar pattern, with partial failures between 7:36 and 7:42 UTC and near-total downtime between 7:42 and 8:11 UTC. The chatroom was also degraded during both periods. OpenRouter does not publish request totals, customer counts, or affected-account numbers, so none are inferred here.

4. Error semantics: what 401 and 503 actually communicate

OpenRouter reports that database lookup timeouts were surfaced to customers as 401 "User not found" errors. The postmortem does not describe the internal exception-handling path that mapped those timeouts to an authentication response. The important documented fact is that infrastructure failures were exposed as authentication failures.

The practical cost landed on customers. Some spent time debugging their own API key configurations when nothing on their side was wrong. That is the signature of a mislabeled failure: the error points at the caller, so the caller investigates, while the actual fault is in the provider's infrastructure.

The two status codes communicate different failure classes. A 401 tells the client that the request lacks valid authentication credentials; a 503 signals that the service is temporarily unable to handle the request. During an authorization-lookup timeout, the service has not established that the caller's key is invalid, so returning 401 misrepresents the failure. A client may prompt for a replacement key or trigger authentication troubleshooting after a 401, whereas a 503 can direct it toward a bounded retry policy with backoff and jitter; neither response guarantees a particular client action, because retry behavior depends on the client implementation, and retries must be bounded so they do not amplify load on an already struggling service.

Error codes are part of the contract between a service and its clients. When infrastructure failures are reported as authentication failures, the status-code metric may show an increase in 401 responses without identifying the underlying database or cache failure. Distinguishing invalid credentials from infrastructure-related lookup failures requires metrics that capture the cause of the failed authorization lookup, not just its HTTP status.

The two remediations solve different problems. Circuit breakers and controlled cache repopulation address the overload mechanism; accurate error codes help clients and operators recognize the failure correctly. The error-code change improves diagnosis and client behavior, but it does not, by itself, restore database capacity or prevent the original overload.

5. Misattribution: the DoS that was not the cause

The first outage had a convenient suspect. A denial-of-service attack was ramping up at the same time the caching layer failed on February 17. Because DoS attacks can cause cascading problems across the stack, and because the caching dependency had historically been extremely reliable, OpenRouter initially attributed the outage to the DoS and prioritized hardening against that attack vector.

The caching provider investigated after the first incident, but diagnosing and fixing the underlying issue took time. When the same caching failure recurred on February 19 without an accompanying DoS, the true root cause became clear. OpenRouter deployed targeted protections within hours.

The DoS was real, and WAF rules were deployed, but OpenRouter's postmortem identifies the caching failure—not the attack—as the root cause.

6. Remediations and additional safeguards

The postmortem identifies three categories of deployed remediation and separately describes a planned fallback caching mechanism. The table distinguishes changes OpenRouter reported as deployed from the improvement whose subsequent deployment could not be verified.

From a failure-prevention perspective, controlled cache repopulation is the most directly relevant remediation described in the postmortem: it addresses the sudden database-load increase that turned the cache failure into a broader outage. The report does not provide enough implementation detail or post-fix performance data to establish which remediation contributed most to reliability. OpenRouter does not publish which techniques it implemented, so the exact mechanism is not reconstructed here; only the stated intent, managed repopulation, is reported. It does not disclose whether the circuit breaker uses singleflight, token buckets, bounded queues, or any specific database protection mechanism. The source describes OpenRouter's intended protection and controlled repopulation, not a formal guarantee against every cold-cache failure.

Accurate error handling is a relatively localized remediation compared with introducing another caching layer. Its implementation cost depends on the existing authorization path, but the design principle is broadly applicable: distinguish a confirmed authentication failure from an infrastructure failure that prevents the credential from being evaluated.

One more safeguard follows from the phase table but is not in the report: cache warming. If the cache can be pre-populated with the hot key set after a connection drop, the cold-recovery phase can be reduced, though warming may be incomplete and concurrent misses may still overwhelm the database. OpenRouter's planned fallback cache addresses a related but distinct failure mode: extended downtime in the caching layer. The postmortem does not establish that either cache warming or the fallback mechanism was implemented as part of the deployed remediation.

Additional safeguards for systems with the same failure mode

The following are Buildopsy's engineering recommendations, not undocumented details of OpenRouter's fix.

• Bounded database concurrency: set a maximum number of concurrent fallback lookups and define what happens when that limit is reached, whether requests queue, get rejected, or fall through to a degraded path. An unbounded queue converts a spike into latency and then into timeouts.
• Request coalescing: concurrent requests for the same key can share a single in-flight lookup. Coalescing only helps when concurrent requests share lookup keys; it does not eliminate a flood of distinct-key misses.
• Rate-limited cache warming: repopulate the cache at a controlled rate after a connection drop, and define how incomplete warming is handled, whether misses fall through, are served stale, or are rejected.
• Load shedding: reject or defer requests once database queue depth or lookup latency exceeds a defined threshold, rather than allowing work to accumulate.
• Observability: measure cache hit ratio, database queue depth, lookup latency, timeout rate, and authentication failures separated by cause, so an infrastructure failure is not hidden inside an authentication-failure metric.
• Cold-cache failure testing: drop cache connections, restore an empty cache, and verify that database load stays within a defined limit and that error codes remain accurate.

7. Verdict

The postmortem establishes the mechanism and the outcome: 38 and 35 minutes of impact, peak failure rates of 80-90%, and a concurrent DoS on February 17 that misdirected the initial investigation until the February 19 recurrence exposed the true cause. OpenRouter shipped circuit breakers with controlled repopulation, 503 instead of 401, and provider-side fixes, and planned a fallback cache.

Several things remain open. OpenRouter does not name the caching provider or the database, does not publish request totals, affected-customer counts, or financial impact, and does not disclose the exact repopulation mechanism behind the circuit breaker. The postmortem records occasional caching-layer errors beginning at 5:20 UTC on February 17 and 7:16 UTC on February 19, followed by connection drops at 5:27 and 7:36 UTC, respectively. It does not establish when the underlying fault first began. The 80-90% peak failure rate is a published range, not a measured distribution. The planned fallback cache is described as coming soon, with no date.

The transferable lesson is that cache recovery can trigger a larger outage when invalidating the cache forces requests onto a database that cannot absorb the resulting burst. The authentication path also needs to distinguish an invalid credential from an infrastructure failure: accurate error codes make the failure legible, while controlled fallback load protects the underlying dependency. Neither measure substitutes for the other.

Sources and Method

Primary source: OpenRouter's postmortem, "OpenRouter Outages on February 17 and 19, 2026," published February 20, 2026 and updated June 24, 2026. I checked the What Happened, Impact, Remediations, What's Next, and Timeline sections. All timestamps, failure rates, and remediation statements are quoted or paraphrased from that document. Durations are arithmetic from the published timestamps: 5:27 to 6:05 UTC is 38 minutes; 7:36 to 8:11 UTC is 35 minutes. The 20% and 80-90% failure rates are published ranges. No request totals, customer counts, revenue figures, or provider identities are inferred, because the report publishes none. The cache-state table in Section 2, the illustrative load model, and the additional safeguards in Section 6 are Buildopsy's design analysis, not OpenRouter statements. The controlled-repopulation mechanism behind the circuit breaker is unspecified in the report and is not reconstructed here. As of the June 24, 2026 update, the fallback caching mechanism was still described as a planned improvement; its subsequent deployment could not be verified from the public sources reviewed for this article. This is an independent analysis, not an OpenRouter statement.

• OpenRouter: OpenRouter Outages on February 17 and 19, 2026 (published Feb 20, 2026; updated Jun 24, 2026)
• Buildopsy: Database Connection Pool Calculator
• Buildopsy: Cache Hit Ratio Savings Calculator
• Buildopsy: CircleCI WAL saturation, a separate recovery-queue case
• Buildopsy: Azure OpenAI Sweden Central, a separate cache-miss and restart-loop case

Figure 1 is a Buildopsy reconstruction of the published sequence. It does not depict OpenRouter's private topology, the unnamed caching provider, or internal dashboards.