Back to System Design Index

Architectural Critique July 2026 12 min read

One Outage Should Never Eat Every Region

Target Scale
50k Config Updates/s
Max Outage Target
< 5% Total Impact
Primary Tech
Golang, Raft, AWS
Pattern
Cellular Isolation

A bad config ships in one region. Minutes later every region agrees to fail together. Cell-based architecture answers with bulkheads: split the control plane into isolated Raft-backed cells so a blast in one stays in one. This is the design, the Go router, plus the trade-offs.

TL;DR: One bad config should never take every region. Cell-based control plane isolates failures with autonomous Raft-backed cells for zero blast radius. Global sync caused the outage.

1. The Problem: Global Configuration Outages

Monolithic control planes that propagate configuration changes globally introduce systemic risks. A single corrupted payload or high-frequency polling spike can cascade across all availability zones simultaneously.

Architectural Flaw: In a unified global database setup, a schema locking issue or bad feature flag deployment impacts 100% of tenants simultaneously, violating basic blast-radius constraints required for Tier-0 cloud services.
Pen sketched cartoon of a skeptical cat watching a Raft election inside one cell
Figure 2. The Raft round in one cartoon, drawn for this postmortem. Leader falls. Followers vote. The cell never blinks.

2. Proposed Target Architecture

By restructuring the control plane into isolated, self-contained Cells, where each cell operates its own independent state machine, we decouple failures entirely.

Pen sketched cartoon of a grinning cat beside isolated region cells
Figure 1. Blast radius in one cartoon, drawn for this postmortem. One cell quarantined. The rest stay fine.
graph TD A[Global Edge Router / API Gateway] -->|Tenant Hash Routing| B[Cell 01 - US-East] A -->|Tenant Hash Routing| C[Cell 02 - US-West] A -->|Tenant Hash Routing| D[Cell 03 - EU-Central] subgraph Cell 01 Isolation Boundary B --> B1[Go Ingress Proxy] B1 --> B2[Local Raft Cluster] B2 --> B3[Local DynamoDB/Scylla] end subgraph Cell 02 Isolation Boundary C --> C1[Go Ingress Proxy] C1 --> C2[Local Raft Cluster] C2 --> C3[Local DynamoDB/Scylla] end

Figure 2: Cell-based routing showing zero shared runtime dependencies between regions.

3. Go Implementation: Partition Hash Router

Below is the core hashing implementation used at the gateway layer to route incoming client requests directly to dedicated cell partitions without shared state calls: 

package main

import (
	"hash/fnv"
)

type Router struct {
	TotalCells int
}

func (r *Router) GetCellID(tenantID string) int {
	h := fnv.New32a()
	h.Write([]byte(tenantID))
	return int(h.Sum32() % uint32(r.TotalCells))
}
Operator rule: Cells must fail without calling home. If failover needs the control plane, the control plane is the blast radius.

4. Trade-Off Evaluation

Vector Monolithic Global Control Plane Cell-Based Architecture (Selected)
Blast Radius 100% of global traffic <. 5% (Limited to single cell)
Operational Complexity Low (Single DB cluster) High (Requires automated cell provisioning)
Cross-Cell Data Sync Centralized Async Event-Driven (EventBridge/Kafka)