I’ve spent enough time debugging distributed systems to know that the simplest architecture is rarely the right one at scale. Meta’s ZippyDB story illustrates this beautifully through ZGateway, their proxy layer that sits between millions of clients and a database fleet. It’s a case study in how where you solve a problem matters as much as how.
The origin story is familiar: direct client-to-database connections work fine until they don’t. With a million clients each potentially touching tens of thousands of shards across hundreds of thousands of hosts, you get a many-to-many mesh so dense it becomes a reliability hazard. Every client holds tens of thousands of open connections. Every database host accepts tens of thousands inbound. Memory bleeds away. File descriptors exhaust. One routing bug triggers a reconnection storm that brings down the fleet.
The instinct is to fix this in the client library. Implement connection pooling. Add exponential backoff. Tune the batch sizes. But here’s what nobody talks about: you now own that complexity in a million binaries across hundreds of teams. When deployment strategies shift, when traffic patterns change, when you discover a subtle bug in your pooling logic, you’re jumping between two moving cars. The client fleet evolves on its own schedule. The database fleet evolves on another. Coupling them directly becomes a source of cascading failures.
The Asymmetry of Control
ZGateway’s real insight is structural. A proxy sits in a fundamentally different position than any client library. It sees traffic from many clients at once. It’s a single tier you control and operate. When a problem appears, you don’t coordinate rollouts across a million deployments; you fix it in one place.
With ZGateway in the path, the topology changes from O(clients × hosts) connections to something bounded by the proxy fleet size and shard distribution. Each client needs only a sticky pool to its regional gateway. Each database host only sees connections from the proxy tier. Total persistent connections drop by roughly 19x. But more importantly, the scaling behavior changes fundamentally: database-host fan-in becomes independent of the client population. An unbounded problem driven by everyone else becomes a bounded problem you control.
This is why proxies keep appearing in different contexts. Connection poolers, service mesh sidecars, CDNs, API gateways. Whenever you have many diverse clients talking to shared backends and you can’t change the clients quickly, a proxy becomes not an optimization but a necessity.
The Batching Opportunity
Once traffic flows through a single tier, capabilities emerge that are impossible elsewhere. Request batching is the clearest example. A client-side batcher can only merge its own process’s requests. A shared gateway batcher can merge requests from any client headed for the same shard, amortizing the fixed overhead of each RPC across more work.
This has second-order effects. A thousand simultaneous callers hitting the same key collapse into a single backend fetch. The hot key can never become a stampede. Meanwhile, the QPS billing is based on requests sent, so batching stretches rate-limit budgets for free. Years of fragile, individually-tuned client-side batching libraries could finally be retired.
Safety matters though. Requests linger in memory waiting for the batch to flush, creating OOM risks. ZGateway handles this with idle eviction and an in-flight cap: when backend latency rises and coroutines pile up faster than they drain, new executions get rejected rather than queued indefinitely. Steady housekeeping plus an acute safety valve is what makes batching safe by default.
The Other Lessons
Beyond batching, a shared tier becomes the natural home for capabilities clients would otherwise duplicate. Per-tenant load shedding through discriminant buckets. Read-through caching with CDC-driven invalidation. Traffic steering that balances load across heterogeneous hosts. A control plane that computes weighted consistent hashing to handle hosts ranging from 26 cores to 126 cores. None of these are novel, but they become feasible when you have one place to implement them.
Migration is designed as reversible configuration: a percentage knob ramps eligible traffic, a region filter limits blast radius, a kill switch enables instant rollback. No client code changes required. This is how you move infrastructure at scale without waking up on-call engineers at 3 AM.
The tradeoff is real: you add a hop and one more tier to operate. But when the client population is large, diverse, and not yours to change, that tradeoff pays off structurally. Read the details on distributed systems design at systems-design for more context, and explore the broader infrastructure patterns that make this approach sustainable.
The question isn’t whether to add a proxy layer; it’s whether you can afford not to when your fleet reaches a certain complexity. At what point do the operational costs of direct connectivity exceed the costs of managing another tier?