I find one of the most instructive patterns in systems design isn’t what you build, but where you build it. Meta’s ZGateway is a perfect case study: a proxy tier that sits between a million clients and a shared key-value store, solving problems that no amount of client-side engineering could crack.
The problem it solves is deceptively simple to state but vicious in practice. Direct client-to-database connections at scale create a dense many-to-many mesh. Each of a million clients might hold tens of thousands of outbound connections, and each of hundreds of thousands of database hosts accepts tens of thousands of inbound ones. The connections are mostly idle, but they consume memory, CPU, and file descriptors on both ends. Worse, when something breaks (a deploy, a routing bug, a restart), that mesh becomes a liability: a reconnection storm can exhaust file descriptors and push the fleet into a reboot loop.
The insight is structural. A proxy sits in a different position than any individual client. It sees many callers at once, which buys three capabilities that no client library can offer: centralized connection management, cross-client request batching, and the ability to apply policy uniformly.
The Asymmetry That Scales
What I find elegant about ZGateway’s design is how it inverts the scaling equation. In the direct model, database-host fan-in grows linearly with the client population: every new cohort of clients makes every backend host worse. At a million clients, this is a disaster waiting to happen.
With ZGateway in the path, the client population drops out of the equation entirely. Fan-in reduces to roughly regions multiplied by shard density per host, independent of how many clients exist. An unbounded number driven by everyone else becomes a bounded number the infrastructure team controls.
This matters because it decouples two previously coupled systems: the client fleet keeps changing and expanding while the database fleet consolidates on its own schedule. A proxy breaks that coupling. Jumping between moving cars becomes tractable again.
The numbers are worth noting. Meta saw an 19x reduction in total persistent connections, though the real win isn’t the one-time drop. It’s that the scaling curve changed shape. Database hosts now experience bounded fan-in regardless of adoption.
Batching as a Proxy Superpower
The second implication is subtler and, I think, more interesting. Because a proxy sees multiple clients’ requests in a shared queue, it can do something impossible at the client level: batch across unrelated callers.
Imagine thousands of simultaneous requests for the same key hitting the proxy from different services. The proxy fetches it once and fans the result out. A client-side batcher can only merge its own process’s requests; the proxy collapses them across processes, services, and teams.
This has cascading effects. First, fewer, larger backend RPCs mean lower QPS and CPU. Every RPC carries fixed overhead, so amortizing it across more operations is pure efficiency. Second, hot keys can never become a stampede against a single replica; they collapse into a single backend read at the proxy layer. Third, the proxy can retire client-side batching libraries that were fragile, CPU-hungry, and a steady source of production incidents because their complexity lived in a million binaries nobody controlled.
Request batching also stretches rate-limit budgets. Because use cases are billed by QPS, fewer, merged requests mean lower perceived QPS and less throttling with zero changes on the client side.
What This Teaches About Proxy Layers
Proxies show up everywhere you have a large, diverse client population talking to shared backends: connection poolers, service mesh sidecars, CDNs, API gateways. The tradeoff is an extra hop and another tier to operate. But when the client population is large, diverse, and not yours to change, that tradeoff inverts.
ZGateway handles over a billion operations per second while adding only 6% computational overhead on average. It currently carries 40% of ZippyDB traffic and is growing. The fact that this is possible speaks to the power of the pattern: moving a capability to the right layer at the right time can solve problems at scale that no amount of distributed client engineering could.
What strikes me most is the admission that direct access was the right design early on. ZGateway didn’t exist because it was always needed; it emerged because Meta first built the foundational pieces that made a shared tier feasible (better service routing, overload protection, thin clients). This is a humbling reminder that architectural decisions are valid for their era, and scaling often means recognizing when the era has changed.
The real question isn’t whether you need a proxy, but at what scale does the cost of coordination across a million binaries exceed the cost of one more managed tier?