multi-tenant api gateway · go · redis-lua rate limiting · loading engine…

tollgate.

A gateway for teams sharing one LLM API key: everyone gets a personal key and a budget, the real key never leaves the gateway. That only works if the limit is a limit, and rate limiting per replica is quietly wrong. The admission-control code below is the gateway's own, the same algorithms its Redis Lua scripts implement, compiled to WebAssembly on a virtual clock.

github.com/lgoyal6/tollgate skip to the bill ↓
2.9×the policy ceiling admitted by per-replica counters across 3 replicas (k6, measured)
9,000/9,000exact with the Redis-Lua shared store at the same ceiling
46msp99 at a sustained 2,000 req/s, zero errors

measured against the kind cluster with k6 · docs/ratelimit-proof.md · suites in loadtest/

01

Run the experiment

one tenant, one policy, N replicas behind a round-robin balancer; only the counter's home differs

Fig. 1. Admitted requests per second over 30 simulated seconds. The dotted line is the offered load; the fine rule is the policy. The dashed gateway lets every replica count on its own, so each one happily admits a full ceiling; the solid gateway makes every replica check one shared store before admitting, the exact check the Redis Lua scripts perform in production. The solid line's first second sits above the ceiling by design: that's the token bucket honoring its configured burst, and the policy bound accounts for it.

02

Read the bill

on a shared LLM key, over-admission is not an abstraction; it is someone's money
Table 1. The experiment, 30 simulated seconds.
requests offered–
policy bound for the horizon (ceiling × 30s + burst)–
admitted, per-replica counters–
admitted, one shared store–
over-admission, per-replica–
Table 2. The receipt, per-replica gateway.
requests admitted above the policy bound–
at $0.002 per request (illustrative LLM call)–
same leak, extrapolated to a day at this load–

running…

03

What you just saw

every behavior above is a specific piece of the gateway
THE SPLIT COUNTER
Each replica counting alone, each admitting a full ceiling.
The in-memory limiter is kept in the codebase deliberately: it makes the failure mode demonstrable and doubles as the reference implementation for the algorithm unit tests: internal/ratelimit/memory.go
THE SHARED STORE
One counter every replica must ask before admitting.
Check-and-decrement made atomic in Redis Lua, timed by the Redis server's own clock so replicas cannot disagree about now: internal/ratelimit/tokenbucket.lua, internal/ratelimit/slidingwindow.lua
THE CLOCK
Thirty seconds of traffic simulated in milliseconds, reproducibly.
The demo drives the same limiter interfaces on a virtual clock, the same fake clock the algorithm unit tests use: internal/ratelimit/clock.go, cmd/demo-wasm/main.go
THE GATEWAY
The limiter is one stage of a production request path.
In front of upstreams this sits with per-upstream circuit breakers, jittered-backoff retries, and request hedging, all hand-rolled, plus OTel traces and Prometheus metrics: internal/proxy, internal/resilience