multi-tenant api gateway · go · redis-lua rate limiting · loading engine…

tollgate.

A gateway for teams sharing one LLM API key: everyone gets a personal key and a budget, the real key never leaves the gateway. That only works if the limit is a limit, and rate limiting per replica is quietly wrong. The admission-control code below is the gateway's own, the same algorithms its Redis Lua scripts implement, compiled to WebAssembly on a virtual clock.

github.com/lgoyal6/tollgate try the live one ↓
2.9×the policy ceiling admitted by per-replica counters across 3 replicas (k6, measured)
9,000/9,000exact with the Redis-Lua shared store at the same ceiling
46msp99 at a sustained 2,000 req/s, zero errors

measured against the kind cluster with k6 · docs/ratelimit-proof.md · suites in loadtest/

01

Spend from a bucket you share with strangers

a real gateway, a real Redis, one token bucket for everyone reading this page
checking the gateway…
your browserissues an HTTP request
→
the gatewaychecks your key, finds the route
→
the limitertakes a token from Redis, or refuses
→
the upstreamanswers, and you get a 200

Press the button. One request, one token out of the shared bucket. When the bucket is empty the limiter stops it and nothing reaches the upstream.

tokens left
–
bucket size
–
refills
1/s
your requests
0
you were limited
0

Fig. 1. This is not a simulation. The button calls a tollgate gateway running on Railway, which checks one token bucket in Redis and then proxies to a demo upstream. The bucket holds 12 and refills at 1 a second, and it is the same bucket for every visitor: if the number drops without you touching anything, that was somebody else. A 429 here is the gateway's real answer, carrying the real Retry-After. The service sleeps when nobody is using it, so the first request after a quiet spell takes a few seconds to wake it.

02

Run the experiment

one tenant, one policy, N replicas behind a round-robin balancer; only the counter's home differs

Fig. 2. Admitted requests per second over 30 simulated seconds. The dotted line is the offered load; the fine rule is the policy. The dashed gateway lets every replica count on its own, so each one happily admits a full ceiling; the solid gateway makes every replica check one shared store before admitting, the exact check the Redis Lua scripts perform in production. The solid line's first second sits above the ceiling by design: that's the token bucket honoring its configured burst, and the policy bound accounts for it.

03

Read the bill

on a shared LLM key, over-admission is not an abstraction; it is someone's money
Table 1. The experiment, 30 simulated seconds.
requests offered–
policy bound for the horizon (ceiling × 30s + burst)–
admitted, per-replica counters–
admitted, one shared store–
over-admission, per-replica–
Table 2. The receipt, per-replica gateway.
requests admitted above the policy bound–
at $0.002 per request (illustrative LLM call)–
same leak, extrapolated to a day at this load–

running…

04

What you just saw

every behavior above is a specific piece of the gateway
THE SPLIT COUNTER
Each replica counting alone, each admitting a full ceiling.
The in-memory limiter is kept in the codebase deliberately: it makes the failure mode demonstrable and doubles as the reference implementation for the algorithm unit tests: internal/ratelimit/memory.go
THE SHARED STORE
One counter every replica must ask before admitting.
Check-and-decrement made atomic in Redis Lua, timed by the Redis server's own clock so replicas cannot disagree about now: internal/ratelimit/tokenbucket.lua, internal/ratelimit/slidingwindow.lua
THE CLOCK
Thirty seconds of traffic simulated in milliseconds, reproducibly.
The demo drives the same limiter interfaces on a virtual clock, the same fake clock the algorithm unit tests use: internal/ratelimit/clock.go, cmd/demo-wasm/main.go
THE GATEWAY
The limiter is one stage of a production request path.
In front of upstreams this sits with per-upstream circuit breakers, jittered-backoff retries, and request hedging, all hand-rolled, plus OTel traces and Prometheus metrics: internal/proxy, internal/resilience