A distributed webhook-delivery service across five nodes with no coordinator, built on a reliable-UDP transport and SWIM membership written from scratch. A job is durable on a majority before the submit is acknowledged.
The knee is at about 800 jobs a second, and past 1000 it does not degrade gracefully. Those rows are in the CSV. They are shown here as what they are rather than left out or drawn as a curve.
Twenty seconds at each offered rate. Up to 600 a second the tail sits in a flat band: the worker poll interval plus delivery plus two majority commits. The band is what a healthy system looks like, and leaving it is the signal.
Every job costs roughly nine fsyncs cluster-wide: submit, lease and done on the owner, plus the replica appends behind each. At 800 a second that is about 1,400 fsyncs per second per node, and the design commits one at a time. The transport is not the limit; the disk barrier is.
Group commit already exists in the write-ahead log as an unbatched call and a separate sync. Turning it on is the known next step, and it is not implemented, so the number on this page is the number without it.
Held at 200 jobs a second, well under the knee, so loss is the only thing changing. Loss is injected on every node's UDP ingress, which means a dropped packet can be a submit, a replica append, or a membership probe.
Nothing is lost up to 20% packet loss. Every job that was accepted still completed at every point on this chart, which is what the retransmit and the majority commit are for. What changes is the tail.
At 10% loss the p99 is 127 ms, roughly double the clean figure. At 20% it is 1,019 ms, sixteen times. A retry costs a round trip and the round trips compound, so the tail moves long before the throughput does.
Two steps past the knee are in the CSV and are not in the results table. Here they are, and here is why they are not a measurement of anything.
Past about 1000 a second the cluster does not degrade gracefully: nodes become intermittently unresponsive, submits and scrapes time out, and the step stops offering the load it was told to. At 1000 the harness offered 2,960 of the 20,000 it was asked for and nothing completed inside the window, so the percentile columns are a sentinel rather than a latency. At 1200 the completions are backlog from earlier steps draining, which is why that row reports more completed than it offered.