Floods, Fairness and Other Acts of God.
Twenty million events, one two-core gateway, and what FairGate’s numbers really say about group commit, tail latency and the gap between accepting data and delivering it.
· Updated · 14 min read
FairGate is an event gateway I wrote in Go. Producers connect over TCP and send events. Each one gets a rate limit and a bounded queue. Every accepted event goes into a write-ahead log, gets acknowledged only once it is on disk, and is later shipped to ClickHouse exactly once per event identity. The name is not subtle about the goal: one noisy producer should not ruin things for everyone else.
So I built the machinery you would expect. A token bucket per producer. Bounded per-producer queues. A weighted deficit round robin scheduler, because nothing says “fair” like an algorithm with a 1996 paper behind it.
Then I tested it properly. I let 15 producers flood the gateway beside 285 polite ones, and the polite ones’ median latency moved from 4.6 ms to 4.8 ms. Lovely. Slightly less lovely was the rest of the data, which calmly explained that the rate limiter had never fired, the queues had never filled, and the acknowledgment doesn’t wait for the scheduler at all.
This is the long version of that story, 20.5 million events of it, told roughly in the order my assumptions fell over. First, the thirty-second version of how FairGate works, and the rules I set so I couldn’t fool myself. Then speed: two vCPUs sustaining 23,245 durable events a second, a disk with an alibi, and a tail I went hunting for that never showed up. Then the awkward part, fairness, and who actually deserves the credit for it. And finally delivery, where a bug that no test had caught was hiding, and where the number worth quoting turned out to be a good deal smaller than the one I wanted to.
FairGate in thirty seconds
The full tour of the internals has a post of its own. Everything here depends on just three facts, so they are worth a minute.
- An ACK means the event is on disk. Every accepted event is appended to a write-ahead log, and the acknowledgment waits for the fsync. So an acknowledgment always includes at least one trip through the disk.
- A connection is served strictly one event at a time. Each connection’s goroutine reads one frame, handles it completely, sends the acknowledgment, and only then reads the next frame, however many frames its owner has stuffed into the socket. I wrote it that way because it was the simple thing to do. It would turn out to be the most important line in the system, which is a humbling thing to learn about a loop.
- Delivery is a separate path. A shipper reads durable records from the log and sends them to ClickHouse in batches, so a slow database grows a backlog rather than slowing anyone’s acknowledgment, at least until that backlog reaches 1 GiB and the overload controller starts saying no.
Trying not to lie to myself
Benchmarks are easy to win if nobody is checking, and I was the only one checking. So the rules came first.
The first rule: no burstable instances. A t3 shares its physical cores, and unlimited CPU credits don’t stop the hypervisor from taking CPU time away, so a benchmark on one is mostly a benchmark of your neighbours. Everything here ran on m7i and c7i instances, and CPU steal stayed at or below 0.5% on every host.
The gateway was one m7i.large (two vCPUs, 8 GiB, a 30 GiB gp3 volume) running FairGate with Prometheus, Alertmanager and Grafana alongside it. ClickHouse got an m7i.large of its own, and three c7i.large machines in the same availability zone generated the load, all starting on the same wall-clock second. Five instances at on-demand prices come to about $0.53 an hour, and the whole suite cost about a dollar, which is the cheapest humility I have ever bought.
The build under test is FairGate with a reworked shipper from an experimental branch. Why it needed reworking is its own story, further down. Before any load, the gateway ran go vet, the full test suite under the race detector and the ClickHouse integration tests against the real ClickHouse instance. Then came eight scenarios, three repeats each: 24 runs, 20.5 million events and 1,531 seconds of load. Producers came in two flavours, because they answer different questions:
- Closed loop. Send, wait for the ACK, pause about 3 ms, repeat. The gateway sets the pace, so this finds its capacity.
- Open loop. Send on a fixed schedule whatever happens. This is how real load behaves, and measuring from each event’s intended send time avoids coordinated omission, the classic way a benchmark quietly forgets the requests a stalled system never got round to.
The load generator is Python. Yes, I know. That makes it the first suspect in any slowdown, so every run had to prove its innocence: event-loop lag under 50 ms at p99, worker CPU under 85%, steal under 5%, every ACK matched to an event in flight, and the gateway’s count within 0.1% of the client’s. All 24 runs passed. Event-loop lag peaked at 31 ms, worker CPU at 31%, and no ACK ever went unmatched.
There was one false alarm worth confessing. The load generator’s own whole-run check flagged all three mixed runs, because the gateway had counted 0.3% fewer events than the client. Only one load generator scrapes the gateway, and it stops 1.5 seconds after its own producers finish, while slow producers on the other two are still sending. Compared over the same window, the two counts agree to within 0.0003%. The check now uses that window.
There is also a free check that arithmetic offers anyone who asks. Little’s law says the number of producers must equal throughput times the time each spends per event: N = X(R + Z). Plugging in the measured numbers gave back the real producer count to within 2% in every closed-loop scenario. When a queueing law from 1961 agrees with your dashboard, you can relax a little.
Surprise one: the disk had an alibi
Here is the headline. With durable acknowledgment on a two-vCPU gateway, FairGate sustained 23,245 events per second with 1,500 producers connected. Every one of those was on disk before its producer heard back. The trick is group commit, which the internals post walks through line by line.
It is a carpool. One writer owns the log, and everyone who turns up while the previous fsync is running shares the next one. The busier it gets, the fuller each car.
Throughput is just commits per second times events per commit, and at both ends of the range that identity holds to within 0.2%: 279.2 × 37.5 is 10,470, against 10,457 measured at 100 producers. Watch what happens to the two factors as producers pile in:
| Producers | fsyncs/s | Events per fsync | fsync (ms) | Host busy | I/O wait |
|---|---|---|---|---|---|
| 100 | 279.2 | 37.5 | 3.3 | 49% | 29% |
| 250 | 234.8 | 82.6 | 3.6 | 84% | 10% |
| 500 | 189.9 | 121.2 | 4 | 88% | 8% |
| 1,000 | 153 | 149.6 | 4.8 | 90% | 8% |
| 1,500 | 147.5 | 157.6 | 5.1 | 91% | 7% |
From 100 to 1,000 producers the commit rate nearly halves while the batch size quadruples, and that is the entire scaling story. At 1,500 producers, about 160 events share a single fsync. Each commit cycle stretches from 3.6 to 6.8 ms, and the time a handler waits for its append grows from 6.0 to 22.2 ms as the queue in front of the writer gets longer.
Now, if you ask anyone what limits a durable system, they will point at the disk. I would have too. But look at the last two columns. From 250 producers onward the host is 84–91% busy with only 7–10% I/O wait, and fsync barely slows down: 3.3 ms at 100 producers, 5.1 ms at 1,500. The disk has an alibi. This is a CPU story: FairGate climbs from 70% to 132% of one vCPU, and Prometheus, Grafana, Alertmanager, container logging and the kernel fight over what remains. Past that point, throughput grows only because bigger batches spread each commit’s cost over more events, and batches stop growing much at around 160.
Extra producers beyond the plateau buy you nothing except a longer line. From 500 producers to 1,000, every latency percentile roughly doubles, exactly as Little’s law says it must.
- 99th percentile
- Median
One number here still bothers me. At 1,500 producers, clients measure a mean of 62.5 ms, but the gateway’s own timer, which starts when it reads a frame, says 25.6 ms. So about 37 ms pass before FairGate even looks at the event. The likely culprit is scheduling delay on a CPU-saturated host, plus a smaller share in the client, but “likely” is doing real work in that sentence. A frame-arrival timestamp on the server would settle it.
Open loop: exactly as boring as promised
The open-loop sweep is where a system should look boring, and this one obliged. FairGate accepted exactly what it was offered at every step, all the way to 20,000 events per second, the highest the test went. I didn’t find the ceiling, which will take a bigger sweep. The median stayed between 4.6 and 7.2 ms up to 15,000 events per second and reached 14.6 ms at 20,000. The floor is pleasingly explicable: a lone request waits up to 2 ms for company, then pays an fsync of about 2.9 ms.
- 99th percentile
- Median
See the numbers
| Offered/s | Accepted/s | Median (ms) | p99 (ms) | p99 from intended (ms) |
|---|---|---|---|---|
| 1,000 | 1,000 | 4.6 | 7 | 8 |
| 2,000 | 2,000 | 4.6 | 7 | 8 |
| 4,000 | 4,000 | 4.8 | 8 | 9 |
| 6,000 | 6,000 | 5 | 10 | 10 |
| 8,000 | 8,000 | 5.6 | 22 | 22 |
| 10,000 | 10,000 | 6.5 | 24 | 24 |
| 12,000 | 12,000 | 6.5 | 31 | 31 |
| 15,000 | 15,000 | 7.2 | 66 | 66 |
| 20,000 | 20,000 | 14.6 | 196 | 196 |
The tail climbs smoothly with the load, from 7 ms at 1,000 events per second to 66 ms at 15,000 and 196 ms at 20,000. I built the sweep to catch coordinated omission, by measuring each event from when it was supposed to be sent. It caught nothing. No send fell more than 1.5 ms behind schedule, so the p99 from the intended send time is identical at every step. A trap that comes back empty is mildly disappointing and entirely good news.
The tail that wasn’t there
On a durable system, tail latency usually comes from the disk, so I watched it second by second. Here is one isolation run: 285 producers sending ten events a second on a schedule and, from about 21 seconds in, 15 others flooding for 38 seconds.
- 15 producers flooding
The flooders got 172 events per second each, seventeen times a paced producer’s rate, and a median of 190 ms for their trouble. The paced producers barely noticed:
| Paced producers | Median | p99 | p99.9 |
|---|---|---|---|
| No flood | 4.6 ms | 7.4 ms | 18 ms |
| Flood active | 4.8 ms | 8.6 ms | 20 ms |
The disk had nothing to report either. In all 1,485 measured seconds there wasn’t a single storage stall, which is my name for a second in which the WAL fsync’s p99 goes past 40 ms and four times its usual value. The gateway typically wrote 11–16 MiB/s and peaked at 66–102 MiB/s, all under the volume’s baseline, and the WAL itself was only 1–8 MiB/s of that. The rest is Prometheus, container logs and the operating system. The fsync p99 stayed at its load-dependent baseline: 4–8 ms at low load, and up to 29 ms at 1,500 producers, where batches are largest.
The ClickHouse machine is a different story. It sat idle on disk most of the time, then wrote in bursts of up to 183 MiB/s as inserts landed and parts merged, more than the gateway volume’s entire baseline. On its own disk, that’s nobody else’s problem, which is exactly why it lives there.
Surprise two: where fairness actually comes from
The mixed scenario is the rude one. Of 750 producers, 40 misbehave in six different ways: bursting, flooding, sending slowly, disconnecting constantly, sending malformed events, sending oversized ones. On top of that, 426 otherwise polite producers pipeline 50 events at a time during two synchronised waves. The question is simple: who pays?
- Median
- 99th percentile
See the numbers
| Behaviour | Median (ms) | p99 (ms) |
|---|---|---|
| Well-behaved | 25.3 | 84 |
| Keeps reconnecting | 29.6 | 94.1 |
| Sends malformed events | 30.1 | 94.5 |
| Sends oversized events | 30.1 | 94.9 |
| Sends slowly | 28.2 | 95.4 |
| Pipelines in bursts | 321.2 | 895 |
| Floods | 731.1 | 1,767.2 |
The answer is beautifully specific. Well-behaved producers saw a median of 25 ms. Producers that disconnected, sent garbage or dawdled saw 28–30 ms, so bad manners alone cost them a few milliseconds and cost everyone else nothing. Throughput across the 710 well-behaved producers was almost perfectly even, with a Jain’s fairness index of 0.9997, where 1 is perfect. And the ones pipelining? Bursters waited a median of 321 ms and flooders 731 ms. The queue they were stuck in was their own.
The waves tell the same story. While 426 producers pipelined at once, throughput rose to about 28,000 events per second, because pipelined frames are cheaper per event. The participants waited about 700 ms, their own queue again, while everyone else carried on at 34 ms.
The flood itself got very little for its trouble: about 62 events per second against a polite producer’s 31, at about 29 times the latency. In the isolation scenario, on a quieter host, flooders reached 172 events per second with a median of 190 ms, while the 285 paced producers carried on at 4.8 ms.
Time to collect my prize, then. A token bucket, bounded queues and DRR, all doing their jobs. Except I checked the counters. Across all 24 runs, not one event was rate limited. Not one was refused for overload. Sampled once a second, the fullest queue never got past 1% of its capacity. So I went and read what the scheduler actually does with an event:
err := scheduler.Run(schedulerCtx, queues, func(event wire.Event) error {
metrics.EventProcessed(event.ProducerId)
logger.Info("Event processed by DRR",
"producer_id", event.ProducerId,
"seq", event.Seq,
)
return nil
})It increments a counter and writes a log line. The ACK doesn’t wait for any of this, because it is sent right after the enqueue, and the shipper reads directly from the WAL. In this design, DRR cannot affect ACK latency unless a queue fills, and in these tests none did. It can’t be what protected anyone. I had built a very fair way of writing log lines.
The real protection is two things I didn’t design as fairness features at all:
- Everyone waits in the same group commit. There is no separate fast lane to starve. One fsync covers whoever happens to be waiting.
- Each connection is served one event at a time. A window of k pipelined frames waits about k service times, in its own socket, delaying nobody else. You can’t push in front of other people when your own frames are the queue.
The second one explains the numbers too. A connection can’t exceed about one event per service time: 16.8 ms of ingest time in mixed is about 60 events per second, and 4.6 ms in isolation is about 220. Both sit well under the 500-per-second admission limit, so a single connection can never reach it.
I don’t want to overcorrect. This doesn’t make the rate limiter or the overload controller decoration. They exist for the cases this suite never created: one producer with many connections, much faster storage, a throttled shipper. But that’s precisely the point. An untested safety net isn’t a safety net yet, it’s a hypothesis. And DRR’s fairness on the delivery side wasn’t measured at all. Those need tests of their own.
While we’re being honest: the malformed events
The gateway’s handling of bad input was reassuringly dull. Every event with an empty or missing identifier was answered invalid, the connection stayed usable, and every frame over the WAL’s 1 MiB record limit was refused while everything from 0.5 to 1 MiB got through. No connection and no event failed on the client.
| Input | Sent | Accepted | Answered invalid |
|---|---|---|---|
| Empty producer_id | 144 | 0 | 144 |
| Empty event_type | 171 | 0 | 171 |
| Missing event_type | 141 | 0 | 141 |
| seq = 0 | 129 | 129 | 0 |
| Frame over 1 MiB | 150 | 0 | 150 |
| Frame of 0.5–1 MiB | 72 | 72 | 0 |
You’ll notice 129 “malformed” events with seq = 0 were accepted. I spent a moment worrying about this before remembering that seq is an unsigned integer and zero is a perfectly good one. FairGate was right. My test was wrong. I’m choosing to count this as the gateway passing an exam I failed.
Surprise three: the bug that only appeared at scale
Everything so far is about accepting events. Getting them into ClickHouse is the shipper’s job, and FairGate’s default shipper was built for a gentler life. It sent 200 events per batch, one batch at a time, and every batch paid a fixed toll: two checkpoint commits, each with a file fsync and a directory fsync, an existence query to keep delivery exactly-once, and a synchronous insert. Small batches pay that toll constantly. So before this suite I changed the shipper in three steps on an experimental branch, and measured each one on the two heaviest scenarios, with ClickHouse on its own machine:
- B: batches of 5,000, so the toll is paid 25 times less often.
- C: the existence check in chunks of at most 1,000 IDs, with every failed insert logged.
- D: read-ahead and merged commits. The next batch is read and decoded while the current one is inserted, and one checkpoint commit records the current batch as delivered and the next as pending. Inserts still go one at a time, so each existence check sees every row before it, and exactly-once stays exactly once.
Step B is where it got interesting. Delivery at 1,500 producers fell to 540 events per second. Every run showed six or seven failed insert attempts, spaced out by the growing retry back-off: one batch, failing over and over. The ClickHouse client writes bound values into the query text, so the existence check for 5,000 IDs of 64 characters came to about 335 KB, comfortably over ClickHouse’s default max_query_size of 256 KiB, or about 3,900 IDs. Batches that happened to be smaller got through. Full ones never did. And nothing in FairGate’s logs said why, because failed inserts only incremented a metric.
Chunking the query fixed it outright: every batch succeeded, and delivery rose to 11,409 events per second at 1,500 producers and 16,221 in mixed. Reading ahead added another 7% to 9%, to 12,470 and 17,317.
| Shipper step | Delivered/s, 1,500 producers | Delivered/s, mixed | Peak lag, 1,500 (MB) |
|---|---|---|---|
| B: Batches of 5,000 | 540 | 3,530 | 279 |
| C: + existence check in chunks of 1,000 | 11,409 | 16,221 | 119 |
| D: + read-ahead and merged commits | 12,470 | 17,317 | 97 |
Those runs used two load generators, which were at the edge of their headroom around 21,000 events per second, so they understate ingest. With a third, the same build accepted 23,245. Step D is what every other number in this post ran on.
Surprise four: the number you should actually quote
So here is the shipper in the full suite, against what the gateway accepted over the same period. This is the chart that made me stop quoting the big number with quite so much enthusiasm.
- Accepted into the WAL
- Delivered to ClickHouse
See the numbers
| Scenario | Accepted/s | Delivered/s | Share delivered | Peak lag (MB) |
|---|---|---|---|---|
| scale-100 | 9,678 | 9,678 | 100% | 1 |
| scale-250 | 17,973 | 17,945 | 100% | 4 |
| scale-500 | 21,370 | 20,635 | 97% | 13 |
| scale-1000 | 22,109 | 18,702 | 85% | 49 |
| scale-1500 | 22,137 | 13,839 | 63% | 113 |
| mixed | 21,368 | 18,242 | 85% | 91 |
| isolation | 3,674 | 3,668 | 100% | 1 |
| sweep | 8,590 | 8,285 | 96% | 17 |
Up to 500 producers, delivery keeps up: every event at 250 producers and 97% at 500, at up to 20,635 events per second, with backlogs of 1–13 MB. Past that it falls behind: 85% at 1,000 producers, 63% at 1,500 (13,839 per second) and 85% in mixed, with the backlog peaking at 113 MB.
The bottleneck isn’t ClickHouse, which was never more than 60% busy and only 24% at 1,500 producers. It is the shipper’s insert step, existence queries plus insert, one after another. At 1,500 producers each full step took about 360 ms, a ceiling of about 13,900 events per second, and delivery landed almost exactly on it. In mixed, 271 ms per step works out to about 18,500, against 18,242 delivered. Delivery also slows at 1,500 producers because the shipper lives on the gateway and competes with ingest for a CPU that is already 91% busy. The measurements don’t separate the existence check from the insert, so I don’t yet know which half to attack first.
And at peak it isn’t keeping up. At 1,500 producers the backlog grows by about 2.8 MB every second, enough to fill the overload controller’s 1 GiB limit in about six minutes, at which point it starts refusing everything. None of my runs lasted that long, which is convenient for the charts and not at all for the truth.
The limits of this work deserve some honesty too. These numbers come from one configuration in one availability zone, with default settings apart from the shipper’s batch size, so absolute numbers will differ elsewhere. The build includes shipper changes that aren’t in the main repository yet. Prometheus, Alertmanager and Grafana shared the gateway’s CPU and disk, so FairGate alone would have a little more headroom. The sweep stopped before FairGate did. Runs lasted 40 to 135 seconds, too short to see WAL rotation or ClickHouse at steady state. Events were synthetic, about 325 bytes each. And crashes, outages and failure injection were out of scope here. Exactly-once delivery has its own tests and its own chaos tooling, but this benchmark says nothing about them.
Still, I came away with something better than a big number. I went in wanting to show that the scheduler worked. I came out knowing that the disk wasn’t my bottleneck, my delivery path was my real capacity, a batch size can hide a bug that no unit test will find, and the fairness I’d built an algorithm for was being delivered, mostly, by a plain loop that reads one frame at a time.
I’d call that a successful benchmark. It just took me a while to stop being annoyed by it.