Skip to content
← All writing

Floods, Fairness and Other Acts of God.

Twenty million events, one two-core gateway, and what FairGate’s numbers really say about group commit, tail latency and the gap between accepting data and delivering it.

Pratyush Vishal

· Updated · 14 min read

FairGate is an event gateway I wrote in Go. Producers connect over TCP and send events. Each one gets a rate limit and a bounded queue. Every accepted event goes into a write-ahead log, gets acknowledged only once it is on disk, and is later shipped to ClickHouse exactly once per event identity. The name is not subtle about the goal: one noisy producer should not ruin things for everyone else.

So I built the machinery you would expect. A token bucket per producer. Bounded per-producer queues. A weighted deficit round robin scheduler, because nothing says “fair” like an algorithm with a 1996 paper behind it.

Then I tested it properly. I let 15 producers flood the gateway beside 285 polite ones, and the polite ones’ median latency moved from 4.6 ms to 4.8 ms. Lovely. Slightly less lovely was the rest of the data, which calmly explained that the rate limiter had never fired, the queues had never filled, and the acknowledgment doesn’t wait for the scheduler at all.

This is the long version of that story, 20.5 million events of it, told roughly in the order my assumptions fell over. First, the thirty-second version of how FairGate works, and the rules I set so I couldn’t fool myself. Then speed: two vCPUs sustaining 23,245 durable events a second, a disk with an alibi, and a tail I went hunting for that never showed up. Then the awkward part, fairness, and who actually deserves the credit for it. And finally delivery, where a bug that no test had caught was hiding, and where the number worth quoting turned out to be a good deal smaller than the one I wanted to.

FairGate in thirty seconds

The full tour of the internals has a post of its own. Everything here depends on just three facts, so they are worth a minute.

Trying not to lie to myself

Benchmarks are easy to win if nobody is checking, and I was the only one checking. So the rules came first.

The first rule: no burstable instances. A t3 shares its physical cores, and unlimited CPU credits don’t stop the hypervisor from taking CPU time away, so a benchmark on one is mostly a benchmark of your neighbours. Everything here ran on m7i and c7i instances, and CPU steal stayed at or below 0.5% on every host.

The gateway was one m7i.large (two vCPUs, 8 GiB, a 30 GiB gp3 volume) running FairGate with Prometheus, Alertmanager and Grafana alongside it. ClickHouse got an m7i.large of its own, and three c7i.large machines in the same availability zone generated the load, all starting on the same wall-clock second. Five instances at on-demand prices come to about $0.53 an hour, and the whole suite cost about a dollar, which is the cheapest humility I have ever bought.

The build under test is FairGate with a reworked shipper from an experimental branch. Why it needed reworking is its own story, further down. Before any load, the gateway ran go vet, the full test suite under the race detector and the ClickHouse integration tests against the real ClickHouse instance. Then came eight scenarios, three repeats each: 24 runs, 20.5 million events and 1,531 seconds of load. Producers came in two flavours, because they answer different questions:

The load generator is Python. Yes, I know. That makes it the first suspect in any slowdown, so every run had to prove its innocence: event-loop lag under 50 ms at p99, worker CPU under 85%, steal under 5%, every ACK matched to an event in flight, and the gateway’s count within 0.1% of the client’s. All 24 runs passed. Event-loop lag peaked at 31 ms, worker CPU at 31%, and no ACK ever went unmatched.

There was one false alarm worth confessing. The load generator’s own whole-run check flagged all three mixed runs, because the gateway had counted 0.3% fewer events than the client. Only one load generator scrapes the gateway, and it stops 1.5 seconds after its own producers finish, while slow producers on the other two are still sending. Compared over the same window, the two counts agree to within 0.0003%. The check now uses that window.

There is also a free check that arithmetic offers anyone who asks. Little’s law says the number of producers must equal throughput times the time each spends per event: N = X(R + Z). Plugging in the measured numbers gave back the real producer count to within 2% in every closed-loop scenario. When a queueing law from 1961 agrees with your dashboard, you can relax a little.

Surprise one: the disk had an alibi

Here is the headline. With durable acknowledgment on a two-vCPU gateway, FairGate sustained 23,245 events per second with 1,500 producers connected. Every one of those was on disk before its producer heard back. The trick is group commit, which the internals post walks through line by line.

It is a carpool. One writer owns the log, and everyone who turns up while the previous fsync is running shares the next one. The busier it gets, the fuller each car.

05k10k15k20k25k1005001,0001,500Connected producers100 producers: 10,457 events/s250 producers: 19,395 events/s500 producers: 22,549 events/s1,000 producers: 23,202 events/s1,500 producers: 23,245 events/s10,45723,245
Closed-loop throughput, mean of three repeats. It more than doubles from 100 to 500 producers, then gains just 3% more by 1,500.

Throughput is just commits per second times events per commit, and at both ends of the range that identity holds to within 0.2%: 279.2 × 37.5 is 10,470, against 10,457 measured at 100 producers. Watch what happens to the two factors as producers pile in:

Producersfsyncs/sEvents per fsyncfsync (ms)Host busyI/O wait
100279.237.53.349%29%
250234.882.63.684%10%
500189.9121.2488%8%
1,000153149.64.890%8%
1,500147.5157.65.191%7%

From 100 to 1,000 producers the commit rate nearly halves while the batch size quadruples, and that is the entire scaling story. At 1,500 producers, about 160 events share a single fsync. Each commit cycle stretches from 3.6 to 6.8 ms, and the time a handler waits for its append grows from 6.0 to 22.2 ms as the queue in front of the writer gets longer.

Now, if you ask anyone what limits a durable system, they will point at the disk. I would have too. But look at the last two columns. From 250 producers onward the host is 84–91% busy with only 7–10% I/O wait, and fsync barely slows down: 3.3 ms at 100 producers, 5.1 ms at 1,500. The disk has an alibi. This is a CPU story: FairGate climbs from 70% to 132% of one vCPU, and Prometheus, Grafana, Alertmanager, container logging and the kernel fight over what remains. Past that point, throughput grows only because bigger batches spread each commit’s cost over more events, and batches stop growing much at around 160.

Extra producers beyond the plateau buy you nothing except a longer line. From 500 producers to 1,000, every latency percentile roughly doubles, exactly as Little’s law says it must.

  • 99th percentile
  • Median
0 ms40 ms80 ms120 ms160 ms1005001,0001,500Connected producers100 producers: 99th percentile 10.7 ms100 producers: median 6.2 ms250 producers: 99th percentile 26.1 ms250 producers: median 8.7 ms500 producers: 99th percentile 51.9 ms500 producers: median 16.7 ms1,000 producers: 99th percentile 98.9 ms1,000 producers: median 36.5 ms1,500 producers: 99th percentile 143.9 ms1,500 producers: median 60.5 ms144 ms61 ms
ACK latency against concurrency. Once throughput has flattened, more concurrency simply becomes waiting time.

One number here still bothers me. At 1,500 producers, clients measure a mean of 62.5 ms, but the gateway’s own timer, which starts when it reads a frame, says 25.6 ms. So about 37 ms pass before FairGate even looks at the event. The likely culprit is scheduling delay on a CPU-saturated host, plus a smaller share in the client, but “likely” is doing real work in that sentence. A frame-arrival timestamp on the server would settle it.

Open loop: exactly as boring as promised

The open-loop sweep is where a system should look boring, and this one obliged. FairGate accepted exactly what it was offered at every step, all the way to 20,000 events per second, the highest the test went. I didn’t find the ceiling, which will take a bigger sweep. The median stayed between 4.6 and 7.2 ms up to 15,000 events per second and reached 14.6 ms at 20,000. The floor is pleasingly explicable: a lone request waits up to 2 ms for company, then pays an fsync of about 2.9 ms.

  • 99th percentile
  • Median
3 ms10 ms30 ms100 ms300 ms1k2k4k6k8k10k12k15k20kOffered load (events/s)1,000/s offered: p99 7 ms1,000/s offered: accepted 1,000/s, median 4.6 ms2,000/s offered: p99 7 ms2,000/s offered: accepted 2,000/s, median 4.6 ms4,000/s offered: p99 8 ms4,000/s offered: accepted 4,000/s, median 4.8 ms6,000/s offered: p99 10 ms6,000/s offered: accepted 6,000/s, median 5 ms8,000/s offered: p99 22 ms8,000/s offered: accepted 8,000/s, median 5.6 ms10,000/s offered: p99 24 ms10,000/s offered: accepted 10,000/s, median 6.5 ms12,000/s offered: p99 31 ms12,000/s offered: accepted 12,000/s, median 6.5 ms15,000/s offered: p99 66 ms15,000/s offered: accepted 15,000/s, median 7.2 ms20,000/s offered: p99 196 ms20,000/s offered: accepted 20,000/s, median 14.6 ms15 ms196 ms
Open-loop sweep, 500 producers, 12-second windows, log scale. Accepted equals offered at every step.
See the numbers
Offered/sAccepted/sMedian (ms)p99 (ms)p99 from intended (ms)
1,0001,0004.678
2,0002,0004.678
4,0004,0004.889
6,0006,00051010
8,0008,0005.62222
10,00010,0006.52424
12,00012,0006.53131
15,00015,0007.26666
20,00020,00014.6196196

The tail climbs smoothly with the load, from 7 ms at 1,000 events per second to 66 ms at 15,000 and 196 ms at 20,000. I built the sweep to catch coordinated omission, by measuring each event from when it was supposed to be sent. It caught nothing. No send fell more than 1.5 ms behind schedule, so the p99 from the intended send time is identical at every step. A trap that comes back empty is mildly disappointing and entirely good news.

The tail that wasn’t there

On a durable system, tail latency usually comes from the disk, so I watched it second by second. Here is one isolation run: 285 producers sending ten events a second on a schedule and, from about 21 seconds in, 15 others flooding for 38 seconds.

  • 15 producers flooding
Paced producers: 99th percentile ACK latency3 ms10 ms30 ms100 msWAL fsync: 99th percentile0 ms5 ms10 msDisk writes (MiB/s): host, WAL at bottom050100gp3 baseline, 1250 s30 s60 s90 s
One isolation run (repeat 1). The flood (shaded) barely moves the paced producers’ latency, the WAL’s fsync stays flat, and the gateway’s disk writes stay well under the gp3 volume’s 125 MiB/s baseline.

The flooders got 172 events per second each, seventeen times a paced producer’s rate, and a median of 190 ms for their trouble. The paced producers barely noticed:

Paced producersMedianp99p99.9
No flood4.6 ms7.4 ms18 ms
Flood active4.8 ms8.6 ms20 ms

The disk had nothing to report either. In all 1,485 measured seconds there wasn’t a single storage stall, which is my name for a second in which the WAL fsync’s p99 goes past 40 ms and four times its usual value. The gateway typically wrote 11–16 MiB/s and peaked at 66–102 MiB/s, all under the volume’s baseline, and the WAL itself was only 1–8 MiB/s of that. The rest is Prometheus, container logs and the operating system. The fsync p99 stayed at its load-dependent baseline: 4–8 ms at low load, and up to 29 ms at 1,500 producers, where batches are largest.

The ClickHouse machine is a different story. It sat idle on disk most of the time, then wrote in bursts of up to 183 MiB/s as inserts landed and parts merged, more than the gateway volume’s entire baseline. On its own disk, that’s nobody else’s problem, which is exactly why it lives there.

Surprise two: where fairness actually comes from

The mixed scenario is the rude one. Of 750 producers, 40 misbehave in six different ways: bursting, flooding, sending slowly, disconnecting constantly, sending malformed events, sending oversized ones. On top of that, 426 otherwise polite producers pipeline 50 events at a time during two synchronised waves. The question is simple: who pays?

  • Median
  • 99th percentile
10 ms30 ms100 ms300 ms1 s3 sWell-behavedWell-behaved: 99th percentile 84 msWell-behaved: median 25.3 msKeeps reconnectingKeeps reconnecting: 99th percentile 94.1 msKeeps reconnecting: median 29.6 msSends malformed eventsSends malformed events: 99th percentile 94.5 msSends malformed events: median 30.1 msSends oversized eventsSends oversized events: 99th percentile 94.9 msSends oversized events: median 30.1 msSends slowlySends slowly: 99th percentile 95.4 msSends slowly: median 28.2 msPipelines in burstsPipelines in bursts: 99th percentile 895 msPipelines in bursts: median 321.2 msFloodsFloods: 99th percentile 1,767.2 msFloods: median 731.1 ms
Latency per behaviour during the misbehaviour phase, log scale. Five groups sit within a few milliseconds of each other. The two that pipeline wait behind their own frames.
See the numbers
BehaviourMedian (ms)p99 (ms)
Well-behaved25.384
Keeps reconnecting29.694.1
Sends malformed events30.194.5
Sends oversized events30.194.9
Sends slowly28.295.4
Pipelines in bursts321.2895
Floods731.11,767.2

The answer is beautifully specific. Well-behaved producers saw a median of 25 ms. Producers that disconnected, sent garbage or dawdled saw 28–30 ms, so bad manners alone cost them a few milliseconds and cost everyone else nothing. Throughput across the 710 well-behaved producers was almost perfectly even, with a Jain’s fairness index of 0.9997, where 1 is perfect. And the ones pipelining? Bursters waited a median of 321 ms and flooders 731 ms. The queue they were stuck in was their own.

The waves tell the same story. While 426 producers pipelined at once, throughput rose to about 28,000 events per second, because pipelined frames are cheaper per event. The participants waited about 700 ms, their own queue again, while everyone else carried on at 34 ms.

The flood itself got very little for its trouble: about 62 events per second against a polite producer’s 31, at about 29 times the latency. In the isolation scenario, on a quieter host, flooders reached 172 events per second with a median of 190 ms, while the 285 paced producers carried on at 4.8 ms.

Time to collect my prize, then. A token bucket, bounded queues and DRR, all doing their jobs. Except I checked the counters. Across all 24 runs, not one event was rate limited. Not one was refused for overload. Sampled once a second, the fullest queue never got past 1% of its capacity. So I went and read what the scheduler actually does with an event:

err := scheduler.Run(schedulerCtx, queues, func(event wire.Event) error {
    metrics.EventProcessed(event.ProducerId)
    logger.Info("Event processed by DRR",
        "producer_id", event.ProducerId,
        "seq", event.Seq,
    )
    return nil
})
The scheduler’s callback, from internal/server/server.go.

It increments a counter and writes a log line. The ACK doesn’t wait for any of this, because it is sent right after the enqueue, and the shipper reads directly from the WAL. In this design, DRR cannot affect ACK latency unless a queue fills, and in these tests none did. It can’t be what protected anyone. I had built a very fair way of writing log lines.

The real protection is two things I didn’t design as fairness features at all:

  1. Everyone waits in the same group commit. There is no separate fast lane to starve. One fsync covers whoever happens to be waiting.
  2. Each connection is served one event at a time. A window of k pipelined frames waits about k service times, in its own socket, delaying nobody else. You can’t push in front of other people when your own frames are the queue.

The second one explains the numbers too. A connection can’t exceed about one event per service time: 16.8 ms of ingest time in mixed is about 60 events per second, and 4.6 ms in isolation is about 220. Both sit well under the 500-per-second admission limit, so a single connection can never reach it.

I don’t want to overcorrect. This doesn’t make the rate limiter or the overload controller decoration. They exist for the cases this suite never created: one producer with many connections, much faster storage, a throttled shipper. But that’s precisely the point. An untested safety net isn’t a safety net yet, it’s a hypothesis. And DRR’s fairness on the delivery side wasn’t measured at all. Those need tests of their own.

While we’re being honest: the malformed events

The gateway’s handling of bad input was reassuringly dull. Every event with an empty or missing identifier was answered invalid, the connection stayed usable, and every frame over the WAL’s 1 MiB record limit was refused while everything from 0.5 to 1 MiB got through. No connection and no event failed on the client.

InputSentAcceptedAnswered invalid
Empty producer_id1440144
Empty event_type1710171
Missing event_type1410141
seq = 01291290
Frame over 1 MiB1500150
Frame of 0.5–1 MiB72720

You’ll notice 129 “malformed” events with seq = 0 were accepted. I spent a moment worrying about this before remembering that seq is an unsigned integer and zero is a perfectly good one. FairGate was right. My test was wrong. I’m choosing to count this as the gateway passing an exam I failed.

Surprise three: the bug that only appeared at scale

Everything so far is about accepting events. Getting them into ClickHouse is the shipper’s job, and FairGate’s default shipper was built for a gentler life. It sent 200 events per batch, one batch at a time, and every batch paid a fixed toll: two checkpoint commits, each with a file fsync and a directory fsync, an existence query to keep delivery exactly-once, and a synchronous insert. Small batches pay that toll constantly. So before this suite I changed the shipper in three steps on an experimental branch, and measured each one on the two heaviest scenarios, with ClickHouse on its own machine:

Step B is where it got interesting. Delivery at 1,500 producers fell to 540 events per second. Every run showed six or seven failed insert attempts, spaced out by the growing retry back-off: one batch, failing over and over. The ClickHouse client writes bound values into the query text, so the existence check for 5,000 IDs of 64 characters came to about 335 KB, comfortably over ClickHouse’s default max_query_size of 256 KiB, or about 3,900 IDs. Batches that happened to be smaller got through. Full ones never did. And nothing in FairGate’s logs said why, because failed inserts only incremented a metric.

Chunking the query fixed it outright: every batch succeeded, and delivery rose to 11,409 events per second at 1,500 producers and 16,221 in mixed. Reading ahead added another 7% to 9%, to 12,470 and 17,317.

Shipper stepDelivered/s, 1,500 producersDelivered/s, mixedPeak lag, 1,500 (MB)
B: Batches of 5,0005403,530279
C: + existence check in chunks of 1,00011,40916,221119
D: + read-ahead and merged commits12,47017,31797

Those runs used two load generators, which were at the edge of their headroom around 21,000 events per second, so they understate ingest. With a third, the same build accepted 23,245. Step D is what every other number in this post ran on.

Surprise four: the number you should actually quote

So here is the shipper in the full suite, against what the gateway accepted over the same period. This is the chart that made me stop quoting the big number with quite so much enthusiasm.

  • Accepted into the WAL
  • Delivered to ClickHouse
05k10k15k20k25k1005001,0001,500Connected producers100 producers: 9,678 events/s accepted100 producers: 9,678 events/s delivered (100%)250 producers: 17,973 events/s accepted250 producers: 17,945 events/s delivered (100%)500 producers: 21,370 events/s accepted500 producers: 20,635 events/s delivered (97%)1,000 producers: 22,109 events/s accepted1,000 producers: 18,702 events/s delivered (85%)1,500 producers: 22,137 events/s accepted1,500 producers: 13,839 events/s delivered (63%)22,13713,839
Accepted and delivered events per second during the load, mean of three repeats. Delivery keeps pace up to 500 producers, then falls behind. The share delivered is of all events accepted during the load.
See the numbers
ScenarioAccepted/sDelivered/sShare deliveredPeak lag (MB)
scale-1009,6789,678100%1
scale-25017,97317,945100%4
scale-50021,37020,63597%13
scale-100022,10918,70285%49
scale-150022,13713,83963%113
mixed21,36818,24285%91
isolation3,6743,668100%1
sweep8,5908,28596%17

Up to 500 producers, delivery keeps up: every event at 250 producers and 97% at 500, at up to 20,635 events per second, with backlogs of 1–13 MB. Past that it falls behind: 85% at 1,000 producers, 63% at 1,500 (13,839 per second) and 85% in mixed, with the backlog peaking at 113 MB.

The bottleneck isn’t ClickHouse, which was never more than 60% busy and only 24% at 1,500 producers. It is the shipper’s insert step, existence queries plus insert, one after another. At 1,500 producers each full step took about 360 ms, a ceiling of about 13,900 events per second, and delivery landed almost exactly on it. In mixed, 271 ms per step works out to about 18,500, against 18,242 delivered. Delivery also slows at 1,500 producers because the shipper lives on the gateway and competes with ingest for a CPU that is already 91% busy. The measurements don’t separate the existence check from the insert, so I don’t yet know which half to attack first.

And at peak it isn’t keeping up. At 1,500 producers the backlog grows by about 2.8 MB every second, enough to fill the overload controller’s 1 GiB limit in about six minutes, at which point it starts refusing everything. None of my runs lasted that long, which is convenient for the charts and not at all for the truth.

The limits of this work deserve some honesty too. These numbers come from one configuration in one availability zone, with default settings apart from the shipper’s batch size, so absolute numbers will differ elsewhere. The build includes shipper changes that aren’t in the main repository yet. Prometheus, Alertmanager and Grafana shared the gateway’s CPU and disk, so FairGate alone would have a little more headroom. The sweep stopped before FairGate did. Runs lasted 40 to 135 seconds, too short to see WAL rotation or ClickHouse at steady state. Events were synthetic, about 325 bytes each. And crashes, outages and failure injection were out of scope here. Exactly-once delivery has its own tests and its own chaos tooling, but this benchmark says nothing about them.

Still, I came away with something better than a big number. I went in wanting to show that the scheduler worked. I came out knowing that the disk wasn’t my bottleneck, my delivery path was my real capacity, a batch size can hide a bug that no unit test will find, and the fairness I’d built an algorithm for was being delivered, mostly, by a plain loop that reads one frame at a time.

I’d call that a successful benchmark. It just took me a while to stop being annoyed by it.