Project 01 · Interactive deep dive

FairGate, from the inside.

A TCP ingestion gateway in Go that keeps one noisy producer from starving the rest, and makes every accepted event durable before acknowledging it.

Each chapter below is a small, working model of one part of the gateway. They follow the behaviour documented in the repository, with numbers scaled down so you can watch them happen.

Source on GitHub

Language: Go
Storage: Segmented WAL → ClickHouse
Operations: Prometheus · Grafana · Docker Compose
Status: Open source · MIT · beta

01Follow one event

Every exit from the pipeline answers the producer.

Each event takes the same path through the connection handler: decode, admit, check its identity, persist, enqueue. Wherever it stops, the producer gets an explicit ACK saying whether to move on, back off, or fix the request. Pick a scenario and watch where it ends up.

Event tracer

0 / 6 stages

A producer sends an event the gateway has never seen.

Decodepending
wire.DecodeEvent
Admissionpending
admission.Allow
Identitypending
dedupManager.PersistIfNew
WAL + syncpending
walLog.Append
Enqueuepending
queues.EnqueueContext
ACKpending
sendAck

Event frame · type 1

{
  "producer_id": "sensor-01",  "producer_epoch": "1",  "event_time": "2026-10-02T08:00:00Z",  "seq": 1,  "idx": 0,  "event_type": "temperature",  "payload": "{\"value\":24.7}"}

Server

Press play to send the event.

ACK frame · type 2

Waiting for the gateway…

Stage order, function names and ACK values from internal/server/server.go

02Admission

A flood is refused at the door.

Admission is the first layer of isolation. Each producer has its own token bucket: tokens refill at a steady rate up to a burst capacity, and every event spends one. With less than one token left, the event is refused immediately rather than buffered, so a noisy producer costs everyone else nothing.

Admission: token bucket

One producer

Refill rate4 tokens/s

Burst (capacity)10 tokens

Producer send rate8 events/s

Tokens in the bucket

10.0 / 10

Accepted: 0
Rejected · rate_limited: 0

Last 48 ACKs, oldest first

accepted rejected

Start a stream faster than the refill rate, or fire a burst bigger than the capacity, to see events refused.

Same algorithm as internal/admit/token_bucket.go · FairGate default: 500 events/s, burst 1,000, per producer · scaled down here

03Group commit

Durable, without one fsync per event.

An accepted ACK promises the event is on disk, and that takes a sync. Syncing every record separately would make each producer wait behind every other producer’s write. Instead, one writer goroutine batches whatever is waiting and covers the whole batch with a single sync.

Durability: group commit

FairGate

Concurrent Append() calls6

Time each caller waits before Append() returns

Sync() calls: 1
Slowest caller waits: 1 sync

A single writerLoop goroutine drains all 6 waiting requests, writes them, calls Sync() once, then releases every caller. However many handlers are appending, each waits about one sync. A request that arrives alone waits up to 2 ms for company.

Behaviour from docs/wal.md · time is in units of one disk sync, not measured milliseconds

04Crash recovery

Telling an interrupted write from corruption.

On restart, the WAL scans its segments in ID order and checks every record’s length and CRC32. The rules are deliberately asymmetric: a half-written record at the very end is expected after a crash, but anywhere else it means something is genuinely wrong. Break the log, then restart it.

Crash recovery

Segments 1–2 already reclaimed

segment 0003

segment 0004

segment 0005final · active

Inject a fault, then restart. Recovery scans segments in ID order, record by record.

Rules from docs/wal.md · click any record to flip a bit in it

05Fair scheduling

Shares follow weights, not volume.

Accepted events wait in per-producer bounded queues. A weighted Deficit Round Robin scheduler visits them in rounds and serves each backlogged producer in proportion to its weight. Turn on the flood and change the weights to see that a noisy producer only ever fills its own queue.

Weighted DRR

Three producers

Round 000

A
Share 0%
B
Share 0%
C
Share 0%

Press play or step to run the scheduler.

Illustrative · quantum 2 · capacity 12 · share over last 8 rounds

Same rules as the scheduler in internal/sched: quantum × weight credit per round, one event per credit, idle queues reset to zero, full queues block.

06Exactly once

Durable isn’t stored, until the shipper says so.

A separate shipper streams durable WAL records into ClickHouse from a checkpoint. Two layers keep storage exactly-once: a pending-batch record that makes every replay cover exactly the same range, and an event_id existence check that skips anything already stored. Step through a batch, then try to break it.

Shipping exactly once

Phase: idle

Write-ahead log · durable records

seg 3

e1▼
e2
e3

seg 4

seg 5

▼ checkpoint current batch pending (durable) stored

Shipper started at checkpoint e1.

ClickHouse · fairgate.events0 rows

event_idrows

No rows yet.

Shipped: 0 / 8
Max rows per event: 0

Try crashing between insert and commit, or losing the checkpoint after a few batches. The replay re-reads the same records, but every event still lands exactly once.

Semantics from docs/delivery-semantics.md · batches of 2, segments of 3 records, scaled down

Per the repository docs, the exactly-once guarantee is per event identity and assumes a single gateway writes each identity.

Read the real thing

FairGate on GitHub

NextThe paper, interactive