Orisha vs Rust on Campfire's WebSocket Fanout

A benchmark is only interesting if it can hurt. So we picked one that could: Campfire, 37signals’ real chat product — Rails reference, a Rust/axum port, and our own port in Koru on Orisha. Real authentication, real SQLite, real Action Cable: subscriptions, signed stream names, turbo-stream frames, heartbeats, and a fanout that has to reach every connected client before the next post arrives.

Same app, same 169-message seed database, same protocol bytes, same load generator, same machine. The only variable is the stack.

The results — dedicated hardware

One 8-vCPU DigitalOcean droplet (s-8vcpu-16gb-amd), each server pinned to cores 0–3, the load generator pinned to 4–7, loopback transport, a fresh copy of the same 169-row seed per cell, cells interleaved. Every run ready=N, failed=0, complete==posted. Delivered counts a post only when every subscriber’s client received it. Orisha is ahead in eight of eight paired cells — ~1.6× at N≤200, ~1.2× at the top of the sweep.

(One honesty note: at N=50 the Orisha cells ran so hot — ~2,900 posts/s — that the pinned four-core loadgen itself dropped frames; the 2,894 above is the cleanest cell, and the dropped cells are floors, not ceilings. The loadgen burns more CPU than either server at this workload: mid-run it sits at ~2.5 cores while each server holds ~1.8.)

More users means fewer messages — for everyone

delivered/s is a quotient, not a capability. One broadcast to N subscribers costs N frame writes, so when a box has a fixed frame budget — call it F — the most honest model of this benchmark is:

delivered/s ≈ min(post rate, F / N)

Three weeks ago Orisha’s F at 500 clients was ~87K frames/s — the broadcast path spent the budget on copies and locks — and the fanout was the bottleneck by a mile. The six mechanisms below lifted F past the post rate, which is why every complete cell now reports delivered == posted: the frame budget stopped being the bound. What the droplet table measures is the other term of the minimum — how fast each stack can take a post from HTTP request to committed broadcast — and there the gap is a mechanism, not noise.

The capacity law didn’t disappear; it moved. On the laptop, an interleaved N-sweep on grown databases showed both stacks converging on a ~250–270K frames/s plateau — Orisha 523 vs Rust 496 delivered/s at N=500, inside noise — and at the time that read as parity. The dedicated box removed the shared-box confound (the loadgen there burns more CPU than the server it feeds), and on it the same workload lands differently: Rust’s per-post cost climbs faster with subscriber count, its delivered curve sits below ours at every N, and the “parity” turns out to have been two different bounds sitting on the same curve.

The laptop sweep, for context: same protocol, one interleaved pair per cell, identically grown databases.

Finding 1: push vs pull inverts at scale

The first fanout shape was the obvious one: the broadcast thread walks the subscriber list and calls writev on each socket itself. At 100 clients it beats every alternative — no queue hop, no wake. At 500 it collapses: clients can’t drain 9.4KB frames as fast as the thread can offer them, sockets fill, and the producer sits inside a lock holding everyone else’s sends.

Owner-pull inverts exactly where push dies. The producer only appends each frame to each connection’s pending queue and marks the owning reactor write-ready; the reactor — the only thread that ever touches its own sockets — writes on readiness. N=500 went 174 → ~600. The same change lost at N=100 when we first tried it months ago — the economics genuinely invert, and which side of the line you’re on is a property of socket writability, not of the code.

Finding 2: a broadcast is one frame, not N copies

Owner-pull still copied: pending was a byte array, so every broadcast physically memcpy’d ~9.4KB into each of 500 queues — ~4.7MB per post — plus realloc churn under a per-connection mutex. The profile read memmove 1,210 samples, mmap/munmap ~600, ulock_wait ~23k.

The fix is the shape every mature fanout uses and we hadn’t: one refcounted Frame per broadcast — header and payload allocated once — and a queue of {frame, offset} segments per connection. Producers pay an incref and a pointer push; owners writev straight from the shared bytes and advance the offset on partial writes. memmove fell to ~200 samples and the delivery p99 collapsed ~470ms → ~27ms.

Finding 3: the producer handoff must not take a mutex

Zero-copy moved the cost again: producers still locked each connection’s ws_mu to push a segment while owners locked the same mutex to drain — 500 acquire/release pairs ping-ponging between 4 producers and 12 owners per broadcast, ~42k lock-wait samples.

So the queue went lock-free: a Vyukov-style single-producer ring per connection (the drainer shards fds, fd % K, so every connection has exactly one producer and ordering is free), an atomic armed flag so a conn wakes at most once per drain epoch, and an atomic byte counter enforcing the pending bound. Lock waits fell ~40×. Throughput barely moved — the lock wasn’t the throughput limiter, it was the latency poison — but the tail and the CPU bill both dropped.

Finding 4: the collector should index, not scan

Every broadcast also found its subscribers by scanning a 4,096-slot connection registry under a global lock — four shards, every broadcast, ~37M slot-checks a second at N=100 rates. The registry now keeps a (stream, identifier) → fd-list map maintained on subscribe/unsubscribe. Collect is a map lookup and a list copy; the scan is gone, and the profile shows zero mutex samples where reg_mu used to live. The lingering N=100 delivery tail (~170ms at p99) died with it.

Finding 5: a request should never queue behind a drain

The 12 owner threads do two jobs: answer HTTP requests and drain WebSocket queues — a POST’s 0.41ms of real work measured 5.2ms at 500 clients because it landed behind in-flight drains in the same kevent batch. The worker loop now runs two passes per batch — reads and requests first, write-drains second. Post latency improved ~0.2ms: real, but not the whole story — the rest of the wait is inherent to sharing threads, and splitting request handling from delivery is the one structural lever left.

Finding 6: the allocator is a syscall in disguise

The last profile had no locks, no scans, no copies — and ~2,000 mmap/munmap samples. Every broadcast allocated its bucket list, its frame buffer, its job envelope, and its render scratch through the page allocator — every alloc and free a syscall pair. The drainer scratch moved to a retained arena, the fanout envelope to a slab freelist, the writer’s statements to prepare-once/reset-per-post. munmap on the hot path: ~16 per POST → 0.

What bounds it now

delivered == posted, drain ≈ 0 on every complete cell — the fanout keeps up, so the post rate is the throughput. That makes the post p50 columns the interesting ones, and they diverge: Orisha runs 1.2→8.5ms across the sweep while Rust runs 2.1→10.3ms, with the gap widening as subscribers accumulate. Both fanout architectures turned out to be the same three ideas — a refcounted shared frame, a bounded per-connection queue, one batched vectored drain — which is why the laptop’s aggregate ceilings matched. The difference is what a post pays per subscriber at broadcast time: the Rust port’s in-process adapter is a tokio::sync::broadcast ring per stream, and each send walks every receiver under the hub’s mutex; Orisha’s is an O(1) intrusive link per connection with the wake work deferred to the owning reactor. On fast laptop cores the difference stays small enough to hide inside cell noise. On shared cloud vCPUs it prices itself directly into the post round-trip — and under a closed-loop load generator, post latency is delivered throughput.

The remaining lever on our side is unchanged and now sharper: a request/delivery thread split, so a POST’s commit-and-broadcast never queues behind a drain. The request path itself is where the next round of specialization is already landing — the HttpArena entry is getting a handler-lite fast path that lets provably-pure route arms skip Request construction entirely. Different surface, same thesis: the cost left to remove lives on the request side of the socket, not the fanout side.

What we distrusted, and why you can trust the table

• Cross-machine fiction. Rust’s first quoted number (2,914/s) was measured on a droplet. Same box: ~2,200. Every number above is interleaved, same hardware, same minute.
• Three servers one port. Orisha sets SO_REUSEPORT; a failed pkill once left three different builds answering — every run after that started with a listener count. On the droplet it bit differently: pkill hadn’t finished inside a half-second sleep, the new server bound the same port beside the old, and connections split across two processes holding different databases — login minted a session on one, posts and subscriptions hit the other, and a whole cell of “benchmark” was instant 401s. That failure looked like a catastrophic WebSocket bug for a full evening before iperf and one clean N=5 run named the real walls.
• Finite wires. The first dedicated attempt put the loadgen and server on separate droplets — right instinct, wrong workload. Every client carries every broadcast: at 2,800 posts/s, N=10 alone needs ~2Gbps. iperf measured the link at 1.94Gbps; the per-connection backpressure cap did its job and dropped frames, and 42% “completion” recorded a NIC, not a bug. The capacity law eats networks too — which is why the comparison ran on one box, loopback, CPUs split instead of packets.
• A Debug binary. -O ReleaseFast was silently ignored by the compiler frontend — a day of early numbers ran unoptimized. That bug is now fixed in the toolchain (-O maps all four optimize modes or refuses loudly), which is what a benchmark is for.
• Fresh state. Session rows live in the database; a “fresh” seed invalidates cookies, and a stale WAL file once corrupted a copied database mid-run — twice, in fact: on the droplet a killed process left -wal/-shm sidecars beside the seed copy and the next server boot refused the “malformed” image. Every cell now deletes sidecars before copying the seed. The six-mechanism measurements ran identical 169-row seeds; the scaling sweep ran both stacks on identically grown databases, since absolute post cost drifts with table size.
• The subscription set is part of the claim. The loadgen scrapes the room page for stream names, and on the droplet two cells subscribed to whatever the scrape happened to catch: one got the sidebar’s Turbo::StreamsChannel entries instead of the room’s RoomMessagesChannel (five subscriptions, zero posts delivered), another got nothing at all (three subscriptions, 47K posts delivered to nobody). Both recorded “benchmark” cells against different workloads. The harness now requires the room channel’s signed stream before a cell may run — and every quoted cell reports subscriptions_per_client: 4.
• Shared vCPUs are still shared. The droplet is DigitalOcean’s basic tier — steal time measured ~0 during runs, but a shared vCPU is not a dedicated core. Where the comparison could be load-bearing we ran pairs, not singles: the N=300 and N=500 cells each have two-plus interleaved readings, and the ordering held in all of them.

The verdict

The oracle did what it was commissioned to do — it moved the bound and then named it. First the frame budget was the wall (174 delivered/s at N=500), and six mechanisms — owner-pull, shared frames, a lock-free handoff, a stream index, two-pass loops, allocator discipline — lifted it past the post rate. Then dedicated hardware asked the harder question, and the answer came back better than parity: Orisha delivers more complete messages per second at every client count we tested — ~1.6× below the knee, ~1.2× at 500 — because the cost left on the post→broadcast path is O(1) per subscriber where the Rust adapter’s is O(subscribers) under a mutex. Same app, same seed, same protocol, same box, and the gap is a mechanism, not a mystery.