Operate · 03

How fast is it?

The latest measured run, what it measures, and how to run it yourself.

The benchmark runs examples/goblin-pizza.ts, which uses methods, derived stats, timers, worker leases, retries and fencing. It starts its own fresh local cluster and never touches an existing one.

LATEST MEASURED RUN ·

A busy day in the garden.

Global customer calls / s
54,754
Independent Raft groups
8 × 3 replicas

Reads2,303,662 calls

Customer reads: 2,303,662 calls; p50 1.03 ms, p99 11.1 ms, max 1.16 s. Log axis from 0.0456 ms to 81.1 ms; 0.1% of calls fall outside it.0.1 ms1 ms10 ms0.0456 ms–0.0535 ms: 0.043% of calls (989)0.0535 ms–0.0628 ms: 0.2% of calls (3,471)0.0628 ms–0.0736 ms: 0.3% of calls (6,277)0.0736 ms–0.0863 ms: 0.4% of calls (9,286)0.0863 ms–0.101 ms: 0.6% of calls (13,559)0.101 ms–0.119 ms: 0.8% of calls (19,268)0.119 ms–0.139 ms: 1.1% of calls (26,215)0.139 ms–0.163 ms: 1.5% of calls (33,934)0.163 ms–0.191 ms: 1.8% of calls (41,885)0.191 ms–0.224 ms: 2.2% of calls (50,101)0.224 ms–0.263 ms: 2.5% of calls (58,569)0.263 ms–0.308 ms: 2.9% of calls (66,518)0.308 ms–0.362 ms: 3.2% of calls (74,820)0.362 ms–0.424 ms: 3.7% of calls (84,508)0.424 ms–0.497 ms: 4.2% of calls (95,783)0.497 ms–0.583 ms: 4.7% of calls (108,766)0.583 ms–0.684 ms: 5.2% of calls (120,714)0.684 ms–0.802 ms: 5.6% of calls (129,799)0.802 ms–0.94 ms: 5.8% of calls (133,282)0.94 ms–1.1 ms: 5.9% of calls (134,807)1.1 ms–1.29 ms: 5.7% of calls (131,737)1.29 ms–1.52 ms: 5.6% of calls (128,520)1.52 ms–1.78 ms: 5.5% of calls (126,023)1.78 ms–2.08 ms: 5.5% of calls (126,716)2.08 ms–2.44 ms: 5.4% of calls (124,564)2.44 ms–2.86 ms: 4.9% of calls (113,544)2.86 ms–3.36 ms: 4.1% of calls (94,796)3.36 ms–3.94 ms: 3.1% of calls (71,309)3.94 ms–4.62 ms: 2.1% of calls (48,856)4.62 ms–5.42 ms: 1.5% of calls (34,942)5.42 ms–6.35 ms: 1.0% of calls (24,020)6.35 ms–7.45 ms: 0.7% of calls (17,261)7.45 ms–8.73 ms: 0.6% of calls (12,704)8.73 ms–10.2 ms: 0.4% of calls (9,398)10.2 ms–12 ms: 0.3% of calls (6,880)12 ms–14.1 ms: 0.2% of calls (5,327)14.1 ms–16.5 ms: 0.2% of calls (4,083)16.5 ms–19.4 ms: 0.1% of calls (2,435)19.4 ms–22.7 ms: 0.068% of calls (1,565)22.7 ms–26.6 ms: 0.058% of calls (1,329)26.6 ms–31.2 ms: 0.048% of calls (1,106)31.2 ms–36.6 ms: 0.031% of calls (722)36.6 ms–42.9 ms: 0.021% of calls (484)42.9 ms–50.3 ms: 0.011% of calls (251)50.3 ms–59 ms: 0.0043% of calls (98)59 ms–69.2 ms: 0.0045% of calls (103)69.2 ms–81.1 ms: 0.000043% of calls (1)p50 1.03 msp99 11.1 ms

Writes988,978 calls

Customer writes: 988,978 calls; p50 107 ms, p99 213 ms, max 1.01 s. Log axis from 34.5 ms to 910 ms; 0.085% of calls fall outside it.50 ms100 ms200 ms500 ms34.5 ms–37 ms: 0.030% of calls (299)37 ms–39.6 ms: 0.055% of calls (540)39.6 ms–42.5 ms: 0.1% of calls (1,187)42.5 ms–45.5 ms: 0.2% of calls (2,160)45.5 ms–48.8 ms: 0.4% of calls (3,886)48.8 ms–52.4 ms: 0.7% of calls (7,285)52.4 ms–56.1 ms: 1.4% of calls (13,883)56.1 ms–60.2 ms: 2.4% of calls (23,456)60.2 ms–64.5 ms: 3.3% of calls (33,042)64.5 ms–69.2 ms: 4.2% of calls (41,634)69.2 ms–74.2 ms: 4.6% of calls (45,875)74.2 ms–79.5 ms: 4.6% of calls (45,896)79.5 ms–85.2 ms: 4.2% of calls (41,098)85.2 ms–91.4 ms: 4.5% of calls (44,465)91.4 ms–98 ms: 6.6% of calls (64,991)98 ms–105 ms: 9.9% of calls (97,700)105 ms–113 ms: 11.8% of calls (116,900)113 ms–121 ms: 11.7% of calls (115,391)121 ms–129 ms: 10.0% of calls (99,025)129 ms–139 ms: 7.1% of calls (69,777)139 ms–149 ms: 4.4% of calls (43,765)149 ms–160 ms: 3.0% of calls (29,675)160 ms–171 ms: 1.6% of calls (15,449)171 ms–183 ms: 1.1% of calls (10,546)183 ms–197 ms: 0.6% of calls (6,243)197 ms–211 ms: 0.4% of calls (4,165)211 ms–226 ms: 0.3% of calls (2,914)226 ms–242 ms: 0.1% of calls (1,135)242 ms–260 ms: 0.1% of calls (1,072)260 ms–279 ms: 0.077% of calls (764)279 ms–299 ms: 0.066% of calls (657)299 ms–320 ms: 0.062% of calls (617)320 ms–343 ms: 0.019% of calls (188)343 ms–368 ms: 0.046% of calls (454)368 ms–395 ms: 0.016% of calls (160)395 ms–423 ms: 0.026% of calls (256)423 ms–454 ms: 0.0079% of calls (78)454 ms–486 ms: 0.014% of calls (143)599 ms–643 ms: 0.00030% of calls (3)643 ms–689 ms: 0.018% of calls (182)689 ms–739 ms: 0.022% of calls (220)739 ms–792 ms: 0.011% of calls (109)792 ms–849 ms: 0.038% of calls (376)849 ms–910 ms: 0.048% of calls (477)p50 107 msp99 213 ms
Customer call latency, merged across groups. Each log time axis spans p0.1 to p99.9 of its calls; bar height is a bin's share of calls, and lines mark p50 and p99.
Show as a table
Customer reads, all groups
LatencyCallsShareCumulative
0.0332 ms–0.0389 ms6<0.01%0.00%
0.0389 ms–0.0456 ms1230.01%0.01%
0.0456 ms–0.0535 ms9890.04%0.05%
0.0535 ms–0.0628 ms3,4710.15%0.20%
0.0628 ms–0.0736 ms6,2770.27%0.47%
0.0736 ms–0.0863 ms9,2860.40%0.87%
0.0863 ms–0.101 ms13,5590.59%1.46%
0.101 ms–0.119 ms19,2680.84%2.30%
0.119 ms–0.139 ms26,2151.14%3.44%
0.139 ms–0.163 ms33,9341.47%4.91%
0.163 ms–0.191 ms41,8851.82%6.73%
0.191 ms–0.224 ms50,1012.17%8.90%
0.224 ms–0.263 ms58,5692.54%11.45%
0.263 ms–0.308 ms66,5182.89%14.33%
0.308 ms–0.362 ms74,8203.25%17.58%
0.362 ms–0.424 ms84,5083.67%21.25%
0.424 ms–0.497 ms95,7834.16%25.41%
0.497 ms–0.583 ms108,7664.72%30.13%
0.583 ms–0.684 ms120,7145.24%35.37%
0.684 ms–0.802 ms129,7995.63%41.00%
0.802 ms–0.94 ms133,2825.79%46.79%
0.94 ms–1.1 ms134,8075.85%52.64%
1.1 ms–1.29 ms131,7375.72%58.36%
1.29 ms–1.52 ms128,5205.58%63.94%
1.52 ms–1.78 ms126,0235.47%69.41%
1.78 ms–2.08 ms126,7165.50%74.91%
2.08 ms–2.44 ms124,5645.41%80.32%
2.44 ms–2.86 ms113,5444.93%85.25%
2.86 ms–3.36 ms94,7964.12%89.36%
3.36 ms–3.94 ms71,3093.10%92.46%
3.94 ms–4.62 ms48,8562.12%94.58%
4.62 ms–5.42 ms34,9421.52%96.09%
5.42 ms–6.35 ms24,0201.04%97.14%
6.35 ms–7.45 ms17,2610.75%97.89%
7.45 ms–8.73 ms12,7040.55%98.44%
8.73 ms–10.2 ms9,3980.41%98.85%
10.2 ms–12 ms6,8800.30%99.14%
12 ms–14.1 ms5,3270.23%99.38%
14.1 ms–16.5 ms4,0830.18%99.55%
16.5 ms–19.4 ms2,4350.11%99.66%
19.4 ms–22.7 ms1,5650.07%99.73%
22.7 ms–26.6 ms1,3290.06%99.78%
26.6 ms–31.2 ms1,1060.05%99.83%
31.2 ms–36.6 ms7220.03%99.86%
36.6 ms–42.9 ms4840.02%99.88%
42.9 ms–50.3 ms2510.01%99.90%
50.3 ms–59 ms98<0.01%99.90%
59 ms–69.2 ms103<0.01%99.90%
69.2 ms–81.1 ms1<0.01%99.90%
81.1 ms–95.1 ms114<0.01%99.91%
112 ms–131 ms28<0.01%99.91%
131 ms–153 ms17<0.01%99.91%
153 ms–180 ms1<0.01%99.91%
180 ms–211 ms2<0.01%99.91%
211 ms–247 ms2<0.01%99.91%
247 ms–290 ms3<0.01%99.91%
290 ms–340 ms3<0.01%99.91%
340 ms–399 ms10<0.01%99.91%
399 ms–467 ms32<0.01%99.91%
467 ms–548 ms66<0.01%99.92%
548 ms–643 ms1290.01%99.92%
643 ms–753 ms1360.01%99.93%
753 ms–884 ms2900.01%99.94%
884 ms–1.04 s8850.04%99.98%
1.04 s–1.21 s4900.02%100%
Customer writes, all groups
LatencyCallsShareCumulative
13.9 ms–14.9 ms1<0.01%0.00%
14.9 ms–16 ms1<0.01%0.00%
16 ms–17.2 ms2<0.01%0.00%
17.2 ms–18.4 ms1<0.01%0.00%
18.4 ms–19.7 ms3<0.01%0.00%
19.7 ms–21.2 ms2<0.01%0.00%
21.2 ms–22.7 ms6<0.01%0.00%
22.7 ms–24.3 ms14<0.01%0.00%
24.3 ms–26.1 ms31<0.01%0.01%
26.1 ms–28 ms36<0.01%0.01%
28 ms–30 ms24<0.01%0.01%
30 ms–32.2 ms49<0.01%0.02%
32.2 ms–34.5 ms1720.02%0.03%
34.5 ms–37 ms2990.03%0.06%
37 ms–39.6 ms5400.05%0.12%
39.6 ms–42.5 ms1,1870.12%0.24%
42.5 ms–45.5 ms2,1600.22%0.46%
45.5 ms–48.8 ms3,8860.39%0.85%
48.8 ms–52.4 ms7,2850.74%1.59%
52.4 ms–56.1 ms13,8831.40%2.99%
56.1 ms–60.2 ms23,4562.37%5.36%
60.2 ms–64.5 ms33,0423.34%8.70%
64.5 ms–69.2 ms41,6344.21%12.91%
69.2 ms–74.2 ms45,8754.64%17.55%
74.2 ms–79.5 ms45,8964.64%22.19%
79.5 ms–85.2 ms41,0984.16%26.35%
85.2 ms–91.4 ms44,4654.50%30.84%
91.4 ms–98 ms64,9916.57%37.42%
98 ms–105 ms97,7009.88%47.30%
105 ms–113 ms116,90011.82%59.12%
113 ms–121 ms115,39111.67%70.78%
121 ms–129 ms99,02510.01%80.80%
129 ms–139 ms69,7777.06%87.85%
139 ms–149 ms43,7654.43%92.28%
149 ms–160 ms29,6753.00%95.28%
160 ms–171 ms15,4491.56%96.84%
171 ms–183 ms10,5461.07%97.91%
183 ms–197 ms6,2430.63%98.54%
197 ms–211 ms4,1650.42%98.96%
211 ms–226 ms2,9140.29%99.25%
226 ms–242 ms1,1350.11%99.37%
242 ms–260 ms1,0720.11%99.48%
260 ms–279 ms7640.08%99.55%
279 ms–299 ms6570.07%99.62%
299 ms–320 ms6170.06%99.68%
320 ms–343 ms1880.02%99.70%
343 ms–368 ms4540.05%99.75%
368 ms–395 ms1600.02%99.76%
395 ms–423 ms2560.03%99.79%
423 ms–454 ms780.01%99.80%
454 ms–486 ms1430.01%99.81%
599 ms–643 ms3<0.01%99.81%
643 ms–689 ms1820.02%99.83%
689 ms–739 ms2200.02%99.85%
739 ms–792 ms1090.01%99.86%
792 ms–849 ms3760.04%99.90%
849 ms–910 ms4770.05%99.95%
910 ms–976 ms3390.03%99.98%
976 ms–1.05 s1590.02%100%

Replica-local reads: lag is allowed. 70% reads / 30% mutations · HTTP/2 · 60.1 measured seconds.

Run passed. 8/8 group audits passed. 8 injected leader failures; quorum recovery 560–704 ms.

Apple M5 Pro; all replicas and load generators share one machine. Completed customer calls use the union measurement window; retries, worker traffic, and explicit replays do not inflate throughput. Reads may be stale; mutations and audits retain fresh checks.

Charts & every group →Raw measurements ↓

Run it yourself

Run and stress
npm ci
npm run build
cargo build --release --bin flower
npm run bench
cargo build --release --bin flower --bin flower-bench-driver
npm run bench:stress

# Customize the base command, with one copy of each flag:
npm run bench -- --duration 30 --concurrency 16 --workers 4 \
  --max-orders 64 --drain 180 --json bench/results/custom.json
npm run bench -- --help
  • What counts: successful customer calls (shop reads, orders, tips). A retried call counts once. Worker polls, deliveries, replays, setup and the audit don't count.
  • The workload is closed-loop: each customer waits for one call before starting the next. The mix is 30% orders, 40% reads and 30% tips until the order cap, then orders turn into reads.
  • Everything runs on one machine, servers and load generator together. Your production numbers depend on your workload and hardware.
  • The run fails on unexpected errors, an unfinished drain, bad fencing or replays, or a failed business check.
  • Reports go to bench/results/ as JSON and HTML, with the settings and binary used.

The base command starts one three-node group. bench:stress runs eight independent groups, each with its own customers, workers and leader crash. Its preset sets 256 customers per group, serial writer preparation and a 1,024-request writer queue, overriding the normal defaults. Environment variables override the preset. See the benchmark guide for all options and the measurement analysis for the current result.

To publish a new result to this page, run inside nix develop, build first and leave profiling off:

Measure independent Raft groups
npm ci
cargo build --release --locked --bin flower --bin flower-bench-driver
npm run bench:stress
node scripts/publish-bench-results.mjs bench/results/latest.json
node scripts/publish-bench-results.mjs --check

Watch the live dashboard

The dashboard shows one tenant's stores, orders, oven timers, drone leases and rankings from a single watched pizza.dashboard({tenant}) value.

Run the live kitchen board
cargo build --release --bin flower
npm run demo:pizza
# Optional: fixed port, paused arrivals, stop automatically after two minutes
npm run demo:pizza -- --port 3030 --paused --duration 120

Open the printed URL. It starts a fresh three-node cluster with three tenants. Place orders, tip goblins, pause arrivals or crash the leader. Ctrl+C stops everything and deletes its data.

The dashboard reads replica-local data. It can lag with no bound, and a reconnect may show an older revision. The benchmark's shop reads use pizza.shop.local the same way; pass --read-consistency fresh to measure fresh reads instead.

Compare HTTP/1.1 and HTTP/2

Method calls use HTTP/1.1 by default. Add --http2 to use pooled HTTP/2. Node-to-node traffic always uses HTTP/2.

HTTP/2 methods with leader failover
npm run bench -- --http2 --duration 30 --concurrency 32 --workers 4 \
  --max-orders 96 --chaos --json bench/results/http2.json

Keep the workload and binary the same when you compare. The benchmark reuses request IDs when it retries; the HTTP/2 client itself doesn't retry uncertain calls.

Profile before tuning

Profile under the same mixed load before changing settings. Skip the leader crash for a first comparison.

macOS · native stack profile
CARGO_PROFILE_RELEASE_DEBUG=1 cargo build --release --bin flower
npm run bench -- --duration 30 --concurrency 32 --workers 4 \
  --max-orders 96 --drain 180 \
  --cpu-profile bench/results/profile-mixed.sample.txt \
  --json bench/results/profile-mixed.json

# Batch timing for the same mixed workload (one group).
FLOWER_PROFILE_STORAGE=1 FLOWER_PROFILE_EVALUATOR=1 \
node bench/profile-mixed.mjs --http2 --duration 30 --concurrency 128 \
  --workers 4 --max-orders 96 --json /tmp/flower-mixed-profile.json
  • The first command samples native stacks on the initial leader with macOS /usr/bin/sample. It shows where threads spend time, not CPU percentages or TypeScript lines.
  • The second writes an extra *-groups.json with per-batch timings. Phases overlap, so don't add them up as wall time.
  • node bench/profile-otel.mjs takes the same flags and writes an HTML report of request, writer, storage and Raft timing. See the OpenTelemetry guide.
  • Profiling slows things down. Rerun without it before quoting throughput.

More options are in the profiling commands.

Run the checks

Run the test suite from the checkout:

Validation from the checkout
npm run check
cargo test
cargo build
node tests/e2e.mjs
node tests/e2e-http2.mjs
node tests/e2e-watch.mjs

Benchmarks clean up their temporary data when they stop; pass --keep-data to keep it for inspection.