Customer goodput
6,825.4 / s
70% reads · 30% mutations among successful customer calls.
All-call latency
p50 2.06 ms · p99 176 ms. All load-phase logical calls, including worker traffic, replays, discovery, and retries.
Orders delivered
96 / 96
246 pizzas · 1,722 copper in delivery revenue.
Quorum recovery
592.9 ms
First recorded crash to a serving quorum; restart catch-up is tracked separately.
Replica-local customer previews: replication lag is allowed. The selected pizza.shop.local method uses coherent locally applied snapshots without a quorum fence. Data, code, and aliases may lag without a bound; requests across replicas can return decreasing revisions. Replica-local is the benchmark default; choose --read-consistency fresh to measure quorum-fenced previews. Mutations and the final pizza.world audit retain fresh semantics.
The customer's view
Successful work, precisely counted.
- Successful customer mix
- 70% reads / 30% mutations
- Successful customer calls
- 410,391
- Failed customer calls
- 0
- Measured load duration
- 60.13 s
- Load worker completions excluded
- 1,055
- Load replay completions excluded
- 12,424
| Load measurement | Successful completions | Goodput / s | Successful customer share |
|---|---|---|---|
| Primary customer reads | 287,005 | 4,773.3 | 70% |
| Primary customer mutations | 123,386 | 2,052.1 | 30% |
| All primary customer calls | 410,391 | 6,825.4 | 100% |
| All logical calls, including workers and replays | 423,870 | 7,049.6 | Includes non-customer traffic |
| Raw successful HTTP responses | 423,870 | 7,049.6 | Includes retries and non-customer traffic |
Counting definition: successful pizza.shop.local, pizza.order, and pizza.tip logical calls per measured load second. Transport retries count once when their customer call succeeds. Explicit .replay probes, worker claims and deliveries, polling, setup, audit, and every failed call contribute zero customer goodput. Actual successful read/mutation shares appear above.
The run verdict covers correctness and completion of any requested CPU profile. Customer goodput is an average over the recorded load phase; individual intervals can differ.
The rush, second by second
How the run unfolded
Observed completion rate
Each point summarizes the interval since the previous sample.
Cumulative HTTP tail latency
p95 of attempts since load began; not an interval percentile.
Orders placed and delivered
Acknowledgements observed by the benchmark driver.
Driver resident memory
Samples include the driver's runtime and retained workload records.
Lines connect observed samples; no measurements are interpolated into the underlying data. Throughput is an interval rate, while p95 is cumulative. The driver and database share a machine. Any crash marker is positioned relative to the start of measured load.
Recorded time samples
| Elapsed | Phase | HTTP attempts / s | HTTP success / s | Logical complete / s | Retries | HTTP failures | Cumulative p95 | Orders | Deliveries | Driver MiB |
|---|---|---|---|---|---|---|---|---|---|---|
| 1.07 s | load | 18.6 | 18.6 | 18.6 | 0 | 0 | 333.2 ms | 0 | 4 | 317.8 |
| 2.08 s | load | 3,542.9 | 3,542.9 | 3,542.9 | 0 | 0 | 368.1 ms | 96 | 12 | 328.2 |
| 3.19 s | load | 10,046.5 | 10,046.5 | 10,046.5 | 0 | 0 | 212.9 ms | 96 | 27 | 335.0 |
| 4.19 s | load | 5,708.4 | 5,708.4 | 5,708.4 | 0 | 0 | 202.6 ms | 96 | 43 | 336.9 |
| 5.19 s | load | 7,138.0 | 7,138.0 | 7,138.0 | 0 | 0 | 194.7 ms | 96 | 62 | 339.4 |
| 6.20 s | load | 6,228.1 | 6,228.1 | 6,228.1 | 0 | 0 | 190.9 ms | 96 | 76 | 340.8 |
| 7.20 s | load | 6,276.8 | 6,276.8 | 6,276.8 | 0 | 0 | 183.4 ms | 96 | 89 | 342.5 |
| 8.21 s | load | 6,937.7 | 6,937.7 | 6,937.7 | 0 | 0 | 171.1 ms | 96 | 94 | 345.7 |
| 9.21 s | load | 6,829.9 | 6,829.9 | 6,829.9 | 0 | 0 | 161.2 ms | 96 | 96 | 347.2 |
| 10.22 s | load | 6,588.5 | 6,588.5 | 6,588.5 | 0 | 0 | 158.0 ms | 96 | 96 | 348.9 |
| 11.22 s | load | 7,022.1 | 7,022.1 | 7,022.1 | 0 | 0 | 154.9 ms | 96 | 96 | 350.4 |
| 12.22 s | load | 7,265.4 | 7,265.4 | 7,265.4 | 0 | 0 | 151.8 ms | 96 | 96 | 352.0 |
| 13.22 s | load | 6,337.5 | 6,337.5 | 6,337.5 | 0 | 0 | 148.8 ms | 96 | 96 | 353.8 |
| 14.22 s | load | 5,580.8 | 5,580.8 | 5,580.8 | 0 | 0 | 151.8 ms | 96 | 96 | 357.6 |
| 15.22 s | load | 5,781.6 | 5,781.6 | 5,781.6 | 0 | 0 | 153.3 ms | 96 | 96 | 362.0 |
| 16.23 s | load | 7,038.4 | 7,038.4 | 7,038.4 | 0 | 0 | 153.3 ms | 96 | 96 | 363.6 |
| 17.23 s | load | 6,941.8 | 6,941.8 | 6,941.8 | 0 | 0 | 150.3 ms | 96 | 96 | 365.2 |
| 18.23 s | load | 7,288.8 | 7,288.8 | 7,288.8 | 0 | 0 | 148.8 ms | 96 | 96 | 366.9 |
| 19.23 s | load | 6,192.6 | 6,192.6 | 6,192.6 | 0 | 0 | 148.8 ms | 96 | 96 | 368.5 |
| 20.23 s | load | 7,678.6 | 7,678.6 | 7,678.6 | 0 | 0 | 145.9 ms | 96 | 96 | 370.1 |
| 21.23 s | load | 6,172.4 | 6,172.4 | 6,172.4 | 0 | 0 | 145.9 ms | 96 | 96 | 371.6 |
| 22.23 s | load | 6,466.2 | 6,466.2 | 6,466.2 | 0 | 0 | 147.4 ms | 96 | 96 | 373.2 |
| 23.23 s | load | 5,988.1 | 5,988.1 | 5,988.1 | 0 | 0 | 147.4 ms | 96 | 96 | 374.8 |
| 24.24 s | load | 5,565.0 | 5,565.0 | 5,565.0 | 0 | 0 | 150.3 ms | 96 | 96 | 376.5 |
| 25.24 s | load | 5,887.4 | 5,887.4 | 5,887.4 | 0 | 0 | 148.8 ms | 96 | 96 | 378.2 |
| 26.24 s | load | 6,930.2 | 6,930.2 | 6,930.2 | 0 | 0 | 148.8 ms | 96 | 96 | 379.8 |
| 27.24 s | load | 6,576.2 | 6,576.2 | 6,576.2 | 0 | 0 | 147.4 ms | 96 | 96 | 381.5 |
| 28.24 s | load | 6,123.5 | 6,123.5 | 6,123.5 | 0 | 0 | 147.4 ms | 96 | 96 | 383.0 |
| 29.24 s | load | 5,676.4 | 5,676.4 | 5,676.4 | 0 | 0 | 147.4 ms | 96 | 96 | 384.6 |
| 30.24 s | load | 6,450.9 | 5,938.9 | 5,938.9 | 256 | 512 | 147.4 ms | 96 | 96 | 386.8 |
| 31.24 s | load | 3,829.0 | 2,518.6 | 2,518.6 | 1,566 | 1,310 | 147.4 ms | 96 | 96 | 389.6 |
| 32.24 s | load | 6,589.6 | 6,508.6 | 6,508.6 | 81 | 81 | 147.4 ms | 96 | 96 | 391.6 |
| 33.24 s | load | 3,257.6 | 3,236.6 | 3,236.6 | 21 | 21 | 147.4 ms | 96 | 96 | 393.7 |
| 34.24 s | load | 6,312.1 | 6,312.1 | 6,312.1 | 0 | 0 | 145.9 ms | 96 | 96 | 395.3 |
| 35.24 s | load | 9,023.2 | 9,023.2 | 9,023.2 | 0 | 0 | 145.9 ms | 96 | 96 | 390.3 |
| 36.24 s | load | 9,086.8 | 9,086.8 | 9,086.8 | 0 | 0 | 144.5 ms | 96 | 96 | 391.9 |
| 37.24 s | load | 9,005.3 | 9,005.3 | 9,005.3 | 0 | 0 | 143.0 ms | 96 | 96 | 393.5 |
| 38.24 s | load | 9,502.4 | 9,502.4 | 9,502.4 | 0 | 0 | 141.6 ms | 96 | 96 | 394.9 |
| 39.24 s | load | 7,738.0 | 7,738.0 | 7,738.0 | 0 | 0 | 141.6 ms | 96 | 96 | 396.5 |
| 40.24 s | load | 8,090.0 | 8,090.0 | 8,090.0 | 0 | 0 | 140.2 ms | 96 | 96 | 398.3 |
| 41.25 s | load | 7,954.9 | 7,954.9 | 7,954.9 | 0 | 0 | 140.2 ms | 96 | 96 | 399.9 |
| 42.25 s | load | 7,780.9 | 7,780.9 | 7,780.9 | 0 | 0 | 140.2 ms | 96 | 96 | 401.4 |
| 43.25 s | load | 6,297.4 | 6,297.4 | 6,297.4 | 0 | 0 | 138.8 ms | 96 | 96 | 403.2 |
| 44.25 s | load | 6,427.1 | 6,427.1 | 6,427.1 | 0 | 0 | 138.8 ms | 96 | 96 | 404.8 |
| 45.25 s | load | 7,964.5 | 7,964.5 | 7,964.5 | 0 | 0 | 138.8 ms | 96 | 96 | 406.4 |
| 46.25 s | load | 9,023.3 | 9,023.3 | 9,023.3 | 0 | 0 | 137.4 ms | 96 | 96 | 404.2 |
| 47.25 s | load | 9,747.7 | 9,747.7 | 9,747.7 | 0 | 0 | 137.4 ms | 96 | 96 | 404.2 |
| 48.25 s | load | 10,278.2 | 10,278.2 | 10,278.2 | 0 | 0 | 137.4 ms | 96 | 96 | 404.2 |
| 49.25 s | load | 7,811.3 | 7,811.3 | 7,811.3 | 0 | 0 | 136.1 ms | 96 | 96 | 404.2 |
| 50.25 s | load | 7,730.9 | 7,730.9 | 7,730.9 | 0 | 0 | 136.1 ms | 96 | 96 | 404.2 |
| 51.25 s | load | 7,686.8 | 7,686.8 | 7,686.8 | 0 | 0 | 136.1 ms | 96 | 96 | 397.0 |
| 52.25 s | load | 6,979.2 | 6,979.2 | 6,979.2 | 0 | 0 | 134.7 ms | 96 | 96 | 397.0 |
| 53.25 s | load | 7,663.8 | 7,663.8 | 7,663.8 | 0 | 0 | 134.7 ms | 96 | 96 | 397.0 |
| 54.25 s | load | 7,550.3 | 7,550.3 | 7,550.3 | 0 | 0 | 134.7 ms | 96 | 96 | 397.0 |
| 55.25 s | load | 7,437.6 | 7,437.6 | 7,437.6 | 0 | 0 | 134.7 ms | 96 | 96 | 279.0 |
| 56.25 s | load | 7,703.7 | 7,703.7 | 7,703.7 | 0 | 0 | 133.4 ms | 96 | 96 | 279.0 |
| 57.25 s | load | 8,692.8 | 8,692.8 | 8,692.8 | 0 | 0 | 133.4 ms | 96 | 96 | 279.0 |
| 58.25 s | load | 9,661.3 | 9,661.3 | 9,661.3 | 0 | 0 | 133.4 ms | 96 | 96 | 279.0 |
| 59.25 s | load | 9,959.0 | 9,959.0 | 9,959.0 | 0 | 0 | 132.1 ms | 96 | 96 | 279.0 |
| 60.26 s | drain | 9,141.5 | 9,141.5 | 9,141.5 | 0 | 0 | 132.1 ms | 96 | 96 | 278.5 |
| 60.28 s | drain | 0.0 | 0.0 | 0.0 | 0 | 0 | 132.1 ms | 96 | 96 | 278.5 |
The shape of the rush
Throughput & response time
Successful work by phase
Actual measured phase durations appear below the bars.
Logical latency by phase
Retry and discovery time remain visible in the tails.
| Population | Calls | Latency |
|---|---|---|
| Setup | 5 | p99 337 ms |
| Warmup | 11,613 | p99 64.5 ms |
| Load | 423,870 | p99 176 ms |
| Drain | 1 | p99 7.28 ms |
| Audit | 20 | p99 7.02 ms |
A successful HTTP response is not necessarily a new business mutation: receipt replays count as responses, and queries also count. Phase percentiles cover successful and failed attempts or calls. Warmup uses reads and is not comparable to the mixed load phase.
Every call accounted for
Requests, retries & methods
- Load HTTP attempts
- 425,794
- Load HTTP successes
- 423,870
- Load HTTP failures
- 1,924
- Load retry attempts
- 1,924
- Load duplicate responses
- 12,424
- Load logical failures
- 0
Network retries belong to the same logical call. Deliberate deduplication probes appear separately as .replay calls. Expected stale-lease failures can appear in these measurements without failing the benchmark.
Setup
- Successful HTTP / s
- 1.2
- Completed logical / s
- 1.2
- HTTP failures
- 0
- Logical failures
- 0
HTTP error categories: None recorded
| HTTP method | Attempts | Succeeded | Failed | Retries | Duplicates | Success / s | Latency |
|---|---|---|---|---|---|---|---|
| deploy | 1 | 1 | 0 | 0 | 0 | 0.2 | p99 290 ms |
| pizza.setup | 1 | 1 | 0 | 0 | 0 | 0.2 | p99 22.1 ms |
| pizza.shop | 3 | 3 | 0 | 0 | 0 | 0.7 | p99 337 ms |
Logical calls, including explicit replays
| Logical method | Calls | Completed | Failed | Duplicates | Complete / s | Latency |
|---|---|---|---|---|---|---|
| deploy | 1 | 1 | 0 | 0 | 0.2 | p99 290 ms |
| pizza.setup | 1 | 1 | 0 | 0 | 0.2 | p99 22.2 ms |
| pizza.shop | 3 | 3 | 0 | 0 | 0.7 | p99 337 ms |
Warmup
- Successful HTTP / s
- 11,558.6
- Completed logical / s
- 11,558.6
- HTTP failures
- 0
- Logical failures
- 0
HTTP error categories: None recorded
| HTTP method | Attempts | Succeeded | Failed | Retries | Duplicates | Success / s | Latency |
|---|---|---|---|---|---|---|---|
| pizza.shop.local | 11,613 | 11,613 | 0 | 0 | 0 | 11,558.6 | p99 64.5 ms |
Logical calls, including explicit replays
| Logical method | Calls | Completed | Failed | Duplicates | Complete / s | Latency |
|---|---|---|---|---|---|---|
| pizza.shop.local | 11,613 | 11,613 | 0 | 0 | 11,558.6 | p99 64.5 ms |
Load
- Successful HTTP / s
- 7,049.6
- Completed logical / s
- 7,049.6
- HTTP failures
- 1,924
- Logical failures
- 0
HTTP error categories: network 1,924
| HTTP method | Attempts | Succeeded | Failed | Retries | Duplicates | Success / s | Latency |
|---|---|---|---|---|---|---|---|
| pizza.claim | 959 | 959 | 0 | 0 | 0 | 15.9 | p99 238 ms |
| pizza.deliver | 106 | 106 | 0 | 0 | 10 | 1.8 | p99 436 ms |
| pizza.order | 105 | 105 | 0 | 0 | 9 | 1.7 | p99 369 ms |
| pizza.shop.local | 287,137 | 287,005 | 132 | 132 | 0 | 4,773.3 | p99 11.2 ms |
| pizza.tip | 137,487 | 135,695 | 1,792 | 1,792 | 12,405 | 2,256.8 | p99 211 ms |
Logical calls, including explicit replays
| Logical method | Calls | Completed | Failed | Duplicates | Complete / s | Latency |
|---|---|---|---|---|---|---|
| pizza.claim | 959 | 959 | 0 | 0 | 15.9 | p99 268 ms |
| pizza.deliver | 96 | 96 | 0 | 0 | 1.6 | p99 448 ms |
| pizza.deliver.replay | 10 | 10 | 0 | 10 | 0.2 | p99 203 ms |
| pizza.order | 96 | 96 | 0 | 0 | 1.6 | p99 264 ms |
| pizza.order.replay | 9 | 9 | 0 | 9 | 0.1 | p99 369 ms |
| pizza.shop.local | 287,005 | 287,005 | 0 | 0 | 4,773.3 | p99 11.2 ms |
| pizza.tip | 123,290 | 123,290 | 0 | 0 | 2,050.5 | p99 215 ms |
| pizza.tip.replay | 12,405 | 12,405 | 0 | 12,405 | 206.3 | p99 203 ms |
Drain
- Successful HTTP / s
- 6.7
- Completed logical / s
- 6.7
- HTTP failures
- 0
- Logical failures
- 0
HTTP error categories: None recorded
| HTTP method | Attempts | Succeeded | Failed | Retries | Duplicates | Success / s | Latency |
|---|---|---|---|---|---|---|---|
| pizza.world | 1 | 1 | 0 | 0 | 0 | 6.7 | p99 7.21 ms |
Logical calls, including explicit replays
| Logical method | Calls | Completed | Failed | Duplicates | Complete / s | Latency |
|---|---|---|---|---|---|---|
| pizza.world | 1 | 1 | 0 | 0 | 6.7 | p99 7.28 ms |
Audit
- Successful HTTP / s
- 26.7
- Completed logical / s
- 26.7
- HTTP failures
- 19
- Logical failures
- 19
HTTP error categories: HTTP_422 19
| HTTP method | Attempts | Succeeded | Failed | Retries | Duplicates | Success / s | Latency |
|---|---|---|---|---|---|---|---|
| pizza.deliver | 19 | 0 | 19 | 0 | 0 | 0.0 | p99 2.18 ms |
| pizza.world | 1 | 1 | 0 | 0 | 0 | 26.7 | p99 6.97 ms |
Logical calls, including explicit replays
| Logical method | Calls | Completed | Failed | Duplicates | Complete / s | Latency |
|---|---|---|---|---|---|---|
| pizza.deliver | 19 | 0 | 19 | 0 | 0.0 | p99 2.29 ms |
| pizza.world | 1 | 1 | 0 | 0 | 26.7 | p99 7.02 ms |
From the oven to the doorstep
Business completion latency
Oven lateness96 samples
Order to door96 samples
Show as tables
| Latency | Samples | Share | Cumulative |
|---|---|---|---|
| 6.95 ms–7.45 ms | 2 | 2.08% | 2.08% |
| 7.98 ms–8.56 ms | 1 | 1.04% | 3.13% |
| 8.56 ms–9.18 ms | 1 | 1.04% | 4.17% |
| 9.84 ms–10.5 ms | 1 | 1.04% | 5.21% |
| 10.5 ms–11.3 ms | 2 | 2.08% | 7.29% |
| 11.3 ms–12.1 ms | 3 | 3.13% | 10.42% |
| 12.1 ms–13 ms | 3 | 3.13% | 13.54% |
| 13.9 ms–14.9 ms | 6 | 6.25% | 19.79% |
| 14.9 ms–16 ms | 5 | 5.21% | 25.00% |
| 16 ms–17.2 ms | 3 | 3.13% | 28.13% |
| 17.2 ms–18.4 ms | 4 | 4.17% | 32.29% |
| 18.4 ms–19.7 ms | 5 | 5.21% | 37.50% |
| 28 ms–30 ms | 1 | 1.04% | 38.54% |
| 30 ms–32.2 ms | 4 | 4.17% | 42.71% |
| 34.5 ms–37 ms | 2 | 2.08% | 44.79% |
| 85.2 ms–91.4 ms | 3 | 3.13% | 47.92% |
| 91.4 ms–98 ms | 8 | 8.33% | 56.25% |
| 98 ms–105 ms | 13 | 13.54% | 69.79% |
| 105 ms–113 ms | 7 | 7.29% | 77.08% |
| 113 ms–121 ms | 8 | 8.33% | 85.42% |
| 121 ms–129 ms | 7 | 7.29% | 92.71% |
| 129 ms–139 ms | 6 | 6.25% | 98.96% |
| 149 ms–160 ms | 1 | 1.04% | 100% |
| Latency | Samples | Share | Cumulative |
|---|---|---|---|
| 717 ms–753 ms | 2 | 2.08% | 2.08% |
| 753 ms–792 ms | 1 | 1.04% | 3.13% |
| 875 ms–919 ms | 1 | 1.04% | 4.17% |
| 966 ms–1.02 s | 1 | 1.04% | 5.21% |
| 1.12 s–1.18 s | 2 | 2.08% | 7.29% |
| 1.44 s–1.51 s | 1 | 1.04% | 8.33% |
| 1.67 s–1.76 s | 4 | 4.17% | 12.50% |
| 1.84 s–1.94 s | 2 | 2.08% | 14.58% |
| 1.94 s–2.04 s | 1 | 1.04% | 15.63% |
| 2.25 s–2.37 s | 3 | 3.13% | 18.75% |
| 2.37 s–2.49 s | 2 | 2.08% | 20.83% |
| 2.49 s–2.61 s | 3 | 3.13% | 23.96% |
| 2.61 s–2.75 s | 2 | 2.08% | 26.04% |
| 2.75 s–2.89 s | 2 | 2.08% | 28.13% |
| 2.89 s–3.03 s | 1 | 1.04% | 29.17% |
| 3.03 s–3.19 s | 1 | 1.04% | 30.21% |
| 3.19 s–3.35 s | 3 | 3.13% | 33.33% |
| 3.35 s–3.52 s | 5 | 5.21% | 38.54% |
| 3.52 s–3.7 s | 4 | 4.17% | 42.71% |
| 3.7 s–3.89 s | 2 | 2.08% | 44.79% |
| 3.89 s–4.09 s | 4 | 4.17% | 48.96% |
| 4.09 s–4.3 s | 4 | 4.17% | 53.13% |
| 4.3 s–4.52 s | 3 | 3.13% | 56.25% |
| 4.52 s–4.75 s | 5 | 5.21% | 61.46% |
| 4.75 s–4.99 s | 3 | 3.13% | 64.58% |
| 4.99 s–5.24 s | 4 | 4.17% | 68.75% |
| 5.24 s–5.51 s | 4 | 4.17% | 72.92% |
| 5.51 s–5.79 s | 6 | 6.25% | 79.17% |
| 5.79 s–6.09 s | 1 | 1.04% | 80.21% |
| 6.09 s–6.4 s | 4 | 4.17% | 84.38% |
| 6.4 s–6.73 s | 4 | 4.17% | 88.54% |
| 6.73 s–7.07 s | 5 | 5.21% | 93.75% |
| 7.07 s–7.43 s | 4 | 4.17% | 97.92% |
| 7.81 s–8.21 s | 2 | 2.08% | 100% |
- Configured oven delay
- 250.0 ms
- Configured lease duration
- 2,000.0 ms
- Delivered orders / s, including drain
- 1.6
- Retained order cap
- 96
Oven lateness measures how far execution runs past the requested deadline. It does not include the configured baking delay. Order-to-door latency includes the entire lifecycle. Each log time axis spans its population's p0.1 to p99.9; the tables list every bin.
The goblin ledger
Business invariants & tenant rankings
96 of 96 acknowledged orders delivered. The independent audit found no invariant violations.
- Orders match the client's acknowledged inputs.
- Stock equals initial dough minus ordered pizzas.
- Revenue equals delivered pizzas × seven copper.
- Tips match the acknowledged mutation ledger.
- Completed jobs match orders and delivery timestamps.
- Derived summaries and each tenant’s ranking match source records.
- Baking and delivery timestamps respect their deadlines.
- No pending or failed oven timers remain after drain.
| Tenant / kitchen | Orders | Pizzas delivered | Stock left | Revenue | Tips | Score |
|---|---|---|---|---|---|---|
| tenant-12 / The Crispy Cauldron · store-0 | 42 | 106 | 288 | 742 | 104,801 | 105,543 |
| tenant-12 / Mushroom Mayhem · store-1 | 2 | 6 | 388 | 42 | 6,319 | 6,361 |
| tenant-12 / The Sizzling Slime · store-2 | 3 | 7 | 387 | 49 | 6,225 | 6,274 |
| tenant-12 / Dough or Die · store-3 | 1 | 1 | 393 | 7 | 6,016 | 6,023 |
| tenant-13 / The Goblin's Slice · store-0 | 41 | 110 | 284 | 770 | 104,948 | 105,718 |
| tenant-13 / The Crispy Cauldron · store-2 | 1 | 1 | 393 | 7 | 6,336 | 6,343 |
| tenant-13 / Dragon Breath Delivery · store-1 | 6 | 15 | 379 | 105 | 6,127 | 6,232 |
| tenant-13 / Mushroom Mayhem · store-3 | 0 | 0 | 394 | 0 | 6,055 | 6,055 |
A dragon ate the leader
Failover, leases & replay safety
Leader crash 1
2026-09-25 05:45:22 UTC| Recovery milestone | Time since crash |
|---|---|
| Candidate observed in metrics | 510.9 ms |
| Leader observed in metrics | 537.5 ms |
| Quorum read probe completed | 592.8 ms |
| Leader discovery returned | 592.9 ms |
| Restarted node caught up | 2,742.1 ms |
Metrics observations follow polling and are not exact internal transition times. A brief candidate state can be missed; observing a leader alone does not establish a serving quorum. Missing milestones were not recorded or did not finish. Recovery and catch-up durations share the crash as time zero.
- Receipt replays verified
- 12,424
- Stale drones rejected by audit
- 19
- Leases claimed
- 115
- Empty claims
- 844
- Deliberately abandoned leases
- 19
- Reclaimed jobs
- 19
- Lease losses during workload
- 0
- Orders issued
- 96
| Replica | Observed role | Observed leader | Last applied index |
|---|---|---|---|
| Node 1 | Follower | 2 | 1,118 |
| Node 2 | Leader | 2 | 1,118 |
| Node 3 | Follower | 2 | 1,118 |
The cost of a pizza rush
Resource footprint
Sampled peak resident memory
- Cluster disk footprint
- 137.75 MiB
- Node controller sampled load cores
- 0.05
- Node controller CPU, user / entire run
- 3,939.2 ms
- Node controller CPU, system / entire run
- 540.2 ms
- Node controller event-loop delay p99
- 51.7 ms
- Node controller sampled peak RSS
- 406.4 MiB
- Rust customer driver peak RSS
- 20.6 MiB
- Rust customer driver sampled load cores
- 0.13
RSS is sampled, not an allocation total. Server CPU, when available, uses cumulative process CPU deltas over sampled load intervals; one core means one CPU-second per elapsed second. Restart and phase-boundary gaps are excluded. Older reports did not measure it. The load generator and all nodes share the same host, so these numbers include local resource contention and do not establish multi-host network performance.
| Server | Mean CPU cores | CPU time covered | Wall time covered | Intervals |
|---|---|---|---|---|
| 1 | 0.39 | 22,570.0 ms | 57,254.6 ms | 57 |
| 2 | 0.43 | 25,460.0 ms | 59,254.7 ms | 59 |
| 3 | 0.13 | 7,940.0 ms | 59,254.7 ms | 59 |
Make the result reproducible
Workload, host & measurement limits
Workload configuration
- Application transport
- HTTP/2 · h2c prior knowledge
- Admin and discovery transport
- HTTP/1.1
- Query routing
- Round-robin across live replicas
- Customer read policy
- Replica-local · lag allowed
- Bundle initialization
- static
- Customer loops / drone loops
- 256 / 4
- Requested load / warmup
- 60.0 s / 1.0 s
- Maximum drain
- 60.0 s
- Tenants / stores per tenant
- 2 / 4
- Hot stores per tenant
- 1
- Hot-set mixture weight
- 80%
- Duplicate / abandonment probability
- 10% / 10%
- Worker polling interval
- 150.0 ms
- Request timeout / retry budget
- 2,000.0 ms / 15,000.0 ms
- Workload seed
- 42:group-6
- Crash injection requested
- Yes
Execution environment
- JavaScript engine
- quickjs
- Runtime settings
- {"TOKIO_WORKER_THREADS":"2","FLOWER_WRITER_BATCH_MODE":"adaptive","FLOWER_WRITER_QUEUE_CAPACITY":"1024","FLOWER_QUERY_WORKERS":"16","FLOWER_PREPARATION_WORKERS":"16","FLOWER_WRITER_PREPARATION_WORKERS":"1","FLOWER_WRITER_WINDOW_MS":"50","FLOWER_WRITER_BATCH_MS":"50"}
- CPU
- Apple M5 Pro
- Logical CPUs
- 18
- Operating system
- darwin 27.2.0
- Architecture
- arm64
- Node.js controller
- v26.10.0
- Customer driver
- rust
- Customer driver SHA-256
- b3ce2a3e598826cd14bf319d0f65cee7a47b3b3822b7474bd1b987d2d5808e7f
- Co-located database nodes
- 3
- Binary SHA-256
- a53ad2ae79f74da0f9ddf5d79a2450beebd0efb0e7e00f0cdfcd70929f34d35d
- Binary size
- 45,119,536 bytes
- Started
- 2026-09-25 05:44:47 UTC
- Finished
- 2026-09-25 05:45:53 UTC
- Preserved database files
- Not retained
How to read these numbers
- Closed-loop concurrency. Each client waits for a response before issuing its next operation. Slow service reduces offered load; these results do not estimate open-loop latency under an independent arrival rate.
- HTTP/2 connection reuse. With
--http2, each driver process reuses a multiplexed session per origin. Rust customers and Node workers use separate pools; this comparison changes the whole driver, including its connection topology. Concurrency controls outstanding operations; it does not create one connection per loop. - Declared read policy.
--query-routing=replicasspreads query evaluation across live replicas. Replica-local previews skip the quorum fence and may be arbitrarily stale; this is the default customer workload. Final audits always use fresh reads. Mutations and deployment use the leader. Failed query attempts rotate through replicas before bounded backoff. - Tenant locality. Store keys are [tenant, store]; order IDs are scoped to that pair. Each tenant has its own queue and leaderboard. Logical separation in this unauthenticated demo is not authorization. A hot tenant still shares one ordered writer lane with the other tenants in this group.
- Timeline sampling. Rust customer statistics arrive in disjoint one-second batches; interval charts can show delivery-boundary artifacts. Whole-run counts and latency histograms merge the actual observations.
- Phase attribution. Calls belong to the phase in which they start. In-flight work can finish after the requested load interval, so measured phase durations may overlap at boundaries and should not be added as exact wall time.
- Bounded order population. After the order cap is reached, the workload continues with queries and tips. Receipt retention and the database's own state can still grow with mutations; order limits do not imply constant database size.
- Latency histograms. Fixed buckets 1% wide cover 0.001 ms to 24 hours; lower values share a floor bucket and larger ones an overflow bucket. Charts merge whole buckets into bins on log axes spanning p0.1 to p99.9. Percentiles are nearest-rank bucket upper bounds, within 1% of the exact rank.
- Fault and safety probes. Retried HTTP errors and intentional stale-lease rejections are recorded. The final verdict also requires successful lifecycle drainage and an independent business audit; a low HTTP error count alone is not sufficient.
- Replication topology. This benchmark uses one Raft group with 3 replicas. Replica observations do not measure sharding or horizontal partitioning.
- One run is an observation. Repeat matched configurations on a quiet host. A seeded workload stabilizes choices, but concurrent execution, scheduling, and elections remain nondeterministic.
Complete run options and bundle identity
{
"options": {
"offeredRate": 0,
"groups": 1,
"tenants": 2,
"duration": 60,
"warmup": 1,
"drain": 60,
"concurrency": 256,
"workers": 4,
"shops": 4,
"hotShops": 1,
"hotProbability": 0.8,
"maxOrders": 96,
"bakeMs": 250,
"leaseMs": 2000,
"duplicateRate": 0.1,
"abandonRate": 0.1,
"pollMs": 150,
"requestTimeoutMs": 2000,
"retryBudgetMs": 15000,
"nodes": 3,
"seed": "42:group-6",
"initialization": "static",
"queryRouting": "replicas",
"readConsistency": "replica-local",
"driver": "rust",
"driverBinary": "/Users/pcarrier/src/flower/target/release/flower-bench-driver",
"binary": "/Users/pcarrier/src/flower/target/release/flower",
"json": "/Users/pcarrier/src/flower/bench/results/latest-groups/group-6.json",
"html": "/Users/pcarrier/src/flower/bench/results/latest-groups/group-6.html",
"baseline": null,
"cpuProfile": null,
"chaos": true,
"http2": true,
"keepData": false,
"help": false,
"runId": "d4e5cf27-0a78-41ed-850e-68ccdca97756",
"tenantIds": [
"tenant-12",
"tenant-13"
]
},
"runtime": {
"engine": "quickjs",
"settings": {
"TOKIO_WORKER_THREADS": "2",
"FLOWER_WRITER_BATCH_MODE": "adaptive",
"FLOWER_WRITER_QUEUE_CAPACITY": "1024",
"FLOWER_QUERY_WORKERS": "16",
"FLOWER_PREPARATION_WORKERS": "16",
"FLOWER_WRITER_PREPARATION_WORKERS": "1",
"FLOWER_WRITER_WINDOW_MS": "50",
"FLOWER_WRITER_BATCH_MS": "50"
}
},
"queryRouting": {
"mode": "replicas",
"consistency": "replica-local",
"auditConsistency": "fresh",
"nodes": [
{
"id": 1,
"url": "http://127.0.0.1:62806",
"attempts": 97194,
"completed": 97062,
"failures": 132
},
{
"id": 2,
"url": "http://127.0.0.1:62817",
"attempts": 100781,
"completed": 100781,
"failures": 0
},
{
"id": 3,
"url": "http://127.0.0.1:62820",
"attempts": 100780,
"completed": 100780,
"failures": 0
}
]
},
"binary": {
"path": "/Users/pcarrier/src/flower/target/release/flower",
"sha256": "a53ad2ae79f74da0f9ddf5d79a2450beebd0efb0e7e00f0cdfcd70929f34d35d",
"bytes": 45119536,
"modifiedAt": "2026-09-25T05:16:03.202Z"
},
"driver": {
"kind": "rust",
"orchestration": "node",
"binary": {
"path": "/Users/pcarrier/src/flower/target/release/flower-bench-driver",
"sha256": "b3ce2a3e598826cd14bf319d0f65cee7a47b3b3822b7474bd1b987d2d5808e7f",
"bytes": 7136160,
"modifiedAt": "2026-09-25T05:15:43.417Z"
},
"warmupRequests": 3
},
"bundleHash": "d3b5c3e6fa9e51548200e79cde504fc967a5d7100a49af12a43f27560721ad64"
}When the oven smokes
Diagnostics
No unexpected failure details were recorded. Intentional stale-lease rejections and retried HTTP failures are reported in their own counters.