Flower / Field notesReactive database · Raft · TypeScript

Goblin Pizza Express / Benchmark report

The rush-hour report.

Goblins bake. Drones deliver. Raft keeps the books.
Scheduled ovens, leased deliveries, and reactive shop totals under load.

Passed

2026-09-25 05:44:47 UTC

65.97 s total wall time · 0 unexpected failures

Customer goodput

6,828.7 / s

70% reads · 30% mutations among successful customer calls.

All-call latency

All-call latency: 423,731 calls; p50 2.04 ms, p99 171 ms, max 1.02 s. Log axis from 0.0329 ms to 1.17 s.

p50 2.04 ms · p99 171 ms. All load-phase logical calls, including worker traffic, replays, discovery, and retries.

Orders delivered

96 / 96

233 pizzas · 1,631 copper in delivery revenue.

Quorum recovery

616.7 ms

First recorded crash to a serving quorum; restart catch-up is tracked separately.

Replica-local customer previews: replication lag is allowed. The selected pizza.shop.local method uses coherent locally applied snapshots without a quorum fence. Data, code, and aliases may lag without a bound; requests across replicas can return decreasing revisions. Replica-local is the benchmark default; choose --read-consistency fresh to measure quorum-fenced previews. Mutations and the final pizza.world audit retain fresh semantics.

The customer's view

Successful work, precisely counted.

One Raft group
Successful customer mix
70% reads / 30% mutations
Successful customer calls
410,283
Failed customer calls
0
Measured load duration
60.08 s
Load worker completions excluded
1,057
Load replay completions excluded
12,391
Customer goodput uses actual successful primary customer calls and the measured load duration.
Load measurementSuccessful completionsGoodput / sSuccessful customer share
Primary customer reads287,6124,787.070%
Primary customer mutations122,6712,041.730%
All primary customer calls410,2836,828.7100%
All logical calls, including workers and replays423,7317,052.5Includes non-customer traffic
Raw successful HTTP responses423,7317,052.5Includes retries and non-customer traffic

Counting definition: successful pizza.shop.local, pizza.order, and pizza.tip logical calls per measured load second. Transport retries count once when their customer call succeeds. Explicit .replay probes, worker claims and deliveries, polling, setup, audit, and every failed call contribute zero customer goodput. Actual successful read/mutation shares appear above.

The run verdict covers correctness and completion of any requested CPU profile. Customer goodput is an average over the recorded load phase; individual intervals can differ.

The rush, second by second

How the run unfolded

60 recorded time samples

Observed completion rate

Each point summarizes the interval since the previous sample.

Interval throughput over timeSuccessful HTTP responses and completed logical calls per second in each sampling interval. Dashed vertical lines mark recorded leader crashes. 60 samples over 60.21 s since load start. HTTP responses: first 12.0, last 9,754.5 events / s. Logical calls: first 12.0, last 9,754.5 events / s.00s2,88515s5,77030s8,65545s11,54060sLeader crash at 30.02 s after load startCrashHTTP responsesLogical calls
HTTP responsesLogical callsevents / s · elapsed load time

Cumulative HTTP tail latency

p95 of attempts since load began; not an interval percentile.

Cumulative HTTP p95 over timeThe latency p95 is cumulative from load start through each sample, including drain. It is not a moving-window or interval percentile. 60 samples over 60.21 s since load start. Cumulative p95: first 305.9, last 134.7 ms.00s8715s17330s26045s34760sLeader crash at 30.02 s after load startCrashCumulative p95
Cumulative p95ms · elapsed load time

Orders placed and delivered

Acknowledgements observed by the benchmark driver.

Acknowledged orders and deliveries over timeCumulative acknowledged order and delivery counts. Their gap shows business work still waiting for completion. 60 samples over 60.21 s since load start. Orders: first 0.0, last 96.0 orders. Deliveries: first 4.0, last 96.0 orders.00s2415s4830s7245s9660sLeader crash at 30.02 s after load startCrashOrdersDeliveries
OrdersDeliveriesorders · elapsed load time

Driver resident memory

Samples include the driver's runtime and retained workload records.

Driver resident memory over timeBenchmark driver resident memory sampled over elapsed load time. 60 samples over 60.21 s since load start. Driver RSS: first 327.6, last 411.9 MiB.00s10315s20630s30945s41260sLeader crash at 30.02 s after load startCrashDriver RSS
Driver RSSMiB · elapsed load time

Lines connect observed samples; no measurements are interpolated into the underlying data. Throughput is an interval rate, while p95 is cumulative. The driver and database share a machine. Any crash marker is positioned relative to the start of measured load.

Recorded time samples
Retries and HTTP failures are counts for each interval. Cumulative p95 covers all attempts since load began.
ElapsedPhaseHTTP attempts / sHTTP success / sLogical complete / sRetriesHTTP failuresCumulative p95OrdersDeliveriesDriver MiB
1.08 sload12.012.012.000305.9 ms04327.6
2.08 sload2,991.22,991.22,991.200346.7 ms9610334.1
3.17 sload11,540.011,540.011,540.000194.7 ms9626335.8
4.17 sload6,351.46,351.46,351.400189.0 ms9641336.9
5.17 sload6,318.66,318.66,318.600179.8 ms9658338.1
6.20 sload6,238.26,238.26,238.200169.4 ms9670339.5
7.20 sload6,193.26,193.26,193.200166.0 ms9685341.3
8.21 sload6,086.56,086.56,086.500164.4 ms9695342.6
9.21 sload6,715.66,715.66,715.600162.8 ms9696344.0
10.21 sload6,668.46,668.46,668.400159.6 ms9696345.5
11.21 sload7,385.87,385.87,385.800156.4 ms9696346.8
12.21 sload6,748.06,748.06,748.000156.4 ms9696348.3
13.21 sload6,421.06,421.06,421.000154.9 ms9696349.7
14.21 sload6,655.06,655.06,655.000153.3 ms9696351.6
15.24 sload4,895.64,895.64,895.600154.9 ms9696353.9
16.24 sload6,674.66,674.66,674.600154.9 ms9696355.3
17.24 sload6,955.56,955.56,955.500153.3 ms9696356.9
18.25 sload6,353.26,353.26,353.200153.3 ms9696358.3
19.25 sload6,257.66,257.66,257.600153.3 ms9696359.8
20.25 sload7,166.47,166.47,166.400151.8 ms9696361.2
21.25 sload7,435.57,435.57,435.500150.3 ms9696362.8
22.25 sload6,402.26,402.26,402.200150.3 ms9696363.9
23.25 sload6,004.56,004.56,004.500150.3 ms9696365.3
24.25 sload6,083.86,083.86,083.800153.3 ms9696366.7
25.26 sload6,270.16,270.16,270.100151.8 ms9696361.5
26.26 sload6,578.96,578.96,578.900151.8 ms9696363.0
27.26 sload6,612.26,612.26,612.200150.3 ms9696364.4
28.26 sload6,759.46,759.46,759.400148.8 ms9696365.8
29.26 sload6,292.56,292.56,292.500150.3 ms9696367.3
30.26 sload7,180.16,667.96,667.9256512148.8 ms9696369.2
31.26 sload4,068.12,757.42,757.41,5671,311148.8 ms9696370.8
32.26 sload7,904.87,817.97,817.98787148.8 ms9696372.6
33.26 sload5,030.34,959.44,959.47171147.4 ms9696374.0
34.26 sload5,057.85,057.85,057.800147.4 ms9696375.5
35.26 sload8,838.78,838.78,838.700147.4 ms9696376.9
36.26 sload8,803.88,803.88,803.800145.9 ms9696378.3
37.26 sload9,299.99,299.99,299.900145.9 ms9696379.6
38.26 sload8,330.98,330.98,330.900144.5 ms9696381.1
39.26 sload7,022.27,022.27,022.200143.0 ms9696382.5
40.27 sload7,442.97,442.97,442.900143.0 ms9696384.0
41.27 sload7,703.77,703.77,703.700141.6 ms9696385.4
42.27 sload7,697.87,697.87,697.800141.6 ms9696386.8
43.27 sload6,830.86,830.86,830.800141.6 ms9696388.3
44.27 sload8,075.08,075.08,075.000140.2 ms9696389.7
45.27 sload8,383.28,383.28,383.200140.2 ms9696391.0
46.27 sload8,993.08,993.08,993.000140.2 ms9696392.3
47.27 sload9,191.29,191.29,191.200138.8 ms9696393.8
48.27 sload9,442.19,442.19,442.100138.8 ms9696395.1
49.27 sload6,936.66,936.66,936.600138.8 ms9696396.5
50.27 sload7,094.47,094.47,094.400137.4 ms9696397.7
51.27 sload8,418.68,418.68,418.600137.4 ms9696399.2
52.27 sload7,460.17,460.17,460.100137.4 ms9696400.5
53.27 sload6,900.36,900.36,900.300137.4 ms9696402.0
54.27 sload7,801.07,801.07,801.000137.4 ms9696403.5
55.27 sload7,623.57,623.57,623.500136.1 ms9696404.8
56.27 sload7,162.57,162.57,162.500136.1 ms9696406.3
57.28 sload9,096.29,096.29,096.200136.1 ms9696407.6
58.28 sload8,817.68,817.68,817.600134.7 ms9696409.0
59.28 sload9,274.69,274.69,274.600134.7 ms9696410.4
60.21 sdrain9,754.59,754.59,754.500134.7 ms9696411.9

The shape of the rush

Throughput & response time

Closed-loop workload

Successful work by phase

Actual measured phase durations appear below the bars.

Successful throughput by phaseSuccessful HTTP responses and completed logical calls per second for each phase. Setup: HTTP responses 1.2 events / s, Logical calls 1.2 events / s. Warmup: HTTP responses 11,427.8 events / s, Logical calls 11,427.8 events / s. Load: HTTP responses 7,052.5 events / s, Logical calls 7,052.5 events / s. Drain: HTTP responses 8.1 events / s, Logical calls 8.1 events / s. Audit: HTTP responses 58.7 events / s, Logical calls 58.7 events / s02,8575,7148,57111,428Setup · HTTP responses: 1.2 events / sSetup · Logical calls: 1.2 events / sSetup4.27 sWarmup · HTTP responses: 11,427.8 events / sWarmup · Logical calls: 11,427.8 events / sWarmup1.02 sLoad · HTTP responses: 7,052.5 events / sLoad · Logical calls: 7,052.5 events / sLoad60.08 sDrain · HTTP responses: 8.1 events / sDrain · Logical calls: 8.1 events / sDrain0.12 sAudit · HTTP responses: 58.7 events / sAudit · Logical calls: 58.7 events / sAudit0.02 s
HTTP responsesLogical callsevents / s · linear scale

Logical latency by phase

Retry and discovery time remain visible in the tails.

Logical calls by the phase they started in. One log axis from 0.0329 ms to 1.17 s; each line marks p99.
PopulationCallsLatency
Setup5Setup: 5 calls; p50 270 ms, p99 354 ms, max 354 ms. Log axis from 0.0329 ms to 1.17 s.p99 354 ms
Warmup11,655Warmup: 11,655 calls; p50 19.9 ms, p99 57.3 ms, max 61.4 ms. Log axis from 0.0329 ms to 1.17 s.p99 57.3 ms
Load423,731Load: 423,731 calls; p50 2.04 ms, p99 171 ms, max 1.02 s. Log axis from 0.0329 ms to 1.17 s.p99 171 ms
Drain1Drain: 1 call; p50 8.22 ms, p99 8.22 ms, max 8.22 ms. Log axis from 0.0329 ms to 1.17 s.p99 8.22 ms
Audit16Audit: 16 calls; p50 0.69 ms, p99 5.73 ms, max 5.73 ms. Log axis from 0.0329 ms to 1.17 s.p99 5.73 ms

A successful HTTP response is not necessarily a new business mutation: receipt replays count as responses, and queries also count. Phase percentiles cover successful and failed attempts or calls. Warmup uses reads and is not comparable to the mixed load phase.

Every call accounted for

Requests, retries & methods

Load details expanded below
Load HTTP attempts
425,712
Load HTTP successes
423,731
Load HTTP failures
1,981
Load retry attempts
1,981
Load duplicate responses
12,521
Load logical failures
0

Network retries belong to the same logical call. Deliberate deduplication probes appear separately as .replay calls. Expected stale-lease failures can appear in these measurements without failing the benchmark.

Setup4.27 s · 5 HTTP attempts
Successful HTTP / s
1.2
Completed logical / s
1.2
HTTP failures
0
Logical failures
0

HTTP error categories: None recorded

Setup HTTP attempts. Latency histograms share one log axis from 2.81 ms to 463 ms; each line marks p99.
HTTP methodAttemptsSucceededFailedRetriesDuplicatesSuccess / sLatency
deploy110000.2deploy HTTP attempts: 1 call; p50 269 ms, p99 269 ms, max 269 ms. Log axis from 2.81 ms to 463 ms.p99 269 ms
pizza.setup110000.2pizza.setup HTTP attempts: 1 call; p50 18.2 ms, p99 18.2 ms, max 18.2 ms. Log axis from 2.81 ms to 463 ms.p99 18.2 ms
pizza.shop330000.7pizza.shop HTTP attempts: 3 calls; p50 350 ms, p99 354 ms, max 354 ms. Log axis from 2.81 ms to 463 ms.p99 354 ms
Logical calls, including explicit replays
One logical call includes discovery, retries, and backoff. Explicit replay checks are separate .replay calls. Latency histograms share one log axis from 3.39 ms to 463 ms.
Logical methodCallsCompletedFailedDuplicatesComplete / sLatency
deploy11000.2deploy logical calls: 1 call; p50 269 ms, p99 269 ms, max 269 ms. Log axis from 3.39 ms to 463 ms.p99 269 ms
pizza.setup11000.2pizza.setup logical calls: 1 call; p50 18.3 ms, p99 18.3 ms, max 18.3 ms. Log axis from 3.39 ms to 463 ms.p99 18.3 ms
pizza.shop33000.7pizza.shop logical calls: 3 calls; p50 350 ms, p99 354 ms, max 354 ms. Log axis from 3.39 ms to 463 ms.p99 354 ms
Warmup1.02 s · 11,655 HTTP attempts
Successful HTTP / s
11,427.8
Completed logical / s
11,427.8
HTTP failures
0
Logical failures
0

HTTP error categories: None recorded

Warmup HTTP attempts. Latency histograms share one log axis from 2.35 ms to 77.2 ms; each line marks p99.
HTTP methodAttemptsSucceededFailedRetriesDuplicatesSuccess / sLatency
pizza.shop.local11,65511,65500011,427.8pizza.shop.local HTTP attempts: 11,655 calls; p50 19.9 ms, p99 57.3 ms, max 61.4 ms. Log axis from 2.35 ms to 77.2 ms; 0.069% of calls fall outside it.p99 57.3 ms
Logical calls, including explicit replays
One logical call includes discovery, retries, and backoff. Explicit replay checks are separate .replay calls. Latency histograms share one log axis from 2.35 ms to 77.2 ms.
Logical methodCallsCompletedFailedDuplicatesComplete / sLatency
pizza.shop.local11,65511,6550011,427.8pizza.shop.local logical calls: 11,655 calls; p50 19.9 ms, p99 57.3 ms, max 61.4 ms. Log axis from 2.35 ms to 77.2 ms; 0.069% of calls fall outside it.p99 57.3 ms
Load60.08 s · 425,712 HTTP attempts
Successful HTTP / s
7,052.5
Completed logical / s
7,052.5
HTTP failures
1,981
Logical failures
0

HTTP error categories: HTTP_503 1 · network 1,980

Load HTTP attempts. Latency histograms share one log axis from 0.0275 ms to 570 ms; each line marks p99.
HTTP methodAttemptsSucceededFailedRetriesDuplicatesSuccess / sLatency
pizza.claim96196100016.0pizza.claim HTTP attempts: 961 calls; p50 91.4 ms, p99 207 ms, max 296 ms. Log axis from 0.0275 ms to 570 ms.p99 207 ms
pizza.deliver10810800121.8pizza.deliver HTTP attempts: 108 calls; p50 131 ms, p99 306 ms, max 306 ms. Log axis from 0.0275 ms to 570 ms.p99 306 ms
pizza.order11111100151.8pizza.order HTTP attempts: 111 calls; p50 215 ms, p99 359 ms, max 359 ms. Log axis from 0.0275 ms to 570 ms.p99 359 ms
pizza.shop.local287,801287,61218918904,787.0pizza.shop.local HTTP attempts: 287,801 calls; p50 1.04 ms, p99 11.8 ms, max 1.02 s. Log axis from 0.0275 ms to 570 ms; 0.088% of calls fall outside it.p99 11.8 ms
pizza.tip136,731134,9391,7921,79212,4942,245.9pizza.tip HTTP attempts: 136,731 calls; p50 107 ms, p99 197 ms, max 460 ms. Log axis from 0.0275 ms to 570 ms.p99 197 ms
Logical calls, including explicit replays
One logical call includes discovery, retries, and backoff. Explicit replay checks are separate .replay calls. Latency histograms share one log axis from 0.0359 ms to 1.67 s.
Logical methodCallsCompletedFailedDuplicatesComplete / sLatency
pizza.claim9619610016.0pizza.claim logical calls: 961 calls; p50 91.4 ms, p99 293 ms, max 588 ms. Log axis from 0.0359 ms to 1.67 s.p99 293 ms
pizza.deliver9696001.6pizza.deliver logical calls: 96 calls; p50 132 ms, p99 306 ms, max 306 ms. Log axis from 0.0359 ms to 1.67 s.p99 306 ms
pizza.deliver.replay12120120.2pizza.deliver.replay logical calls: 12 calls; p50 81.9 ms, p99 182 ms, max 182 ms. Log axis from 0.0359 ms to 1.67 s.p99 182 ms
pizza.order9696001.6pizza.order logical calls: 96 calls; p50 213 ms, p99 282 ms, max 282 ms. Log axis from 0.0359 ms to 1.67 s.p99 282 ms
pizza.order.replay15150150.2pizza.order.replay logical calls: 15 calls; p50 350 ms, p99 359 ms, max 359 ms. Log axis from 0.0359 ms to 1.67 s.p99 359 ms
pizza.shop.local287,612287,612004,787.0pizza.shop.local logical calls: 287,612 calls; p50 1.04 ms, p99 11.8 ms, max 1.02 s. Log axis from 0.0359 ms to 1.67 s.p99 11.8 ms
pizza.tip122,575122,57501302,040.1pizza.tip logical calls: 122,575 calls; p50 108 ms, p99 211 ms, max 806 ms. Log axis from 0.0359 ms to 1.67 s.p99 211 ms
pizza.tip.replay12,36412,364012,364205.8pizza.tip.replay logical calls: 12,364 calls; p50 103 ms, p99 193 ms, max 806 ms. Log axis from 0.0359 ms to 1.67 s.p99 193 ms
Drain0.12 s · 1 HTTP attempts
Successful HTTP / s
8.1
Completed logical / s
8.1
HTTP failures
0
Logical failures
0

HTTP error categories: None recorded

Drain HTTP attempts. Latency histograms share one log axis from 2.13 ms to 31.2 ms; each line marks p99.
HTTP methodAttemptsSucceededFailedRetriesDuplicatesSuccess / sLatency
pizza.world110008.1pizza.world HTTP attempts: 1 call; p50 8.16 ms, p99 8.16 ms, max 8.16 ms. Log axis from 2.13 ms to 31.2 ms.p99 8.16 ms
Logical calls, including explicit replays
One logical call includes discovery, retries, and backoff. Explicit replay checks are separate .replay calls. Latency histograms share one log axis from 2.13 ms to 31.2 ms.
Logical methodCallsCompletedFailedDuplicatesComplete / sLatency
pizza.world11008.1pizza.world logical calls: 1 call; p50 8.22 ms, p99 8.22 ms, max 8.22 ms. Log axis from 2.13 ms to 31.2 ms.p99 8.22 ms
Audit0.02 s · 16 HTTP attempts
Successful HTTP / s
58.7
Completed logical / s
58.7
HTTP failures
15
Logical failures
15

HTTP error categories: HTTP_422 15

Audit HTTP attempts. Latency histograms share one log axis from 0.478 ms to 6.35 ms; each line marks p99.
HTTP methodAttemptsSucceededFailedRetriesDuplicatesSuccess / sLatency
pizza.deliver15015000.0pizza.deliver HTTP attempts: 15 calls; p50 0.67 ms, p99 1.38 ms, max 1.38 ms. Log axis from 0.478 ms to 6.35 ms.p99 1.38 ms
pizza.world1100058.7pizza.world HTTP attempts: 1 call; p50 5.7 ms, p99 5.7 ms, max 5.7 ms. Log axis from 0.478 ms to 6.35 ms.p99 5.7 ms
Logical calls, including explicit replays
One logical call includes discovery, retries, and backoff. Explicit replay checks are separate .replay calls. Latency histograms share one log axis from 0.478 ms to 7.01 ms.
Logical methodCallsCompletedFailedDuplicatesComplete / sLatency
pizza.deliver1501500.0pizza.deliver logical calls: 15 calls; p50 0.69 ms, p99 1.46 ms, max 1.46 ms. Log axis from 0.478 ms to 7.01 ms.p99 1.46 ms
pizza.world110058.7pizza.world logical calls: 1 call; p50 5.73 ms, p99 5.73 ms, max 5.73 ms. Log axis from 0.478 ms to 7.01 ms.p99 5.73 ms

From the oven to the doorstep

Business completion latency

Load and drain combined

Oven lateness96 samples

Oven lateness: 96 samples; p50 96.1 ms, p99 263 ms, max 263 ms. Log axis from 17.9 ms to 296 ms.20 ms50 ms100 ms200 ms19 ms–20.1 ms: 3.1% of samples (3)20.1 ms–21.4 ms: 4.2% of samples (4)21.4 ms–22.7 ms: 2.1% of samples (2)22.7 ms–24.1 ms: 7.3% of samples (7)24.1 ms–25.6 ms: 1.0% of samples (1)25.6 ms–27.1 ms: 3.1% of samples (3)27.1 ms–28.8 ms: 3.1% of samples (3)28.8 ms–30.6 ms: 8.3% of samples (8)30.6 ms–32.5 ms: 6.3% of samples (6)89.6 ms–95.1 ms: 6.3% of samples (6)95.1 ms–101 ms: 20.8% of samples (20)101 ms–107 ms: 5.2% of samples (5)107 ms–114 ms: 2.1% of samples (2)114 ms–121 ms: 8.3% of samples (8)121 ms–128 ms: 4.2% of samples (4)128 ms–136 ms: 6.3% of samples (6)173 ms–183 ms: 6.3% of samples (6)247 ms–262 ms: 1.0% of samples (1)262 ms–279 ms: 1.0% of samples (1)p50 96.1 msp99 263 ms

Order to door96 samples

Order to door: 96 samples; p50 4.38 s, p99 8.45 s, max 8.45 s. Log axis from 717 ms to 9.07 s.1 s2 s5 s753 ms–792 ms: 3.1% of samples (3)875 ms–919 ms: 1.0% of samples (1)1.44 s–1.51 s: 4.2% of samples (4)1.84 s–1.94 s: 2.1% of samples (2)1.94 s–2.04 s: 2.1% of samples (2)2.14 s–2.25 s: 1.0% of samples (1)2.25 s–2.37 s: 2.1% of samples (2)2.49 s–2.61 s: 4.2% of samples (4)2.61 s–2.75 s: 3.1% of samples (3)2.75 s–2.89 s: 4.2% of samples (4)3.03 s–3.19 s: 1.0% of samples (1)3.19 s–3.35 s: 5.2% of samples (5)3.35 s–3.52 s: 4.2% of samples (4)3.52 s–3.7 s: 2.1% of samples (2)3.7 s–3.89 s: 3.1% of samples (3)3.89 s–4.09 s: 4.2% of samples (4)4.09 s–4.3 s: 2.1% of samples (2)4.3 s–4.52 s: 4.2% of samples (4)4.52 s–4.75 s: 3.1% of samples (3)4.75 s–4.99 s: 4.2% of samples (4)4.99 s–5.24 s: 4.2% of samples (4)5.24 s–5.51 s: 5.2% of samples (5)5.51 s–5.79 s: 3.1% of samples (3)5.79 s–6.09 s: 4.2% of samples (4)6.09 s–6.4 s: 4.2% of samples (4)6.4 s–6.73 s: 4.2% of samples (4)6.73 s–7.07 s: 7.3% of samples (7)7.07 s–7.43 s: 4.2% of samples (4)7.43 s–7.81 s: 2.1% of samples (2)8.21 s–8.63 s: 1.0% of samples (1)p50 4.38 sp99 8.45 s
Show as tables
Oven lateness: 96 samples, 0 invalid, 0 beyond 24 hours
LatencySamplesShareCumulative
19 ms–20.1 ms33.13%3.13%
20.1 ms–21.4 ms44.17%7.29%
21.4 ms–22.7 ms22.08%9.38%
22.7 ms–24.1 ms77.29%16.67%
24.1 ms–25.6 ms11.04%17.71%
25.6 ms–27.1 ms33.13%20.83%
27.1 ms–28.8 ms33.13%23.96%
28.8 ms–30.6 ms88.33%32.29%
30.6 ms–32.5 ms66.25%38.54%
89.6 ms–95.1 ms66.25%44.79%
95.1 ms–101 ms2020.83%65.63%
101 ms–107 ms55.21%70.83%
107 ms–114 ms22.08%72.92%
114 ms–121 ms88.33%81.25%
121 ms–128 ms44.17%85.42%
128 ms–136 ms66.25%91.67%
173 ms–183 ms66.25%97.92%
247 ms–262 ms11.04%98.96%
262 ms–279 ms11.04%100%
Order to door: 96 samples, 0 invalid, 0 beyond 24 hours
LatencySamplesShareCumulative
753 ms–792 ms33.13%3.13%
875 ms–919 ms11.04%4.17%
1.44 s–1.51 s44.17%8.33%
1.84 s–1.94 s22.08%10.42%
1.94 s–2.04 s22.08%12.50%
2.14 s–2.25 s11.04%13.54%
2.25 s–2.37 s22.08%15.63%
2.49 s–2.61 s44.17%19.79%
2.61 s–2.75 s33.13%22.92%
2.75 s–2.89 s44.17%27.08%
3.03 s–3.19 s11.04%28.13%
3.19 s–3.35 s55.21%33.33%
3.35 s–3.52 s44.17%37.50%
3.52 s–3.7 s22.08%39.58%
3.7 s–3.89 s33.13%42.71%
3.89 s–4.09 s44.17%46.88%
4.09 s–4.3 s22.08%48.96%
4.3 s–4.52 s44.17%53.13%
4.52 s–4.75 s33.13%56.25%
4.75 s–4.99 s44.17%60.42%
4.99 s–5.24 s44.17%64.58%
5.24 s–5.51 s55.21%69.79%
5.51 s–5.79 s33.13%72.92%
5.79 s–6.09 s44.17%77.08%
6.09 s–6.4 s44.17%81.25%
6.4 s–6.73 s44.17%85.42%
6.73 s–7.07 s77.29%92.71%
7.07 s–7.43 s44.17%96.88%
7.43 s–7.81 s22.08%98.96%
8.21 s–8.63 s11.04%100%
Configured oven delay
250.0 ms
Configured lease duration
2,000.0 ms
Delivered orders / s, including drain
1.6
Retained order cap
96

Oven lateness measures how far execution runs past the requested deadline. It does not include the configured baking delay. Order-to-door latency includes the entire lifecycle. Each log time axis spans its population's p0.1 to p99.9; the tables list every bin.

The goblin ledger

Business invariants & tenant rankings

Audit passed
96 of 96 acknowledged orders delivered. The independent audit found no invariant violations.
Scores are revenue plus tips, in integer copper coins. Rows follow each tenant’s independently derived leaderboard; this is not a global ranking.
Tenant / kitchenOrdersPizzas deliveredStock leftRevenueTipsScore
tenant-6 / The Crispy Cauldron · store-043105289735104,302105,037
tenant-6 / Dough or Die · store-325389356,2346,269
tenant-6 / The Sizzling Slime · store-224390286,0896,117
tenant-6 / Mushroom Mayhem · store-10039405,9645,964
tenant-7 / The Goblin's Slice · store-04093301651104,404105,055
tenant-7 / The Crispy Cauldron · store-2311383776,1606,237
tenant-7 / Mushroom Mayhem · store-3410384706,1026,172
tenant-7 / Dragon Breath Delivery · store-125389355,7885,823

A dragon ate the leader

Failover, leases & replay safety

3 Raft replicas · one group

Leader crash 1

2026-09-25 05:45:22 UTC
Crash recovery: leader 1 killed; serving quorum after 616.7 ms; restarted node caught up after 2,730.0 ms. Both durations are measured from the crash.Leader 1 killed0.0 msQuorum serving; leader 2616.7 msRestarted node caught up2,730.0 ms
Recovery observations, all measured from the leader crash
Recovery milestoneTime since crash
Candidate observed in metrics542.8 ms
Leader observed in metrics569.2 ms
Quorum read probe completed616.7 ms
Leader discovery returned616.7 ms
Restarted node caught up2,730.0 ms

Metrics observations follow polling and are not exact internal transition times. A brief candidate state can be missed; observing a leader alone does not establish a serving quorum. Missing milestones were not recorded or did not finish. Recovery and catch-up durations share the crash as time zero.

Receipt replays verified
12,391
Stale drones rejected by audit
15
Leases claimed
111
Empty claims
850
Deliberately abandoned leases
15
Reclaimed jobs
15
Lease losses during workload
0
Orders issued
96
End-of-run replica observations, collected separately. These are replicas of one Raft group, not independent shards.
ReplicaObserved roleObserved leaderLast applied index
Node 1Follower21,117
Node 2Leader21,117
Node 3Follower21,117

The cost of a pizza rush

Resource footprint

Co-located processes

Sampled peak resident memory

Sampled peak resident memory by processPeak observed resident memory per server and benchmark driver. Samples can miss brief peaks; per-process peaks need not occur together. Node 1: Peak sampled RSS 312.2 MiB. Node 2: Peak sampled RSS 301.4 MiB. Node 3: Peak sampled RSS 279.4 MiB. Driver: Peak sampled RSS 411.9 MiB0103206309412Node 1 · Peak sampled RSS: 312.2 MiBNode 1Node 2 · Peak sampled RSS: 301.4 MiBNode 2Node 3 · Peak sampled RSS: 279.4 MiBNode 3Driver · Peak sampled RSS: 411.9 MiBDriver
Peak sampled RSSMiB · linear scale
Cluster disk footprint
133.96 MiB
Node controller sampled load cores
0.05
Node controller CPU, user / entire run
3,855.5 ms
Node controller CPU, system / entire run
439.9 ms
Node controller event-loop delay p99
59.6 ms
Node controller sampled peak RSS
411.9 MiB
Rust customer driver peak RSS
20.3 MiB
Rust customer driver sampled load cores
0.13

RSS is sampled, not an allocation total. Server CPU, when available, uses cumulative process CPU deltas over sampled load intervals; one core means one CPU-second per elapsed second. Restart and phase-boundary gaps are excluded. Older reports did not measure it. The load generator and all nodes share the same host, so these numbers include local resource contention and do not establish multi-host network performance.

CPU observations during load; node coverage can differ. Includes all process threads.
ServerMean CPU coresCPU time coveredWall time coveredIntervals
10.3922,480.0 ms57,274.4 ms57
20.4325,560.0 ms59,274.5 ms59
30.137,920.0 ms59,274.5 ms59

Make the result reproducible

Workload, host & measurement limits

Report schema 1

Workload configuration

Application transport
HTTP/2 · h2c prior knowledge
Admin and discovery transport
HTTP/1.1
Query routing
Round-robin across live replicas
Customer read policy
Replica-local · lag allowed
Bundle initialization
static
Customer loops / drone loops
256 / 4
Requested load / warmup
60.0 s / 1.0 s
Maximum drain
60.0 s
Tenants / stores per tenant
2 / 4
Hot stores per tenant
1
Hot-set mixture weight
80%
Duplicate / abandonment probability
10% / 10%
Worker polling interval
150.0 ms
Request timeout / retry budget
2,000.0 ms / 15,000.0 ms
Workload seed
42:group-3
Crash injection requested
Yes

Execution environment

JavaScript engine
quickjs
Runtime settings
{"TOKIO_WORKER_THREADS":"2","FLOWER_WRITER_BATCH_MODE":"adaptive","FLOWER_WRITER_QUEUE_CAPACITY":"1024","FLOWER_QUERY_WORKERS":"16","FLOWER_PREPARATION_WORKERS":"16","FLOWER_WRITER_PREPARATION_WORKERS":"1","FLOWER_WRITER_WINDOW_MS":"50","FLOWER_WRITER_BATCH_MS":"50"}
CPU
Apple M5 Pro
Logical CPUs
18
Operating system
darwin 27.2.0
Architecture
arm64
Node.js controller
v26.10.0
Customer driver
rust
Customer driver SHA-256
b3ce2a3e598826cd14bf319d0f65cee7a47b3b3822b7474bd1b987d2d5808e7f
Co-located database nodes
3
Binary SHA-256
a53ad2ae79f74da0f9ddf5d79a2450beebd0efb0e7e00f0cdfcd70929f34d35d
Binary size
45,119,536 bytes
Started
2026-09-25 05:44:47 UTC
Finished
2026-09-25 05:45:53 UTC
Preserved database files
Not retained

How to read these numbers

Complete run options and bundle identity
{
  "options": {
    "offeredRate": 0,
    "groups": 1,
    "tenants": 2,
    "duration": 60,
    "warmup": 1,
    "drain": 60,
    "concurrency": 256,
    "workers": 4,
    "shops": 4,
    "hotShops": 1,
    "hotProbability": 0.8,
    "maxOrders": 96,
    "bakeMs": 250,
    "leaseMs": 2000,
    "duplicateRate": 0.1,
    "abandonRate": 0.1,
    "pollMs": 150,
    "requestTimeoutMs": 2000,
    "retryBudgetMs": 15000,
    "nodes": 3,
    "seed": "42:group-3",
    "initialization": "static",
    "queryRouting": "replicas",
    "readConsistency": "replica-local",
    "driver": "rust",
    "driverBinary": "/Users/pcarrier/src/flower/target/release/flower-bench-driver",
    "binary": "/Users/pcarrier/src/flower/target/release/flower",
    "json": "/Users/pcarrier/src/flower/bench/results/latest-groups/group-3.json",
    "html": "/Users/pcarrier/src/flower/bench/results/latest-groups/group-3.html",
    "baseline": null,
    "cpuProfile": null,
    "chaos": true,
    "http2": true,
    "keepData": false,
    "help": false,
    "runId": "d4e5cf27-0a78-41ed-850e-68ccdca97756",
    "tenantIds": [
      "tenant-6",
      "tenant-7"
    ]
  },
  "runtime": {
    "engine": "quickjs",
    "settings": {
      "TOKIO_WORKER_THREADS": "2",
      "FLOWER_WRITER_BATCH_MODE": "adaptive",
      "FLOWER_WRITER_QUEUE_CAPACITY": "1024",
      "FLOWER_QUERY_WORKERS": "16",
      "FLOWER_PREPARATION_WORKERS": "16",
      "FLOWER_WRITER_PREPARATION_WORKERS": "1",
      "FLOWER_WRITER_WINDOW_MS": "50",
      "FLOWER_WRITER_BATCH_MS": "50"
    }
  },
  "queryRouting": {
    "mode": "replicas",
    "consistency": "replica-local",
    "auditConsistency": "fresh",
    "nodes": [
      {
        "id": 1,
        "url": "http://127.0.0.1:62801",
        "attempts": 96694,
        "completed": 96505,
        "failures": 189
      },
      {
        "id": 2,
        "url": "http://127.0.0.1:62811",
        "attempts": 101384,
        "completed": 101384,
        "failures": 0
      },
      {
        "id": 3,
        "url": "http://127.0.0.1:62816",
        "attempts": 101383,
        "completed": 101383,
        "failures": 0
      }
    ]
  },
  "binary": {
    "path": "/Users/pcarrier/src/flower/target/release/flower",
    "sha256": "a53ad2ae79f74da0f9ddf5d79a2450beebd0efb0e7e00f0cdfcd70929f34d35d",
    "bytes": 45119536,
    "modifiedAt": "2026-09-25T05:16:03.202Z"
  },
  "driver": {
    "kind": "rust",
    "orchestration": "node",
    "binary": {
      "path": "/Users/pcarrier/src/flower/target/release/flower-bench-driver",
      "sha256": "b3ce2a3e598826cd14bf319d0f65cee7a47b3b3822b7474bd1b987d2d5808e7f",
      "bytes": 7136160,
      "modifiedAt": "2026-09-25T05:15:43.417Z"
    },
    "warmupRequests": 3
  },
  "bundleHash": "d3b5c3e6fa9e51548200e79cde504fc967a5d7100a49af12a43f27560721ad64"
}

When the oven smokes

Diagnostics

0 unexpected failures

No unexpected failure details were recorded. Intentional stale-lease rejections and retried HTTP failures are reported in their own counters.