Flower / Field notesReactive database · Raft · TypeScript

Goblin Pizza Express / Benchmark report

The rush-hour report.

Goblins bake. Drones deliver. Raft keeps the books.
Scheduled ovens, leased deliveries, and reactive shop totals under load.

Passed

2026-09-25 05:44:47 UTC

66.00 s total wall time · 0 unexpected failures

Customer goodput

6,819.0 / s

70% reads · 30% mutations among successful customer calls.

All-call latency

All-call latency: 423,317 calls; p50 2 ms, p99 178 ms, max 1.02 s. Log axis from 0.0359 ms to 1.67 s.

p50 2 ms · p99 178 ms. All load-phase logical calls, including worker traffic, replays, discovery, and retries.

Orders delivered

96 / 96

242 pizzas · 1,694 copper in delivery revenue.

Quorum recovery

676.9 ms

First recorded crash to a serving quorum; restart catch-up is tracked separately.

Replica-local customer previews: replication lag is allowed. The selected pizza.shop.local method uses coherent locally applied snapshots without a quorum fence. Data, code, and aliases may lag without a bound; requests across replicas can return decreasing revisions. Replica-local is the benchmark default; choose --read-consistency fresh to measure quorum-fenced previews. Mutations and the final pizza.world audit retain fresh semantics.

The customer's view

Successful work, precisely counted.

One Raft group
Successful customer mix
70% reads / 30% mutations
Successful customer calls
409,847
Failed customer calls
0
Measured load duration
60.10 s
Load worker completions excluded
1,053
Load replay completions excluded
12,417
Customer goodput uses actual successful primary customer calls and the measured load duration.
Load measurementSuccessful completionsGoodput / sSuccessful customer share
Primary customer reads286,5804,768.170%
Primary customer mutations123,2672,050.930%
All primary customer calls409,8476,819.0100%
All logical calls, including workers and replays423,3177,043.1Includes non-customer traffic
Raw successful HTTP responses423,3177,043.1Includes retries and non-customer traffic

Counting definition: successful pizza.shop.local, pizza.order, and pizza.tip logical calls per measured load second. Transport retries count once when their customer call succeeds. Explicit .replay probes, worker claims and deliveries, polling, setup, audit, and every failed call contribute zero customer goodput. Actual successful read/mutation shares appear above.

The run verdict covers correctness and completion of any requested CPU profile. Customer goodput is an average over the recorded load phase; individual intervals can differ.

The rush, second by second

How the run unfolded

60 recorded time samples

Observed completion rate

Each point summarizes the interval since the previous sample.

Interval throughput over timeSuccessful HTTP responses and completed logical calls per second in each sampling interval. Dashed vertical lines mark recorded leader crashes. 60 samples over 60.22 s since load start. HTTP responses: first 15.8, last 9,830.2 events / s. Logical calls: first 15.8, last 9,830.2 events / s.00s3,08615s6,17230s9,25845s12,34460sLeader crash at 30.02 s after load startCrashHTTP responsesLogical calls
HTTP responsesLogical callsevents / s · elapsed load time

Cumulative HTTP tail latency

p95 of attempts since load began; not an interval percentile.

Cumulative HTTP p95 over timeThe latency p95 is cumulative from load start through each sample, including drain. It is not a moving-window or interval percentile. 60 samples over 60.22 s since load start. Cumulative p95: first 391.4, last 133.4 ms.00s9815s19630s29445s39160sLeader crash at 30.02 s after load startCrashCumulative p95
Cumulative p95ms · elapsed load time

Orders placed and delivered

Acknowledgements observed by the benchmark driver.

Acknowledged orders and deliveries over timeCumulative acknowledged order and delivery counts. Their gap shows business work still waiting for completion. 60 samples over 60.22 s since load start. Orders: first 0.0, last 96.0 orders. Deliveries: first 4.0, last 96.0 orders.00s2415s4830s7245s9660sLeader crash at 30.02 s after load startCrashOrdersDeliveries
OrdersDeliveriesorders · elapsed load time

Driver resident memory

Samples include the driver's runtime and retained workload records.

Driver resident memory over timeBenchmark driver resident memory sampled over elapsed load time. 60 samples over 60.22 s since load start. Driver RSS: first 300.7, last 404.4 MiB.00s10115s20230s30345s40460sLeader crash at 30.02 s after load startCrashDriver RSS
Driver RSSMiB · elapsed load time

Lines connect observed samples; no measurements are interpolated into the underlying data. Throughput is an interval rate, while p95 is cumulative. The driver and database share a machine. Any crash marker is positioned relative to the start of measured load.

Recorded time samples
Retries and HTTP failures are counts for each interval. Cumulative p95 covers all attempts since load began.
ElapsedPhaseHTTP attempts / sHTTP success / sLogical complete / sRetriesHTTP failuresCumulative p95OrdersDeliveriesDriver MiB
1.07 sload15.815.815.800391.4 ms04300.7
2.07 sload3,880.03,880.03,880.000317.0 ms9616303.3
3.08 sload6,263.16,263.16,263.100194.7 ms9630307.3
4.08 sload5,567.45,567.45,567.400190.9 ms9647309.1
5.08 sload6,926.46,926.46,926.400190.9 ms9661311.1
6.08 sload6,456.56,456.56,456.500179.8 ms9674313.2
7.14 sload12,343.712,343.712,343.700164.4 ms9683315.5
8.16 sload6,232.66,232.66,232.600164.4 ms9691317.3
9.20 sload6,912.56,912.56,912.500159.6 ms9696319.0
10.20 sload6,533.66,533.66,533.600158.0 ms9696320.6
11.20 sload6,237.76,237.76,237.700154.9 ms9696322.2
12.20 sload6,619.26,619.26,619.200151.8 ms9696323.8
13.21 sload6,499.16,499.16,499.100153.3 ms9696325.4
14.21 sload6,621.36,621.36,621.300151.8 ms9696330.4
15.22 sload5,478.35,478.35,478.300153.3 ms9696334.4
16.23 sload7,314.77,314.77,314.700150.3 ms9696335.8
17.23 sload5,913.85,913.85,913.800151.8 ms9696337.5
18.23 sload6,353.16,353.16,353.100151.8 ms9696340.3
19.23 sload6,803.26,803.26,803.200150.3 ms9696341.9
20.23 sload6,666.06,666.06,666.000150.3 ms9696343.5
21.23 sload6,558.46,558.46,558.400150.3 ms9696345.1
22.23 sload6,876.36,876.36,876.300148.8 ms9696346.7
23.23 sload5,637.95,637.95,637.900148.8 ms9696348.3
24.24 sload5,538.25,538.25,538.200150.3 ms9696349.9
25.24 sload7,122.77,122.77,122.700148.8 ms9696345.0
26.24 sload6,250.96,250.96,250.900148.8 ms9696346.5
27.24 sload6,508.16,508.16,508.100148.8 ms9696348.2
28.24 sload6,565.26,565.26,565.200148.8 ms9696349.8
29.24 sload5,905.15,905.15,905.100148.8 ms9696351.5
30.24 sload5,590.15,078.55,078.5256512150.3 ms9696353.8
31.24 sload3,131.31,584.71,584.71,8031,547148.8 ms9696356.4
32.24 sload8,480.68,361.88,361.8119119148.8 ms9696358.7
33.24 sload5,282.55,200.65,200.68282147.4 ms9696360.6
34.24 sload5,005.55,005.55,005.500147.4 ms9696362.3
35.24 sload8,289.38,289.38,289.300147.4 ms9696364.0
36.24 sload8,590.68,590.68,590.600145.9 ms9696365.5
37.24 sload9,013.89,013.89,013.800144.5 ms9696367.0
38.24 sload8,048.28,048.28,048.200144.5 ms9696368.5
39.24 sload7,382.37,382.37,382.300143.0 ms9696370.1
40.24 sload8,100.58,100.58,100.500143.0 ms9696371.8
41.25 sload7,001.97,001.97,001.900143.0 ms9696373.4
42.24 sload8,014.98,014.98,014.900141.6 ms9696375.1
43.25 sload6,490.16,490.16,490.100141.6 ms9696376.6
44.25 sload6,106.06,106.06,106.000141.6 ms9696378.2
45.25 sload8,676.08,676.08,676.000141.6 ms9696379.8
46.25 sload8,882.88,882.88,882.800140.2 ms9696381.5
47.25 sload9,271.59,271.59,271.500140.2 ms9696383.1
48.25 sload9,280.49,280.49,280.400138.8 ms9696384.7
49.25 sload8,349.38,349.38,349.300138.8 ms9696386.4
50.25 sload7,169.57,169.57,169.500138.8 ms9696388.0
51.25 sload6,994.36,994.36,994.300137.4 ms9696389.5
52.25 sload7,739.87,739.87,739.800137.4 ms9696391.2
53.25 sload7,931.87,931.87,931.800137.4 ms9696392.8
54.25 sload8,225.48,225.48,225.400136.1 ms9696394.4
55.25 sload7,700.67,700.67,700.600136.1 ms9696396.1
56.25 sload7,863.17,863.17,863.100136.1 ms9696397.7
57.25 sload9,532.39,532.39,532.300136.1 ms9696399.4
58.25 sload9,662.79,662.79,662.700134.7 ms9696401.0
59.25 sload10,093.110,093.110,093.100134.7 ms9696402.8
60.22 sdrain9,830.29,830.29,830.200133.4 ms9696404.4

The shape of the rush

Throughput & response time

Closed-loop workload

Successful work by phase

Actual measured phase durations appear below the bars.

Successful throughput by phaseSuccessful HTTP responses and completed logical calls per second for each phase. Setup: HTTP responses 1.2 events / s, Logical calls 1.2 events / s. Warmup: HTTP responses 10,436.7 events / s, Logical calls 10,436.7 events / s. Load: HTTP responses 7,043.1 events / s, Logical calls 7,043.1 events / s. Drain: HTTP responses 8.3 events / s, Logical calls 8.3 events / s. Audit: HTTP responses 46.2 events / s, Logical calls 46.2 events / s02,6095,2187,82810,437Setup · HTTP responses: 1.2 events / sSetup · Logical calls: 1.2 events / sSetup4.23 sWarmup · HTTP responses: 10,436.7 events / sWarmup · Logical calls: 10,436.7 events / sWarmup1.00 sLoad · HTTP responses: 7,043.1 events / sLoad · Logical calls: 7,043.1 events / sLoad60.10 sDrain · HTTP responses: 8.3 events / sDrain · Logical calls: 8.3 events / sDrain0.12 sAudit · HTTP responses: 46.2 events / sAudit · Logical calls: 46.2 events / sAudit0.02 s
HTTP responsesLogical callsevents / s · linear scale

Logical latency by phase

Retry and discovery time remain visible in the tails.

Logical calls by the phase they started in. One log axis from 0.0359 ms to 1.67 s; each line marks p99.
PopulationCallsLatency
Setup5Setup: 5 calls; p50 268 ms, p99 422 ms, max 422 ms. Log axis from 0.0359 ms to 1.67 s.p99 422 ms
Warmup10,485Warmup: 10,485 calls; p50 20.8 ms, p99 73.4 ms, max 86.2 ms. Log axis from 0.0359 ms to 1.67 s.p99 73.4 ms
Load423,317Load: 423,317 calls; p50 2 ms, p99 178 ms, max 1.02 s. Log axis from 0.0359 ms to 1.67 s.p99 178 ms
Drain1Drain: 1 call; p50 6.97 ms, p99 6.97 ms, max 6.97 ms. Log axis from 0.0359 ms to 1.67 s.p99 6.97 ms
Audit17Audit: 17 calls; p50 0.959 ms, p99 5.91 ms, max 5.91 ms. Log axis from 0.0359 ms to 1.67 s.p99 5.91 ms

A successful HTTP response is not necessarily a new business mutation: receipt replays count as responses, and queries also count. Phase percentiles cover successful and failed attempts or calls. Warmup uses reads and is not comparable to the mixed load phase.

Every call accounted for

Requests, retries & methods

Load details expanded below
Load HTTP attempts
425,577
Load HTTP successes
423,317
Load HTTP failures
2,260
Load retry attempts
2,260
Load duplicate responses
12,417
Load logical failures
0

Network retries belong to the same logical call. Deliberate deduplication probes appear separately as .replay calls. Expected stale-lease failures can appear in these measurements without failing the benchmark.

Setup4.23 s · 5 HTTP attempts
Successful HTTP / s
1.2
Completed logical / s
1.2
HTTP failures
0
Logical failures
0

HTTP error categories: None recorded

Setup HTTP attempts. Latency histograms share one log axis from 6.35 ms to 548 ms; each line marks p99.
HTTP methodAttemptsSucceededFailedRetriesDuplicatesSuccess / sLatency
deploy110000.2deploy HTTP attempts: 1 call; p50 266 ms, p99 266 ms, max 266 ms. Log axis from 6.35 ms to 548 ms.p99 266 ms
pizza.setup110000.2pizza.setup HTTP attempts: 1 call; p50 27.3 ms, p99 27.3 ms, max 27.3 ms. Log axis from 6.35 ms to 548 ms.p99 27.3 ms
pizza.shop330000.7pizza.shop HTTP attempts: 3 calls; p50 411 ms, p99 422 ms, max 422 ms. Log axis from 6.35 ms to 548 ms.p99 422 ms
Logical calls, including explicit replays
One logical call includes discovery, retries, and backoff. Explicit replay checks are separate .replay calls. Latency histograms share one log axis from 6.35 ms to 548 ms.
Logical methodCallsCompletedFailedDuplicatesComplete / sLatency
deploy11000.2deploy logical calls: 1 call; p50 266 ms, p99 266 ms, max 266 ms. Log axis from 6.35 ms to 548 ms.p99 266 ms
pizza.setup11000.2pizza.setup logical calls: 1 call; p50 27.4 ms, p99 27.4 ms, max 27.4 ms. Log axis from 6.35 ms to 548 ms.p99 27.4 ms
pizza.shop33000.7pizza.shop logical calls: 3 calls; p50 411 ms, p99 422 ms, max 422 ms. Log axis from 6.35 ms to 548 ms.p99 422 ms
Warmup1.00 s · 10,485 HTTP attempts
Successful HTTP / s
10,436.7
Completed logical / s
10,436.7
HTTP failures
0
Logical failures
0

HTTP error categories: None recorded

Warmup HTTP attempts. Latency histograms share one log axis from 3.04 ms to 100 ms; each line marks p99.
HTTP methodAttemptsSucceededFailedRetriesDuplicatesSuccess / sLatency
pizza.shop.local10,48510,48500010,436.7pizza.shop.local HTTP attempts: 10,485 calls; p50 20.8 ms, p99 73.4 ms, max 86.2 ms. Log axis from 3.04 ms to 100 ms; 0.057% of calls fall outside it.p99 73.4 ms
Logical calls, including explicit replays
One logical call includes discovery, retries, and backoff. Explicit replay checks are separate .replay calls. Latency histograms share one log axis from 3.04 ms to 100 ms.
Logical methodCallsCompletedFailedDuplicatesComplete / sLatency
pizza.shop.local10,48510,4850010,436.7pizza.shop.local logical calls: 10,485 calls; p50 20.8 ms, p99 73.4 ms, max 86.2 ms. Log axis from 3.04 ms to 100 ms; 0.057% of calls fall outside it.p99 73.4 ms
Load60.10 s · 425,577 HTTP attempts
Successful HTTP / s
7,043.1
Completed logical / s
7,043.1
HTTP failures
2,260
Logical failures
0

HTTP error categories: HTTP_503 1 · network 2,259

Load HTTP attempts. Latency histograms share one log axis from 0.0275 ms to 570 ms; each line marks p99.
HTTP methodAttemptsSucceededFailedRetriesDuplicatesSuccess / sLatency
pizza.claim95795700015.9pizza.claim HTTP attempts: 957 calls; p50 95.1 ms, p99 265 ms, max 391 ms. Log axis from 0.0275 ms to 570 ms.p99 265 ms
pizza.deliver10610600101.8pizza.deliver HTTP attempts: 106 calls; p50 115 ms, p99 245 ms, max 254 ms. Log axis from 0.0275 ms to 570 ms.p99 245 ms
pizza.order1051050091.7pizza.order HTTP attempts: 105 calls; p50 191 ms, p99 322 ms, max 322 ms. Log axis from 0.0275 ms to 570 ms.p99 322 ms
pizza.shop.local286,792286,58021221204,768.1pizza.shop.local HTTP attempts: 286,792 calls; p50 1.03 ms, p99 10.5 ms, max 1.02 s. Log axis from 0.0275 ms to 570 ms; 0.089% of calls fall outside it.p99 10.5 ms
pizza.tip137,617135,5692,0482,04812,3982,255.6pizza.tip HTTP attempts: 137,617 calls; p50 106 ms, p99 205 ms, max 397 ms. Log axis from 0.0275 ms to 570 ms.p99 205 ms
Logical calls, including explicit replays
One logical call includes discovery, retries, and backoff. Explicit replay checks are separate .replay calls. Latency histograms share one log axis from 0.0261 ms to 1.59 s.
Logical methodCallsCompletedFailedDuplicatesComplete / sLatency
pizza.claim9579570015.9pizza.claim logical calls: 957 calls; p50 95.1 ms, p99 284 ms, max 681 ms. Log axis from 0.0261 ms to 1.59 s.p99 284 ms
pizza.deliver9696001.6pizza.deliver logical calls: 96 calls; p50 115 ms, p99 254 ms, max 254 ms. Log axis from 0.0261 ms to 1.59 s.p99 254 ms
pizza.deliver.replay10100100.2pizza.deliver.replay logical calls: 10 calls; p50 83.6 ms, p99 244 ms, max 244 ms. Log axis from 0.0261 ms to 1.59 s.p99 244 ms
pizza.order9696001.6pizza.order logical calls: 96 calls; p50 189 ms, p99 262 ms, max 262 ms. Log axis from 0.0261 ms to 1.59 s.p99 262 ms
pizza.order.replay99090.1pizza.order.replay logical calls: 9 calls; p50 323 ms, p99 323 ms, max 323 ms. Log axis from 0.0261 ms to 1.59 s.p99 323 ms
pizza.shop.local286,580286,580004,768.1pizza.shop.local logical calls: 286,580 calls; p50 1.03 ms, p99 10.5 ms, max 1.02 s. Log axis from 0.0261 ms to 1.59 s.p99 10.5 ms
pizza.tip123,171123,171002,049.3pizza.tip logical calls: 123,171 calls; p50 107 ms, p99 213 ms, max 980 ms. Log axis from 0.0261 ms to 1.59 s.p99 213 ms
pizza.tip.replay12,39812,398012,398206.3pizza.tip.replay logical calls: 12,398 calls; p50 103 ms, p99 197 ms, max 984 ms. Log axis from 0.0261 ms to 1.59 s.p99 197 ms
Drain0.12 s · 1 HTTP attempts
Successful HTTP / s
8.3
Completed logical / s
8.3
HTTP failures
0
Logical failures
0

HTTP error categories: None recorded

Drain HTTP attempts. Latency histograms share one log axis from 1.92 ms to 25.6 ms; each line marks p99.
HTTP methodAttemptsSucceededFailedRetriesDuplicatesSuccess / sLatency
pizza.world110008.3pizza.world HTTP attempts: 1 call; p50 6.92 ms, p99 6.92 ms, max 6.92 ms. Log axis from 1.92 ms to 25.6 ms.p99 6.92 ms
Logical calls, including explicit replays
One logical call includes discovery, retries, and backoff. Explicit replay checks are separate .replay calls. Latency histograms share one log axis from 1.92 ms to 25.6 ms.
Logical methodCallsCompletedFailedDuplicatesComplete / sLatency
pizza.world11008.3pizza.world logical calls: 1 call; p50 6.97 ms, p99 6.97 ms, max 6.97 ms. Log axis from 1.92 ms to 25.6 ms.p99 6.97 ms
Audit0.02 s · 17 HTTP attempts
Successful HTTP / s
46.2
Completed logical / s
46.2
HTTP failures
16
Logical failures
16

HTTP error categories: HTTP_422 16

Audit HTTP attempts. Latency histograms share one log axis from 0.478 ms to 7.01 ms; each line marks p99.
HTTP methodAttemptsSucceededFailedRetriesDuplicatesSuccess / sLatency
pizza.deliver16016000.0pizza.deliver HTTP attempts: 16 calls; p50 0.921 ms, p99 1.33 ms, max 1.33 ms. Log axis from 0.478 ms to 7.01 ms.p99 1.33 ms
pizza.world1100046.2pizza.world HTTP attempts: 1 call; p50 5.87 ms, p99 5.87 ms, max 5.87 ms. Log axis from 0.478 ms to 7.01 ms.p99 5.87 ms
Logical calls, including explicit replays
One logical call includes discovery, retries, and backoff. Explicit replay checks are separate .replay calls. Latency histograms share one log axis from 0.528 ms to 7.01 ms.
Logical methodCallsCompletedFailedDuplicatesComplete / sLatency
pizza.deliver1601600.0pizza.deliver logical calls: 16 calls; p50 0.959 ms, p99 1.39 ms, max 1.39 ms. Log axis from 0.528 ms to 7.01 ms.p99 1.39 ms
pizza.world110046.2pizza.world logical calls: 1 call; p50 5.91 ms, p99 5.91 ms, max 5.91 ms. Log axis from 0.528 ms to 7.01 ms.p99 5.91 ms

From the oven to the doorstep

Business completion latency

Load and drain combined

Oven lateness96 samples

Oven lateness: 96 samples; p50 69.2 ms, p99 166 ms, max 166 ms. Log axis from 6.48 ms to 183 ms.10 ms20 ms50 ms100 ms6.95 ms–7.45 ms: 1.0% of samples (1)7.98 ms–8.56 ms: 1.0% of samples (1)9.84 ms–10.5 ms: 4.2% of samples (4)10.5 ms–11.3 ms: 7.3% of samples (7)11.3 ms–12.1 ms: 1.0% of samples (1)12.1 ms–13 ms: 2.1% of samples (2)13.9 ms–14.9 ms: 2.1% of samples (2)14.9 ms–16 ms: 1.0% of samples (1)16 ms–17.2 ms: 4.2% of samples (4)17.2 ms–18.4 ms: 3.1% of samples (3)18.4 ms–19.7 ms: 3.1% of samples (3)19.7 ms–21.2 ms: 3.1% of samples (3)21.2 ms–22.7 ms: 2.1% of samples (2)22.7 ms–24.3 ms: 4.2% of samples (4)64.5 ms–69.2 ms: 17.7% of samples (17)69.2 ms–74.2 ms: 22.9% of samples (22)121 ms–129 ms: 1.0% of samples (1)139 ms–149 ms: 4.2% of samples (4)149 ms–160 ms: 10.4% of samples (10)160 ms–171 ms: 4.2% of samples (4)p50 69.2 msp99 166 ms

Order to door96 samples

Order to door: 96 samples; p50 3.97 s, p99 8.5 s, max 8.5 s. Log axis from 643 ms to 9.43 s.1 s2 s5 s682 ms–724 ms: 3.1% of samples (3)816 ms–866 ms: 1.0% of samples (1)866 ms–919 ms: 3.1% of samples (3)1.24 s–1.32 s: 2.1% of samples (2)1.32 s–1.4 s: 2.1% of samples (2)1.48 s–1.57 s: 1.0% of samples (1)1.57 s–1.67 s: 3.1% of samples (3)1.67 s–1.77 s: 1.0% of samples (1)1.88 s–2 s: 2.1% of samples (2)2 s–2.12 s: 2.1% of samples (2)2.25 s–2.39 s: 2.1% of samples (2)2.39 s–2.54 s: 1.0% of samples (1)2.54 s–2.69 s: 4.2% of samples (4)2.69 s–2.86 s: 3.1% of samples (3)2.86 s–3.03 s: 1.0% of samples (1)3.03 s–3.22 s: 2.1% of samples (2)3.22 s–3.42 s: 3.1% of samples (3)3.42 s–3.63 s: 5.2% of samples (5)3.63 s–3.85 s: 6.3% of samples (6)3.85 s–4.09 s: 4.2% of samples (4)4.09 s–4.34 s: 3.1% of samples (3)4.34 s–4.61 s: 3.1% of samples (3)4.61 s–4.89 s: 6.3% of samples (6)4.89 s–5.19 s: 5.2% of samples (5)5.19 s–5.51 s: 4.2% of samples (4)5.51 s–5.85 s: 2.1% of samples (2)5.85 s–6.21 s: 3.1% of samples (3)6.21 s–6.59 s: 4.2% of samples (4)6.59 s–7 s: 2.1% of samples (2)7 s–7.43 s: 4.2% of samples (4)7.43 s–7.89 s: 4.2% of samples (4)7.89 s–8.37 s: 4.2% of samples (4)8.37 s–8.89 s: 1.0% of samples (1)p50 3.97 sp99 8.5 s
Show as tables
Oven lateness: 96 samples, 0 invalid, 0 beyond 24 hours
LatencySamplesShareCumulative
6.95 ms–7.45 ms11.04%1.04%
7.98 ms–8.56 ms11.04%2.08%
9.84 ms–10.5 ms44.17%6.25%
10.5 ms–11.3 ms77.29%13.54%
11.3 ms–12.1 ms11.04%14.58%
12.1 ms–13 ms22.08%16.67%
13.9 ms–14.9 ms22.08%18.75%
14.9 ms–16 ms11.04%19.79%
16 ms–17.2 ms44.17%23.96%
17.2 ms–18.4 ms33.13%27.08%
18.4 ms–19.7 ms33.13%30.21%
19.7 ms–21.2 ms33.13%33.33%
21.2 ms–22.7 ms22.08%35.42%
22.7 ms–24.3 ms44.17%39.58%
64.5 ms–69.2 ms1717.71%57.29%
69.2 ms–74.2 ms2222.92%80.21%
121 ms–129 ms11.04%81.25%
139 ms–149 ms44.17%85.42%
149 ms–160 ms1010.42%95.83%
160 ms–171 ms44.17%100%
Order to door: 96 samples, 0 invalid, 0 beyond 24 hours
LatencySamplesShareCumulative
682 ms–724 ms33.13%3.13%
816 ms–866 ms11.04%4.17%
866 ms–919 ms33.13%7.29%
1.24 s–1.32 s22.08%9.38%
1.32 s–1.4 s22.08%11.46%
1.48 s–1.57 s11.04%12.50%
1.57 s–1.67 s33.13%15.63%
1.67 s–1.77 s11.04%16.67%
1.88 s–2 s22.08%18.75%
2 s–2.12 s22.08%20.83%
2.25 s–2.39 s22.08%22.92%
2.39 s–2.54 s11.04%23.96%
2.54 s–2.69 s44.17%28.13%
2.69 s–2.86 s33.13%31.25%
2.86 s–3.03 s11.04%32.29%
3.03 s–3.22 s22.08%34.38%
3.22 s–3.42 s33.13%37.50%
3.42 s–3.63 s55.21%42.71%
3.63 s–3.85 s66.25%48.96%
3.85 s–4.09 s44.17%53.13%
4.09 s–4.34 s33.13%56.25%
4.34 s–4.61 s33.13%59.38%
4.61 s–4.89 s66.25%65.63%
4.89 s–5.19 s55.21%70.83%
5.19 s–5.51 s44.17%75.00%
5.51 s–5.85 s22.08%77.08%
5.85 s–6.21 s33.13%80.21%
6.21 s–6.59 s44.17%84.38%
6.59 s–7 s22.08%86.46%
7 s–7.43 s44.17%90.63%
7.43 s–7.89 s44.17%94.79%
7.89 s–8.37 s44.17%98.96%
8.37 s–8.89 s11.04%100%
Configured oven delay
250.0 ms
Configured lease duration
2,000.0 ms
Delivered orders / s, including drain
1.6
Retained order cap
96

Oven lateness measures how far execution runs past the requested deadline. It does not include the configured baking delay. Order-to-door latency includes the entire lifecycle. Each log time axis spans its population's p0.1 to p99.9; the tables list every bin.

The goblin ledger

Business invariants & tenant rankings

Audit passed
96 of 96 acknowledged orders delivered. The independent audit found no invariant violations.
Scores are revenue plus tips, in integer copper coins. Rows follow each tenant’s independently derived leaderboard; this is not a global ranking.
Tenant / kitchenOrdersPizzas deliveredStock leftRevenueTipsScore
tenant-14 / The Crispy Cauldron · store-045108286756105,371106,127
tenant-14 / Dough or Die · store-358386566,0986,154
tenant-14 / Mushroom Mayhem · store-128386566,0816,137
tenant-14 / The Sizzling Slime · store-239385635,9786,041
tenant-15 / The Goblin's Slice · store-03798296686104,360105,046
tenant-15 / Dragon Breath Delivery · store-112392146,2566,270
tenant-15 / Mushroom Mayhem · store-325389356,1336,168
tenant-15 / The Crispy Cauldron · store-214390286,1336,161

A dragon ate the leader

Failover, leases & replay safety

3 Raft replicas · one group

Leader crash 1

2026-09-25 05:45:22 UTC
Crash recovery: leader 1 killed; serving quorum after 676.9 ms; restarted node caught up after 2,744.1 ms. Both durations are measured from the crash.Leader 1 killed0.0 msQuorum serving; leader 3676.9 msRestarted node caught up2,744.1 ms
Recovery observations, all measured from the leader crash
Recovery milestoneTime since crash
Candidate observed in metrics593.6 ms
Leader observed in metrics621.8 ms
Quorum read probe completed676.8 ms
Leader discovery returned676.9 ms
Restarted node caught up2,744.1 ms

Metrics observations follow polling and are not exact internal transition times. A brief candidate state can be missed; observing a leader alone does not establish a serving quorum. Missing milestones were not recorded or did not finish. Recovery and catch-up durations share the crash as time zero.

Receipt replays verified
12,417
Stale drones rejected by audit
16
Leases claimed
112
Empty claims
845
Deliberately abandoned leases
16
Reclaimed jobs
16
Lease losses during workload
0
Orders issued
96
End-of-run replica observations, collected separately. These are replicas of one Raft group, not independent shards.
ReplicaObserved roleObserved leaderLast applied index
Node 1Follower31,126
Node 2Follower31,126
Node 3Leader31,126

The cost of a pizza rush

Resource footprint

Co-located processes

Sampled peak resident memory

Sampled peak resident memory by processPeak observed resident memory per server and benchmark driver. Samples can miss brief peaks; per-process peaks need not occur together. Node 1: Peak sampled RSS 317.1 MiB. Node 2: Peak sampled RSS 288.0 MiB. Node 3: Peak sampled RSS 306.3 MiB. Driver: Peak sampled RSS 404.4 MiB0101202303404Node 1 · Peak sampled RSS: 317.1 MiBNode 1Node 2 · Peak sampled RSS: 288.0 MiBNode 2Node 3 · Peak sampled RSS: 306.3 MiBNode 3Driver · Peak sampled RSS: 404.4 MiBDriver
Peak sampled RSSMiB · linear scale
Cluster disk footprint
137.26 MiB
Node controller sampled load cores
0.05
Node controller CPU, user / entire run
3,698.2 ms
Node controller CPU, system / entire run
444.6 ms
Node controller event-loop delay p99
50.5 ms
Node controller sampled peak RSS
404.4 MiB
Rust customer driver peak RSS
20.6 MiB
Rust customer driver sampled load cores
0.13

RSS is sampled, not an allocation total. Server CPU, when available, uses cumulative process CPU deltas over sampled load intervals; one core means one CPU-second per elapsed second. Restart and phase-boundary gaps are excluded. Older reports did not measure it. The load generator and all nodes share the same host, so these numbers include local resource contention and do not establish multi-host network performance.

CPU observations during load; node coverage can differ. Includes all process threads.
ServerMean CPU coresCPU time coveredWall time coveredIntervals
10.3922,570.0 ms57,253.0 ms57
20.137,900.0 ms59,253.9 ms59
30.4325,550.0 ms59,253.9 ms59

Make the result reproducible

Workload, host & measurement limits

Report schema 1

Workload configuration

Application transport
HTTP/2 · h2c prior knowledge
Admin and discovery transport
HTTP/1.1
Query routing
Round-robin across live replicas
Customer read policy
Replica-local · lag allowed
Bundle initialization
static
Customer loops / drone loops
256 / 4
Requested load / warmup
60.0 s / 1.0 s
Maximum drain
60.0 s
Tenants / stores per tenant
2 / 4
Hot stores per tenant
1
Hot-set mixture weight
80%
Duplicate / abandonment probability
10% / 10%
Worker polling interval
150.0 ms
Request timeout / retry budget
2,000.0 ms / 15,000.0 ms
Workload seed
42:group-7
Crash injection requested
Yes

Execution environment

JavaScript engine
quickjs
Runtime settings
{"TOKIO_WORKER_THREADS":"2","FLOWER_WRITER_BATCH_MODE":"adaptive","FLOWER_WRITER_QUEUE_CAPACITY":"1024","FLOWER_QUERY_WORKERS":"16","FLOWER_PREPARATION_WORKERS":"16","FLOWER_WRITER_PREPARATION_WORKERS":"1","FLOWER_WRITER_WINDOW_MS":"50","FLOWER_WRITER_BATCH_MS":"50"}
CPU
Apple M5 Pro
Logical CPUs
18
Operating system
darwin 27.2.0
Architecture
arm64
Node.js controller
v26.10.0
Customer driver
rust
Customer driver SHA-256
b3ce2a3e598826cd14bf319d0f65cee7a47b3b3822b7474bd1b987d2d5808e7f
Co-located database nodes
3
Binary SHA-256
a53ad2ae79f74da0f9ddf5d79a2450beebd0efb0e7e00f0cdfcd70929f34d35d
Binary size
45,119,536 bytes
Started
2026-09-25 05:44:47 UTC
Finished
2026-09-25 05:45:53 UTC
Preserved database files
Not retained

How to read these numbers

Complete run options and bundle identity
{
  "options": {
    "offeredRate": 0,
    "groups": 1,
    "tenants": 2,
    "duration": 60,
    "warmup": 1,
    "drain": 60,
    "concurrency": 256,
    "workers": 4,
    "shops": 4,
    "hotShops": 1,
    "hotProbability": 0.8,
    "maxOrders": 96,
    "bakeMs": 250,
    "leaseMs": 2000,
    "duplicateRate": 0.1,
    "abandonRate": 0.1,
    "pollMs": 150,
    "requestTimeoutMs": 2000,
    "retryBudgetMs": 15000,
    "nodes": 3,
    "seed": "42:group-7",
    "initialization": "static",
    "queryRouting": "replicas",
    "readConsistency": "replica-local",
    "driver": "rust",
    "driverBinary": "/Users/pcarrier/src/flower/target/release/flower-bench-driver",
    "binary": "/Users/pcarrier/src/flower/target/release/flower",
    "json": "/Users/pcarrier/src/flower/bench/results/latest-groups/group-7.json",
    "html": "/Users/pcarrier/src/flower/bench/results/latest-groups/group-7.html",
    "baseline": null,
    "cpuProfile": null,
    "chaos": true,
    "http2": true,
    "keepData": false,
    "help": false,
    "runId": "d4e5cf27-0a78-41ed-850e-68ccdca97756",
    "tenantIds": [
      "tenant-14",
      "tenant-15"
    ]
  },
  "runtime": {
    "engine": "quickjs",
    "settings": {
      "TOKIO_WORKER_THREADS": "2",
      "FLOWER_WRITER_BATCH_MODE": "adaptive",
      "FLOWER_WRITER_QUEUE_CAPACITY": "1024",
      "FLOWER_QUERY_WORKERS": "16",
      "FLOWER_PREPARATION_WORKERS": "16",
      "FLOWER_WRITER_PREPARATION_WORKERS": "1",
      "FLOWER_WRITER_WINDOW_MS": "50",
      "FLOWER_WRITER_BATCH_MS": "50"
    }
  },
  "queryRouting": {
    "mode": "replicas",
    "consistency": "replica-local",
    "auditConsistency": "fresh",
    "nodes": [
      {
        "id": 1,
        "url": "http://127.0.0.1:62800",
        "attempts": 96081,
        "completed": 95869,
        "failures": 212
      },
      {
        "id": 2,
        "url": "http://127.0.0.1:62807",
        "attempts": 100601,
        "completed": 100601,
        "failures": 0
      },
      {
        "id": 3,
        "url": "http://127.0.0.1:62812",
        "attempts": 100600,
        "completed": 100600,
        "failures": 0
      }
    ]
  },
  "binary": {
    "path": "/Users/pcarrier/src/flower/target/release/flower",
    "sha256": "a53ad2ae79f74da0f9ddf5d79a2450beebd0efb0e7e00f0cdfcd70929f34d35d",
    "bytes": 45119536,
    "modifiedAt": "2026-09-25T05:16:03.202Z"
  },
  "driver": {
    "kind": "rust",
    "orchestration": "node",
    "binary": {
      "path": "/Users/pcarrier/src/flower/target/release/flower-bench-driver",
      "sha256": "b3ce2a3e598826cd14bf319d0f65cee7a47b3b3822b7474bd1b987d2d5808e7f",
      "bytes": 7136160,
      "modifiedAt": "2026-09-25T05:15:43.417Z"
    },
    "warmupRequests": 3
  },
  "bundleHash": "d3b5c3e6fa9e51548200e79cde504fc967a5d7100a49af12a43f27560721ad64"
}

When the oven smokes

Diagnostics

0 unexpected failures

No unexpected failure details were recorded. Intentional stale-lease rejections and retried HTTP failures are reported in their own counters.