[Minor Feature] The Crucible — headless wall-clock microbenchmark for cross-version A/B #354

New issue

Closed

opened 2026-07-11 14:27:50 +00:00 by john · 0 comments

john commented

2026-07-11 14:27:50 +00:00

Owner

Motivation

The Gauntlet (#353) answers "how much load until the frame budget breaks?" — it needs a display, varies run-to-run with desktop noise, and its load-ramp does not cleanly isolate the CPU-bound engine paths that the #329/#332/#348 perf sprint actually changed (allocation, cell writes, TCOD sync, turn manager, pathfinding, FOV).

The Crucible (tests/benchmarks/crucible.py) answers the complementary question: "How many milliseconds of CPU does the engine spend on a FIXED, comically-extreme-but-tractable amount of work?"

Design

Fixed configs, run to completion, timed with time.perf_counter (fastest of N reps). No ramp, no display, no rendering.
Safe — every config fits well under 512 MB and finishes in a couple seconds; peak RSS ~100 MB for the whole run. Installs the #353 RLIMIT_AS backstop anyway.
Headless — pure engine CPU paths; no frame_time dependency.
Deterministic — fixed seeds, so A/B deltas between builds are real signal.
Cross-version — a bench that hits a missing API on an older/newer build reports unsupported instead of crashing the run. Emits JSON (MCRF_CRUCIBLE_OUT); diff a prior build's JSON with MCRF_CRUCIBLE_BASELINE.

Benches: grid_alloc, grid_fill, layer_fill, layer_edit_buffer (#335), step_swarm, fov_storm, path_queries, entity_churn.

First result — current master vs the 0.2.8 release artifact

Safe replacement for the windowed-Gauntlet A/B that crashed the desktop (#353). Both binaries, headless, same script:

bench                   base_ms(0.2.8)  now_ms   delta
grid_alloc               108.14          44.49   -58.9%
grid_fill                128.22          95.59   -25.4%
layer_fill                 5.39           2.36   -56.1%
layer_edit_buffer            --             --   unsupp (new #335 API)
step_swarm                 9.78           9.18    -6.2%
fov_storm                 80.17          71.91   -10.3%
path_queries              89.58          86.70    -3.2%
entity_churn              42.74          24.35   -43.0%
------------------------------------------------------
geomean now/base: 0.673  (32.7% faster overall)

The big wins (grid_alloc, entity_churn, layer_fill) track the #332 SoA plane rewrite + #329 entity indexing; FOV/path/step are modest (not the sprint's targets). Peak RSS also down (~103 MB vs ~119 MB).

Caveat

The JSON commit field reflects the script's repo HEAD, not the binary under test (the script can only read its own .git); version (mcrfpy.__version__) is the authoritative binary identifier. Could stamp binary provenance more explicitly later.

## Motivation The Gauntlet (#353) answers *"how much load until the frame budget breaks?"* — it needs a display, varies run-to-run with desktop noise, and its load-ramp does not cleanly isolate the CPU-bound engine paths that the #329/#332/#348 perf sprint actually changed (allocation, cell writes, TCOD sync, turn manager, pathfinding, FOV). **The Crucible** (`tests/benchmarks/crucible.py`) answers the complementary question: *"How many milliseconds of CPU does the engine spend on a FIXED, comically-extreme-but-tractable amount of work?"* ## Design - **Fixed configs**, run to completion, timed with `time.perf_counter` (fastest of N reps). No ramp, no display, no rendering. - **Safe** — every config fits well under 512 MB and finishes in a couple seconds; peak RSS ~100 MB for the whole run. Installs the #353 `RLIMIT_AS` backstop anyway. - **Headless** — pure engine CPU paths; no `frame_time` dependency. - **Deterministic** — fixed seeds, so A/B deltas between builds are real signal. - **Cross-version** — a bench that hits a missing API on an older/newer build reports `unsupported` instead of crashing the run. Emits JSON (`MCRF_CRUCIBLE_OUT`); diff a prior build's JSON with `MCRF_CRUCIBLE_BASELINE`. Benches: `grid_alloc`, `grid_fill`, `layer_fill`, `layer_edit_buffer` (#335), `step_swarm`, `fov_storm`, `path_queries`, `entity_churn`. ## First result — current master vs the 0.2.8 release artifact Safe replacement for the windowed-Gauntlet A/B that crashed the desktop (#353). Both binaries, headless, same script: ``` bench base_ms(0.2.8) now_ms delta grid_alloc 108.14 44.49 -58.9% grid_fill 128.22 95.59 -25.4% layer_fill 5.39 2.36 -56.1% layer_edit_buffer -- -- unsupp (new #335 API) step_swarm 9.78 9.18 -6.2% fov_storm 80.17 71.91 -10.3% path_queries 89.58 86.70 -3.2% entity_churn 42.74 24.35 -43.0% ------------------------------------------------------ geomean now/base: 0.673 (32.7% faster overall) ``` The big wins (`grid_alloc`, `entity_churn`, `layer_fill`) track the #332 SoA plane rewrite + #329 entity indexing; FOV/path/step are modest (not the sprint's targets). Peak RSS also down (~103 MB vs ~119 MB). ## Caveat The JSON `commit` field reflects the **script's** repo HEAD, not the binary under test (the script can only read its own `.git`); `version` (`mcrfpy.__version__`) is the authoritative binary identifier. Could stamp binary provenance more explicitly later.