john/McRogueFace

Fork 0

[Tooling] Deterministic snippet screenshots as a visual-regression oracle (rendered into the doc-site repo, not the engine) #381

New issue

Open

opened 2026-07-16 00:27:46 +00:00 by john · 1 comment

john commented

2026-07-16 00:27:46 +00:00

Owner

Goal

Generate a rendered preview PNG for each of the ~283 gated snippets in tests/snippets/ and surface it on the docs site next to the code. The snippets-as-tests harness already proves each one runs clean and leaves a scene with children — i.e. every snippet is renderable — so this is a pipeline extension, not new engine capability.

Requirements

Screenshots are a visual-regression oracle, not decoration

A changed image is a signal that behavior changed and needs review — the same golden/snapshot discipline we already apply to the API surface. This is the whole reason to make them deterministic: a diff in the rendered output is a reviewable event, not noise.

Deterministic capture

Seed Python random and the engine RNG in the harness so procgen/WFC/cellular snippets (086_procgen_cellular, 163_grid_dungeon_generator, 170_procgen_cellular_caves, 171_procgen_wfc, 273_part_03_dungeon_generation, ...) reproduce byte-for-byte.
Any snippet that is non-deterministic by nature must be either seeded or explicitly opted out (per-snippet metadata, likely an extension of the # mcrf: header the stamper already owns).

Motion → sim-time frame selection

For animated/timer snippets, advance the render loop with step() to a chosen sim time and capture the frame that best demonstrates the behavior — deterministic frame selection, not an arbitrary "settled" grab. #350 (headless step() now runs updatePythonScenes()) unblocks this; step() also fires exactly one timer event per call regardless of dt, so frame selection must account for that.

Storage: doc-site repo, NEVER the engine repo

PNGs are generated artifacts. By the docs-pipeline rule (the site pulls from the engine; generated things are regenerated, never hand-edited), a make snippet-shots target in the engine writes to a gitignored dir; the doc-site build consumes them and the site repo is where they're committed. The engine repo must not accumulate hundreds of binaries.

Sketch of the pieces

Harness extension (tests/snippets/_harness.py path) to seed RNG + automation.screenshot() at the selected frame.
make snippet-shots target → gitignored output dir.
Per-snippet frame/seed/opt-out convention (extend # mcrf: header + tools/stamp_snippets.py).
Site build_library.py consumption + commit into the doc-site repo.

Docs-as-tests pipeline (engine commits 43a6cc3 → 07442dd; make {docs,stamp-snippets,check-snippets,release-docs}).
Sibling issue: playground "try it live" links + version-pinned source refs (filed alongside — see next issue).
Follow-ups from the same thread: #380 (templates ungated), #374 (tests/demo/screens orphans).

## Goal Generate a rendered preview PNG for each of the ~283 gated snippets in `tests/snippets/` and surface it on the docs site next to the code. The snippets-as-tests harness already proves each one **runs clean and leaves a scene with children** — i.e. every snippet is renderable — so this is a pipeline extension, not new engine capability. ## Requirements ### Screenshots are a visual-regression oracle, not decoration A **changed image is a signal that behavior changed and needs review** — the same golden/snapshot discipline we already apply to the API surface. This is the whole reason to make them deterministic: a diff in the rendered output is a reviewable event, not noise. ### Deterministic capture - Seed Python `random` **and** the engine RNG in the harness so procgen/WFC/cellular snippets (`086_procgen_cellular`, `163_grid_dungeon_generator`, `170_procgen_cellular_caves`, `171_procgen_wfc`, `273_part_03_dungeon_generation`, ...) reproduce byte-for-byte. - Any snippet that is non-deterministic by nature must be either seeded or explicitly opted out (per-snippet metadata, likely an extension of the `# mcrf:` header the stamper already owns). ### Motion → sim-time frame selection For animated/timer snippets, advance the render loop with `step()` to a **chosen sim time** and capture the frame that best demonstrates the behavior — deterministic frame selection, not an arbitrary "settled" grab. #350 (headless `step()` now runs `updatePythonScenes()`) unblocks this; `step()` also fires exactly one timer event per call regardless of dt, so frame selection must account for that. ### Storage: doc-site repo, NEVER the engine repo PNGs are generated artifacts. By the docs-pipeline rule (*the site pulls from the engine; generated things are regenerated, never hand-edited*), a `make snippet-shots` target in the engine writes to a **gitignored** dir; the **doc-site build consumes them and the site repo is where they're committed**. The engine repo must not accumulate hundreds of binaries. ## Sketch of the pieces - Harness extension (`tests/snippets/_harness.py` path) to seed RNG + `automation.screenshot()` at the selected frame. - `make snippet-shots` target → gitignored output dir. - Per-snippet frame/seed/opt-out convention (extend `# mcrf:` header + `tools/stamp_snippets.py`). - Site `build_library.py` consumption + commit into the doc-site repo. ## Related - Docs-as-tests pipeline (engine commits `43a6cc3` → `07442dd`; `make {docs,stamp-snippets,check-snippets,release-docs}`). - Sibling issue: playground "try it live" links + version-pinned source refs (filed alongside — see next issue). - Follow-ups from the same thread: #380 (templates ungated), #374 (`tests/demo/screens` orphans).

john added the

Minor Feature

priority:tier2-foundation

system:documentation

system:rendering

labels

2026-07-16 00:27:46 +00:00

john referenced this issue

2026-07-16 00:28:04 +00:00

[Tooling] "Open in Playground" links from doc snippets — url-encoded short, raw-URL fetch long — with tag/commit version pinning #382

john commented

2026-07-16 00:44:48 +00:00

Author

Owner

Audit results (4-agent fan-out over all 282 snippets)

Capture-strategy taxonomy

Most snippets render their headline visual at load — many that attach on_key/on_click are already fully drawn, so a plain post-setup screenshot is correct (input handler is a secondary tag, not an auto-trigger). Distribution under that lens:

Strategy	~count	Capture
STATIC (incl. static-with-secondary-input)	~145+	setup → N steps → screenshot
ANIMATION	~77	step to a target time (~50–75% of duration)
INTERACTION (payload hidden until input)	~25 truly	automate the specific action → screenshot
NO-VISUAL / opt-out	10	`noshot`: 142, 161, 225, 226, 251, 253, 254, 255, 265, 268

A default STATIC capture (setup + 3 steps + screenshot) covers ~180 snippets with zero per-snippet tuning. Only animation (target time) and truly-hidden interaction (modals/tooltips) need hand-annotation.

Metadata model (`# mcrf:` header extension)

Not a 3-way enum — STATIC is the degenerate default:

{ noshot?: bool, self_driven?: bool, setup_steps: int=3, action?: [automation calls], shot_at?: float }

Traps to encode (not infer)

Handler-only-prints (208, 234, 244, 245, 249, 250): STATIC, don't automate.
246_Keyboard: on_key calls undefined functions → a keypress raises. Never send keys.
130_exit_game: ESC exits the process → never fire its trigger.
Self-driving (168 calls step(), 169 calls automation.screenshot()): don't double-drive.
Fade-to-invisible (227): needs an early frame or it's blank. 215/217 resolve instantly → static.
Static-with-timer-caption (085, 091): need ≥1 step past the timer or the label is placeholder.
Headless mouse is (0,0): hover/tooltip snippets (051, 053, 074, 200–203, 229, 252) need simulated mouse movement — the hard tail.

Determinism — resolved, stays harness-only

25 of 29 random snippets go byte-stable with one random.seed(N) in _harness.py (shared interpreter RNG). NoiseSource/HeightMap snippets already pass explicit seed=. The only gap was BSP: split_recursive() with no seed= pulls libtcod's time(0)-seeded global RNG (mersenne_c.c:150), unreachable from Python (there is no mcrfpy.seed()). Of the 7 BSP snippets, 3 are noshot text-only demos, so only 4 need deterministic pixels: 060, 102, 224, 243.

Decision: edit those 4 to pass seed=42 (also better docs — shows users how to get a reproducible dungeon). No engine change; #381 stays a Minor Feature.

Proposed phasing

Phase 1 — random.seed() in _harness.py + seed=42 in the 4 BSP snippets; default STATIC capture (make snippet-shots → gitignored dir); prove byte-stability by capturing twice and diffing. Ships previews for ~180+ snippets immediately.
Phase 2 — shot_at target-time annotations for the ~77 ANIMATION snippets.
Phase 3 — scripted action for the ~25 hidden-interaction snippets + headless mouse simulation.
noshot on the 10 opt-outs throughout.

## Audit results (4-agent fan-out over all 282 snippets) ### Capture-strategy taxonomy Most snippets **render their headline visual at load** — many that attach `on_key`/`on_click` are already fully drawn, so a plain post-setup screenshot is correct (input handler is a *secondary* tag, not an auto-trigger). Distribution under that lens: | Strategy | ~count | Capture | |---|---|---| | STATIC (incl. static-with-secondary-input) | ~145+ | setup → N steps → screenshot | | ANIMATION | ~77 | step to a target time (~50–75% of duration) | | INTERACTION (payload hidden until input) | ~25 truly | automate the specific action → screenshot | | NO-VISUAL / opt-out | 10 | `noshot`: 142, 161, 225, 226, 251, 253, 254, 255, 265, 268 | **A default STATIC capture (setup + 3 steps + screenshot) covers ~180 snippets with zero per-snippet tuning.** Only animation (target time) and truly-hidden interaction (modals/tooltips) need hand-annotation. ### Metadata model (`# mcrf:` header extension) Not a 3-way enum — STATIC is the degenerate default: ``` { noshot?: bool, self_driven?: bool, setup_steps: int=3, action?: [automation calls], shot_at?: float } ``` ### Traps to encode (not infer) - **Handler-only-prints** (208, 234, 244, 245, 249, 250): STATIC, don't automate. - **246_Keyboard**: `on_key` calls undefined functions → a keypress raises. Never send keys. - **130_exit_game**: ESC exits the process → never fire its trigger. - **Self-driving** (168 calls `step()`, 169 calls `automation.screenshot()`): don't double-drive. - **Fade-to-invisible** (227): needs an early frame or it's blank. 215/217 resolve instantly → static. - **Static-with-timer-caption** (085, 091): need ≥1 step past the timer or the label is placeholder. - **Headless mouse is (0,0)**: hover/tooltip snippets (051, 053, 074, 200–203, 229, 252) need simulated mouse movement — the hard tail. ### Determinism — resolved, stays harness-only 25 of 29 random snippets go byte-stable with one `random.seed(N)` in `_harness.py` (shared interpreter RNG). `NoiseSource`/`HeightMap` snippets already pass explicit `seed=`. The only gap was BSP: `split_recursive()` with no `seed=` pulls libtcod's `time(0)`-seeded global RNG (`mersenne_c.c:150`), unreachable from Python (there is no `mcrfpy.seed()`). Of the 7 BSP snippets, 3 are `noshot` text-only demos, so only **4 need deterministic pixels: 060, 102, 224, 243**. **Decision:** edit those 4 to pass `seed=42` (also better docs — shows users how to get a reproducible dungeon). No engine change; #381 stays a Minor Feature. ### Proposed phasing 1. **Phase 1** — `random.seed()` in `_harness.py` + `seed=42` in the 4 BSP snippets; default STATIC capture (`make snippet-shots` → gitignored dir); prove byte-stability by capturing twice and diffing. Ships previews for ~180+ snippets immediately. 2. **Phase 2** — `shot_at` target-time annotations for the ~77 ANIMATION snippets. 3. **Phase 3** — scripted `action` for the ~25 hidden-interaction snippets + headless mouse simulation. 4. `noshot` on the 10 opt-outs throughout.

john referenced this issue

2026-07-16 01:08:52 +00:00

Headless Timer epoch uses wall clock (runtime), not simulation_time — first fire under step() is nondeterministic #383

john referenced this issue from a commit

2026-07-16 03:56:24 +00:00

feat(docs): deterministic snippet screenshots — Phase 1 (STATIC capture)

john referenced this issue from a commit

2026-07-16 03:56:24 +00:00

feat(docs): animation target-time capture — Phase 2 (#381)

john referenced this issue from a commit

2026-07-16 03:56:24 +00:00

fix(timer): epoch headless timers from simulation_time, not wall clock