[Keystone] Cross-backend screenshot diff harness (desktop-headless oracle vs web canvas) #388

New issue

Open

opened 2026-07-16 16:47:33 +00:00 by john · 0 comments

john commented

2026-07-16 16:47:33 +00:00

Owner

Part of #387. This is the sprint keystone — it composes #347, #239, and #381 into one automated signal.

Idea

#381 already produces a deterministic PNG per snippet on the desktop headless backend. #239 stands up a headless-browser runner for the web (SDL2/WebGL) backend. This harness closes the loop: render the same snippet corpus on the web backend, capture the canvas, and diff against the desktop oracle. A diff above threshold = a desktop↔web divergence to triage.

Scope

Web-side capture: load each snippet in the headless browser (via #239's rig), let it render deterministically, read back the canvas pixels (gl.readPixels / canvas.toDataURL / toBlob), save as PNG keyed by snippet stem.
Diff: compare web PNG against the #381 desktop PNG. Start with exact byte diff; expect it to fail on legitimate AA/subpixel differences, so add a perceptual/threshold mode (per-pixel channel delta with a tolerance + a max-fraction-of-differing-pixels gate).
Report: per-snippet PASS / DIVERGE with the diff magnitude; write divergence images to an artifacts dir for human triage. Feed confirmed divergences back to #347 as granular defect issues.
Determinism prerequisites: the corpus must render frame-identically on both backends (fixed seed, no wall-clock animation in the captured frame). Note any snippet that can't be made deterministic and exclude it explicitly (Fail-Early: no silent skips).

Non-goals

Not a general browser E2E suite (that's #239).
Not fixing the divergences (that's #347 + the SDL hardening issue) — this only detects them.

Dependencies

Blocked by #347 (need the parity investigation to know what's expected to differ), #239 (browser rig), #381 (desktop oracle).

Part of #387. This is the sprint keystone — it composes #347, #239, and #381 into one automated signal. ## Idea #381 already produces a deterministic PNG per snippet on the **desktop headless** backend. #239 stands up a **headless-browser** runner for the **web (SDL2/WebGL)** backend. This harness closes the loop: render the same snippet corpus on the web backend, capture the canvas, and diff against the desktop oracle. A diff above threshold = a desktop↔web divergence to triage. ## Scope 1. **Web-side capture**: load each snippet in the headless browser (via #239's rig), let it render deterministically, read back the canvas pixels (`gl.readPixels` / `canvas.toDataURL` / `toBlob`), save as PNG keyed by snippet stem. 2. **Diff**: compare web PNG against the #381 desktop PNG. Start with exact byte diff; expect it to fail on legitimate AA/subpixel differences, so add a perceptual/threshold mode (per-pixel channel delta with a tolerance + a max-fraction-of-differing-pixels gate). 3. **Report**: per-snippet PASS / DIVERGE with the diff magnitude; write divergence images to an artifacts dir for human triage. Feed confirmed divergences back to #347 as granular defect issues. 4. **Determinism prerequisites**: the corpus must render frame-identically on both backends (fixed seed, no wall-clock animation in the captured frame). Note any snippet that can't be made deterministic and exclude it explicitly (Fail-Early: no silent skips). ## Non-goals - Not a general browser E2E suite (that's #239). - Not fixing the divergences (that's #347 + the SDL hardening issue) — this only *detects* them. ## Dependencies Blocked by #347 (need the parity investigation to know what's expected to differ), #239 (browser rig), #381 (desktop oracle).