john/McRogueFace
1
0
Fork 0
Code Issues 57 Pull requests Projects Releases Wiki Activity

Gauntlet #340: strike-based hard-cap bail (3 over-cap samples per window)

Browse source 
Three full windowed runs showed 10x-100x per-trial score variance: a single
stray desktop frame >100 ms (compositor/GC hiccup) instantly zeroed trials
whose held p95 was 6-9 ms. Require 3 over-cap samples in a hold window before
bailing -- a truly overloaded engine trips that within ~50 ms of wall time,
while lone hiccups now just count toward the window's p95. Documented as
DESIGN.md deviations 8 and 9.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
John McCardle 2026-07-02 20:39:11 -04:00
commit 1aa6624758
2 changed files with 26 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

9
tests/benchmarks/gauntlet/DESIGN.md
View file

@ -31,6 +31,15 @@ preserved verbatim after this section.)*
setup / set_load / teardown directly, not the ramp.
7. Animations use the current `obj.animate(...)` method (the older
`mcrfpy.Animation(...).start()` seen in some demos is not used).
8. **Hard-cap bail requires 3 over-cap samples per window, not 1.** The spec's "bail
immediately if any sample > 100 ms" was observed zeroing entire trials on a live
desktop: a single stray compositor/GC frame >100 ms ended trials whose held p95 was
6-9 ms, producing 10x-100x run-to-run score variance. A genuinely overloaded engine
accumulates 3 strikes within ~50 ms of wall time, so the bail still triggers almost
immediately under true overload; lone hiccups still count toward the window's p95.
9. **Pathfinder Rush starts its ramp at 1 query/tick** (spec table implied a larger
start via the shared default). Queries burst on one 100 ms tick, so even a handful
spike a single frame; starting at 1 lets the ramp find the true sustainable level.
---

21
tests/benchmarks/gauntlet/scoring.py
View file

@ -6,7 +6,13 @@ collects frame-time samples over a hold window, and decides pass/fail:
pass : p95 <= budget_ms -> record as new peak, ramp load up one step
fail : p95 > budget_ms -> stop; score = last passing load
bail : any sample > hard_cap_ms -> stop immediately (fail)
bail : HARD_CAP_STRIKES samples > hard_cap_ms in one window -> stop (fail)
The bail requires sustained overload (3 strikes) rather than a single sample:
on a live desktop one stray >100 ms frame (compositor, GC, notification) is
common and was observed zeroing entire trials, while a genuinely overloaded
engine trips 3 strikes within ~50 ms of wall time anyway. Lone spikes still
count toward p95. (Deviation 8 in DESIGN.md.)
frame_time is read from get_metrics()["frame_time"] and is already in
milliseconds in this engine build (see DESIGN.md deviation 1).
@ -14,6 +20,7 @@ milliseconds in this engine build (see DESIGN.md deviation 1).
BUDGET_MS = 16.67
HARD_CAP_MS = 100.0
HARD_CAP_STRIKES = 3
SETTLE_MS = 1000
HOLD_MS = 2000
MAX_STEPS = 40 # safety ceiling so a trial that never breaks budget still terminates
@ -59,7 +66,9 @@ class RampController:
def __init__(self, trial, metrics_provider,
budget_ms=BUDGET_MS, hard_cap_ms=HARD_CAP_MS,
settle_ms=SETTLE_MS, hold_ms=HOLD_MS, max_steps=MAX_STEPS,
on_finish=None):
hard_cap_strikes=HARD_CAP_STRIKES, on_finish=None):
self.hard_cap_strikes = hard_cap_strikes
self.strikes = 0
self.trial = trial
self.metrics_provider = metrics_provider # callable -> full get_metrics() dict
self.budget_ms = budget_ms
@ -86,6 +95,7 @@ class RampController:
self.phase = "settle"
self.phase_start = None
self.samples = []
self.strikes = 0
self.last_pass = None
self.done = False
self.result = None
@ -114,13 +124,16 @@ class RampController:
self.phase = "hold"
self.phase_start = now_ms
self.samples = []
self.strikes = 0
return
# hold phase
self.samples.append(ft)
if ft > self.hard_cap_ms:
self._evaluate(metrics, forced_fail=True)
return
self.strikes += 1
if self.strikes >= self.hard_cap_strikes:
self._evaluate(metrics, forced_fail=True)
return
if elapsed >= self.hold_ms:
self._evaluate(metrics, forced_fail=False)