[Bugfix] The Gauntlet can OOM / hard-lock the machine — GRID TITAN ramps grid side unbounded #353

New issue

Closed

opened 2026-07-11 14:27:30 +00:00 by john · 0 comments

john commented

2026-07-11 14:27:30 +00:00

Owner

Summary

Running tests/benchmarks/gauntlet/run_gauntlet.py windowed hard-locked the desktop (2026-07-11): the machine consumed all RAM during grid allocation, could not switch to a tty, and forced a logout. Root cause: the auto-ramp has a step ceiling but no memory ceiling.

Root cause

scoring.py ramps load geometrically: load_k = round(base * growth**k), capped at MAX_STEPS = 40. For GRID TITAN the load is the grid side S (base_load=20, growth=1.4), and cost/allocation scales as S² (cells, TileLayer int32, ColorLayer RGBA, TCOD map).

Because that trial renders into a fixed viewport (zoom = VIEW_PX/(S*16)), the whole grid stays on-screen at shrinking zoom, so frame time can stay under budget while S grows. The ramp therefore keeps passing and climbing:

step k	S = 20·1.4^k	cells (S²)	~grid-data bytes
18	~8500	72M	~0.7 GB
20	~16700	279M	~2.8 GB
22	~32700	1.07B	~10 GB → OOM

MAX_STEPS=40 would nominally reach S≈14,000,000 — the OOM-killer / swap-thrash hits ~step 20-22 first, locking the machine.

Fix (implemented)

Defense in depth, weakest failure mode last (tests/benchmarks/gauntlet/safety.py):

Trial.predict_bytes(load) — a trial that can estimate its footprint refuses a load whose predicted allocation exceeds a 512 MB budget before allocating it. GRID TITAN implements it (S²·~28 B) and declares max_load = 4300 (≈494 MB at the cap).
Trial.max_load — absolute hard cap on load; the ramp never sets a load above it.
RSS watchdog — after any set_load, if resident memory exceeds a ceiling (default 1500 MB, MCRF_GAUNTLET_RSS_MB) the ramp stops gracefully.
RLIMIT_AS address-space cap — run_gauntlet.py installs a hard cap (default 2500 MB, MCRF_GAUNTLET_AS_MB) at startup. Verified: under the cap a runaway Grid() throws std::bad_alloc → abort (clean process death, exit 134) instead of taking the machine down. This is the machine-saver backstop, not a graceful path.

RampController records a stop_reason (budget / hard_cap / max_load / mem_predict / mem_rss / max_steps).

Test: tests/unit/gauntlet_safety_test.py proves the ramp bails on the memory guards before over-allocating (via a synthetic runaway trial that records the largest load it was asked to build), and exercises the RSS/address-space probes.

Note

Also observed and fixed a separate #351 render regression during the same run (turn-manager entity moves not repainting) — committed separately (addresses #351).

## Summary Running `tests/benchmarks/gauntlet/run_gauntlet.py` windowed **hard-locked the desktop** (2026-07-11): the machine consumed all RAM during grid allocation, could not switch to a tty, and forced a logout. Root cause: the auto-ramp has a *step* ceiling but no *memory* ceiling. ## Root cause `scoring.py` ramps load geometrically: `load_k = round(base * growth**k)`, capped at `MAX_STEPS = 40`. For **GRID TITAN** the load is the grid **side** `S` (`base_load=20`, `growth=1.4`), and cost/allocation scales as **S²** (cells, TileLayer int32, ColorLayer RGBA, TCOD map). Because that trial renders into a fixed viewport (`zoom = VIEW_PX/(S*16)`), the whole grid stays on-screen at shrinking zoom, so **frame time can stay under budget while S grows**. The ramp therefore keeps passing and climbing: | step k | S = 20·1.4^k | cells (S²) | ~grid-data bytes | |--------|--------------|-----------|------------------| | 18 | ~8500 | 72M | ~0.7 GB | | 20 | ~16700 | 279M | ~2.8 GB | | 22 | ~32700 | 1.07B | ~10 GB → OOM | `MAX_STEPS=40` would nominally reach `S≈14,000,000` — the OOM-killer / swap-thrash hits ~step 20-22 first, locking the machine. ## Fix (implemented) Defense in depth, weakest failure mode last (`tests/benchmarks/gauntlet/safety.py`): 1. **`Trial.predict_bytes(load)`** — a trial that can estimate its footprint refuses a load whose predicted allocation exceeds a 512 MB budget *before* allocating it. GRID TITAN implements it (`S²·~28 B`) and declares `max_load = 4300` (≈494 MB at the cap). 2. **`Trial.max_load`** — absolute hard cap on load; the ramp never sets a load above it. 3. **RSS watchdog** — after any `set_load`, if resident memory exceeds a ceiling (default 1500 MB, `MCRF_GAUNTLET_RSS_MB`) the ramp stops gracefully. 4. **`RLIMIT_AS` address-space cap** — `run_gauntlet.py` installs a hard cap (default 2500 MB, `MCRF_GAUNTLET_AS_MB`) at startup. Verified: under the cap a runaway `Grid()` throws `std::bad_alloc → abort` (clean process death, exit 134) instead of taking the machine down. This is the machine-saver backstop, not a graceful path. `RampController` records a `stop_reason` (`budget` / `hard_cap` / `max_load` / `mem_predict` / `mem_rss` / `max_steps`). Test: `tests/unit/gauntlet_safety_test.py` proves the ramp bails on the memory guards *before* over-allocating (via a synthetic runaway trial that records the largest load it was asked to build), and exercises the RSS/address-space probes. ## Note Also observed and fixed a separate **#351 render regression** during the same run (turn-manager entity moves not repainting) — committed separately (addresses #351).