Free-threading (3.14t) hardening: Py_mod_gil, UIEntity::pyobject race, unlocked registries #336

New issue

Open

opened 2026-07-02 12:29:41 +00:00 by john · 1 comment

john commented

2026-07-02 12:29:41 +00:00

Owner

From the 2026-07-02 memory-model review. The build toggle already exists (MCRF_FREE_THREADED_PYTHON, CMakeLists.txt:31 -> links python3.14t from __lib_debug/, defines Py_GIL_DISABLED at :401), and the architecture is deliberately close (FrameLock/mcrfpy.lock(), GIL released around display, GILState wrappers on all callbacks, mutex+atomic serial in PythonObjectCache). This issue is the punch list to actually run clean on a free-threaded interpreter.

Gaps, in order

Py_mod_gil slot missing. The module is single-phase (m_size=-1, src/McRFPy_API.cpp:397-435); importing mcrfpy on 3.14t silently re-enables the GIL process-wide. Declaring Py_MOD_GIL_NOT_USED requires converting to multi-phase init (slots) -- do this last, it is the commitment that everything below is true. (Multi-phase init does not require heap types; subinterpreter support remains separately out of scope per #220.)
UIEntity::pyobject temp-swap race: raw PyObject* cleared via non-atomic tmp-swap (src/UIEntity.h:140-142) while tp_dealloc can null it from another thread (src/UIEntity.h:263-264). Needs atomic exchange or protection under the frame lock.
GameEngine::timers (std::map, src/GameEngine.h:218) and AnimationManager singleton vectors (src/Animation.h:196-252) are unlocked but reachable from Python threads.
Static enum-class caches (src/PyKey.h:35 and 9 siblings): init-once on main thread, read-only after -- audit and document that assumption.
Audit the un-locked read surface: property getters reading sf::Text/sf::Sprite state that a lock-holding thread may be mutating. The documented contract (lock() required off-main-thread -- see #327) makes this user error, but failure should not be memory-unsafe. Decide per-getter: tolerate torn reads of plain floats vs. assert-on-thread.
TSan pass under a 3.14t build with worker-thread stress tests (tests/ currently have none). #282 (modern clang) helps here.

Packaging note

Shipping 3.14t doubles the interpreter matrix (and any bundled extension like numpy needs cp314t builds) -- see #337.

From the 2026-07-02 memory-model review. The build toggle already exists (`MCRF_FREE_THREADED_PYTHON`, `CMakeLists.txt:31` -> links `python3.14t` from `__lib_debug/`, defines `Py_GIL_DISABLED` at `:401`), and the architecture is deliberately close (FrameLock/`mcrfpy.lock()`, GIL released around display, GILState wrappers on all callbacks, mutex+atomic serial in `PythonObjectCache`). This issue is the punch list to actually run clean on a free-threaded interpreter. ## Gaps, in order 1. **`Py_mod_gil` slot missing.** The module is single-phase (`m_size=-1`, `src/McRFPy_API.cpp:397-435`); importing mcrfpy on 3.14t silently **re-enables the GIL process-wide**. Declaring `Py_MOD_GIL_NOT_USED` requires converting to multi-phase init (slots) -- do this **last**, it is the commitment that everything below is true. (Multi-phase init does not require heap types; subinterpreter support remains separately out of scope per #220.) 2. **`UIEntity::pyobject` temp-swap race**: raw `PyObject*` cleared via non-atomic tmp-swap (`src/UIEntity.h:140-142`) while `tp_dealloc` can null it from another thread (`src/UIEntity.h:263-264`). Needs atomic exchange or protection under the frame lock. 3. **`GameEngine::timers`** (`std::map`, `src/GameEngine.h:218`) and **`AnimationManager`** singleton vectors (`src/Animation.h:196-252`) are unlocked but reachable from Python threads. 4. **Static enum-class caches** (`src/PyKey.h:35` and 9 siblings): init-once on main thread, read-only after -- audit and document that assumption. 5. **Audit the un-locked read surface**: property getters reading `sf::Text`/`sf::Sprite` state that a lock-holding thread may be mutating. The documented contract (lock() required off-main-thread -- see #327) makes this user error, but failure should not be memory-unsafe. Decide per-getter: tolerate torn reads of plain floats vs. assert-on-thread. 6. TSan pass under a 3.14t build with worker-thread stress tests (`tests/` currently have none). #282 (modern clang) helps here. ## Packaging note Shipping 3.14t doubles the interpreter matrix (and any bundled extension like numpy needs cp314t builds) -- see #337.

john added the

priority:tier3-future

system:python-binding

Major Feature

labels

2026-07-02 12:29:41 +00:00

john referenced this issue

2026-07-02 12:30:41 +00:00

[1.0 Decision] Document the threading contract: mcrfpy.lock() is the rule for off-main-thread access #327

john referenced this issue

2026-07-02 12:30:44 +00:00

Secondary Concurrency Model: Subinterpreter Support #220

john referenced this issue

2026-07-11 15:18:26 +00:00

[Minor Feature] Install modern Clang for improved TSan and fuzzing #282

john referenced this issue

2026-07-11 15:18:34 +00:00

numpy availability strategy: bundle, side-venv, or scientific package variant #337

john commented

2026-07-11 15:18:41 +00:00

Author

Owner

Refinement (2026-07-11): scope + hygiene banner

Not a 1.0 requirement — this is organizational hygiene. Staying tier3. The reason to keep this issue live (rather than dormant) is to prevent complications from accumulating as other work adds shared mutable state — not to gate the API freeze.

Standing instruction: any new global registry / singleton / mutable static introduced by other work gets appended to items 3–4 (the un-locked-registry audit). Keep the punch list current so the eventual 3.14t pass is a checklist, not archaeology.

Agent-env decoupling. The multi-agent orchestration (#156) is turn-based and serial — it does not need free-threading on its critical path. So free-threading is explicitly decoupled from the agent-environment destination (#54/#55/#154); it's parallelism headroom for later, not a dependency of that work.

#282 closed as satisfied (clang-18 installed + used by #312). Item 6 (TSan preset + worker-thread stress tests) now carries that thread here.

The Py_mod_gil / multi-phase-init conversion (item 1) is the one structural item worth not deferring indefinitely, but runtime can stay GIL-ful until the rest of the punch list is true — no urgency, and it does not block anything above.

## Refinement (2026-07-11): scope + hygiene banner **Not a 1.0 requirement — this is organizational hygiene.** Staying tier3. The reason to keep this issue *live* (rather than dormant) is to **prevent complications from accumulating** as other work adds shared mutable state — not to gate the API freeze. **Standing instruction:** any new global registry / singleton / mutable static introduced by other work gets appended to items 3–4 (the un-locked-registry audit). Keep the punch list current so the eventual 3.14t pass is a checklist, not archaeology. **Agent-env decoupling.** The multi-agent orchestration (#156) is **turn-based and serial** — it does not need free-threading on its critical path. So free-threading is explicitly *decoupled* from the agent-environment destination (#54/#55/#154); it's parallelism headroom for later, not a dependency of that work. **#282 closed as satisfied** (clang-18 installed + used by #312). Item 6 (TSan preset + worker-thread stress tests) now carries that thread here. The `Py_mod_gil` / multi-phase-init conversion (item 1) is the one structural item worth not deferring indefinitely, but runtime can stay GIL-ful until the rest of the punch list is true — no urgency, and it does not block anything above.

john referenced this issue

2026-07-11 15:18:55 +00:00

Jupyter Notebook Interface #54

john referenced this issue

2026-07-16 16:47:18 +00:00

[Epic] Web deployment parity & divergence-testing sprint #387

john referenced this issue

2026-07-16 16:48:06 +00:00

[Docs] Promote the Web Build Constraints table into an audited parity matrix with test-coverage column #391