john/McRogueFace

Fork 0

[Bugfix] The light/full stdlib preset filter is unreachable dead code — and produces an unbootable interpreter if revived #399

New issue

Open

opened 2026-07-27 03:39:04 +00:00 by john · 0 comments

john commented

2026-07-27 03:39:04 +00:00

Owner

Summary

tools/stdlib_modules.yaml + tools/package_stdlib.py describe a light/full split of the Python standard library. Neither preset is actually applied to any shipped package, and the full preset — the one CLAUDE.md advertises as "includes networking, async, debugging modules" — cannot even import asyncio if you do apply it.

Found while gating the "the shipped interpreter can do sockets/asyncio from a background thread" guarantee (see the packaging test added alongside this issue).

1. The filter is never invoked

tools/package.sh defines create_stdlib_zip() (:91-103), which is the only caller of package_stdlib.py. Nothing calls create_stdlib_zip. Confirmed:

$ grep -n "create_stdlib_zip" tools/package.sh
91:create_stdlib_zip() {

Instead, package_linux() copies the entire stdlib regardless of preset (tools/package.sh:296-301):

# For now, copy entire Lib - filtering happens via package_stdlib.py for zip
# TODO: Implement directory-based filtering for light preset
cp -r "$PROJECT_ROOT/__lib/Python/Lib" "$package_dir/lib/Python/"

package_windows() has the same shape (:177-195). So package-linux-light and package-linux-full ship byte-identical stdlibs; the only real filtering is the hardcoded rm -rf list right below (test/, idlelib/, tkinter/, ...). The documented ~25MB vs ~26MB split does not exist as a mechanism.

2. The `full` preset is broken if revived

Materializing the full preset's module list against __lib/Python/Lib and booting the packaged engine against it:

Traceback (most recent call last):
  File "<string>", line 5, in <module>
  File ".../Lib/ssl.py", line 265, in <module>
    import warnings
  File ".../Lib/warnings.py", line 15, in <module>
    from _py_warnings import (
ModuleNotFoundError: No module named '_py_warnings'

Two distinct gaps found so far:

contextvars is in no category at all, so it is in neither preset. asyncio imports it unconditionally — full therefore cannot import asyncio, which is the single headline module the preset exists to provide.
_py_warnings is in no category. In CPython 3.14 warnings.py delegates to it, so core itself is incomplete — every preset, light included, fails to import warnings, and with it ssl, asyncio and much of the stdlib.

The _py_warnings case in particular means the module lists were written against an older CPython and have never been validated against the 3.14 tree actually shipped. There are likely more; the list should be derived and checked, not hand-maintained.

Why it matters

The categorization reads as a live contract — CLAUDE.md cites it, and a future edit "tidying up" the network:/async: categories would look effective while changing nothing, or would silently break everything the day someone wires the filter up. Either way the failure surfaces far from the edit.

Suggested resolution

Pick one and make it true:

Retire the mechanism: delete create_stdlib_zip() and either drop package_stdlib.py/stdlib_modules.yaml or clearly mark them unused. Document that all packages ship the full stdlib minus the hardcoded rm -rf list.
Implement it properly: wire directory filtering into package_linux()/package_windows(), fix the missing modules, and gate each preset with a boot test that imports a representative module set from the packaged artifact.

Option 2 needs a real dependency-closure step (walk imports from the seed list) rather than a hand-written module list, or it will drift again.

Current state of the guarantee

make package-linux-full does ship working socket, ssl, asyncio and selectors today — verified from the extracted artifact, resolving out of the package's own lib/Python/Lib — but only because the preset is ignored and the whole stdlib is copied. tests/packaging/network_stdlib_packaged_test.py now gates that outcome regardless of which resolution above is chosen.

#337 — numpy availability strategy (same "what does a distribution contain" question)
#380 — shipped artifacts gated by nothing

## Summary `tools/stdlib_modules.yaml` + `tools/package_stdlib.py` describe a light/full split of the Python standard library. **Neither preset is actually applied to any shipped package**, and the `full` preset — the one CLAUDE.md advertises as "includes networking, async, debugging modules" — cannot even import `asyncio` if you do apply it. Found while gating the "the shipped interpreter can do sockets/asyncio from a background thread" guarantee (see the packaging test added alongside this issue). ## 1. The filter is never invoked `tools/package.sh` defines `create_stdlib_zip()` (`:91-103`), which is the only caller of `package_stdlib.py`. **Nothing calls `create_stdlib_zip`.** Confirmed: ``` $ grep -n "create_stdlib_zip" tools/package.sh 91:create_stdlib_zip() { ``` Instead, `package_linux()` copies the entire stdlib regardless of preset (`tools/package.sh:296-301`): ```bash # For now, copy entire Lib - filtering happens via package_stdlib.py for zip # TODO: Implement directory-based filtering for light preset cp -r "$PROJECT_ROOT/__lib/Python/Lib" "$package_dir/lib/Python/" ``` `package_windows()` has the same shape (`:177-195`). So `package-linux-light` and `package-linux-full` ship **byte-identical stdlibs**; the only real filtering is the hardcoded `rm -rf` list right below (test/, idlelib/, tkinter/, ...). The documented ~25MB vs ~26MB split does not exist as a mechanism. ## 2. The `full` preset is broken if revived Materializing the `full` preset's module list against `__lib/Python/Lib` and booting the packaged engine against it: ``` Traceback (most recent call last): File "<string>", line 5, in <module> File ".../Lib/ssl.py", line 265, in <module> import warnings File ".../Lib/warnings.py", line 15, in <module> from _py_warnings import ( ModuleNotFoundError: No module named '_py_warnings' ``` Two distinct gaps found so far: - **`contextvars`** is in no category at all, so it is in neither preset. `asyncio` imports it unconditionally — `full` therefore cannot import `asyncio`, which is the single headline module the preset exists to provide. - **`_py_warnings`** is in no category. In CPython 3.14 `warnings.py` delegates to it, so *`core` itself* is incomplete — every preset, light included, fails to import `warnings`, and with it `ssl`, `asyncio` and much of the stdlib. The `_py_warnings` case in particular means the module lists were written against an older CPython and have never been validated against the 3.14 tree actually shipped. There are likely more; the list should be derived and checked, not hand-maintained. ## Why it matters The categorization reads as a live contract — CLAUDE.md cites it, and a future edit "tidying up" the `network:`/`async:` categories would look effective while changing nothing, or would silently break everything the day someone wires the filter up. Either way the failure surfaces far from the edit. ## Suggested resolution Pick one and make it true: 1. **Retire the mechanism**: delete `create_stdlib_zip()` and either drop `package_stdlib.py`/`stdlib_modules.yaml` or clearly mark them unused. Document that all packages ship the full stdlib minus the hardcoded `rm -rf` list. 2. **Implement it properly**: wire directory filtering into `package_linux()`/`package_windows()`, fix the missing modules, and gate each preset with a boot test that imports a representative module set from the packaged artifact. Option 2 needs a real dependency-closure step (walk imports from the seed list) rather than a hand-written module list, or it will drift again. ## Current state of the guarantee `make package-linux-full` **does** ship working `socket`, `ssl`, `asyncio` and `selectors` today — verified from the extracted artifact, resolving out of the package's own `lib/Python/Lib` — but only because the preset is ignored and the whole stdlib is copied. `tests/packaging/network_stdlib_packaged_test.py` now gates that outcome regardless of which resolution above is chosen. ## Related - #337 — numpy availability strategy (same "what does a distribution contain" question) - #380 — shipped artifacts gated by nothing

john added the

Refactoring & Cleanup

priority:tier2-foundation

Bugfix

labels

2026-07-27 03:39:04 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

john/McRogueFace#399

No description provided.

Rows
Columns