[Bugfix] Embedded interpreter's filesystem encoding is ASCII: open() cannot read a UTF-8 file #378

New issue

Closed

opened 2026-07-14 22:17:55 +00:00 by john · 0 comments

john commented

2026-07-14 22:17:55 +00:00

Owner

Summary

init_python_with_config() — the init path main.cpp actually uses — does no
pre-initialization at all. It never calls Py_PreInitialize, so
PyPreConfig.utf8_mode is never enabled and the filesystem encoding falls back to
ASCII:

sys.getdefaultencoding():      utf-8
sys.getfilesystemencoding():   ascii     <-- wrong
locale.getpreferredencoding(): UTF-8

The correct block already exists in the other init path, init_python()
(McRFPy_API.cpp:969), which does set preconfig.utf8_mode = 1. Only the live path
is missing it.

Impact — scoped precisely

open() with no explicit encoding= defaults to the filesystem encoding, so it
cannot read a UTF-8 file:

open("notes.py").read()
# UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 4:
# ordinal not in range(128)

Any script that reads a data file, a save file, or its own source with a non-ASCII
character in it fails. This is a real, user-facing defect: in any normal CPython 3,
open() defaults to UTF-8.

What is NOT affected (verified, and this corrects the original text of this issue):
--exec on a source file containing non-ASCII characters works fine. The C++ side
reads the file and hands the bytes to Python, which parses them as UTF-8 per PEP 3120.
A ° in a comment does not break --exec.

The long-standing folklore that "--exec scripts must be ASCII-only" is therefore
wrong on the mechanism, but it was pointing at something real — this.

How it was found

Running the 130 documentation snippets from mcrogueface.github.io as a candidate test
corpus. 126 passed outright. Three failures (101_grid_camera_rotation.py,
105_scenes_dict.py, 124_sprite_rotation.py) turned out to be the harness tripping
over this bug — it used open(path).read() — not the snippets, which are fine. Between
them the offending characters are two ° and one •.

Fix

Pre-initialize with utf8_mode = 1 in init_python_with_config(), matching what
init_python() already does.

Acceptance

sys.getfilesystemencoding() reports utf-8
open() with no encoding= reads a UTF-8 file
Regression test asserting both

## Summary `init_python_with_config()` — the init path `main.cpp` actually uses — does **no pre-initialization at all**. It never calls `Py_PreInitialize`, so `PyPreConfig.utf8_mode` is never enabled and the filesystem encoding falls back to ASCII: ``` sys.getdefaultencoding(): utf-8 sys.getfilesystemencoding(): ascii <-- wrong locale.getpreferredencoding(): UTF-8 ``` The correct block already exists in the *other* init path, `init_python()` (`McRFPy_API.cpp:969`), which does set `preconfig.utf8_mode = 1`. Only the live path is missing it. ## Impact — scoped precisely `open()` with no explicit `encoding=` defaults to the filesystem encoding, so **it cannot read a UTF-8 file**: ```python open("notes.py").read() # UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 4: # ordinal not in range(128) ``` Any script that reads a data file, a save file, or its own source with a non-ASCII character in it fails. This is a real, user-facing defect: in any normal CPython 3, `open()` defaults to UTF-8. **What is NOT affected** (verified, and this corrects the original text of this issue): `--exec` on a source file containing non-ASCII characters **works fine**. The C++ side reads the file and hands the bytes to Python, which parses them as UTF-8 per PEP 3120. A `°` in a comment does not break `--exec`. The long-standing folklore that "`--exec` scripts must be ASCII-only" is therefore wrong on the mechanism, but it was pointing at something real — this. ## How it was found Running the 130 documentation snippets from mcrogueface.github.io as a candidate test corpus. 126 passed outright. Three failures (`101_grid_camera_rotation.py`, `105_scenes_dict.py`, `124_sprite_rotation.py`) turned out to be the *harness* tripping over this bug — it used `open(path).read()` — not the snippets, which are fine. Between them the offending characters are two `°` and one `•`. ## Fix Pre-initialize with `utf8_mode = 1` in `init_python_with_config()`, matching what `init_python()` already does. ## Acceptance - `sys.getfilesystemencoding()` reports `utf-8` - `open()` with no `encoding=` reads a UTF-8 file - Regression test asserting both

john added the

system:python-binding

Bugfix

priority:tier1-active

labels

2026-07-14 22:17:55 +00:00

john changed title from ~~[Bugfix] Embedded interpreter's filesystem encoding is ASCII: a non-ASCII character in a script kills --exec~~ to [Bugfix] Embedded interpreter's filesystem encoding is ASCII: open() cannot read a UTF-8 file

2026-07-14 22:19:11 +00:00

john referenced this issue from a commit

2026-07-15 04:32:56 +00:00

fix(engine): UTF-8 filesystem encoding, --run-forever in --help, honest step() docs

john closed this issue

2026-07-15 04:32:56 +00:00

john referenced this issue from a commit

2026-07-15 04:32:56 +00:00

test(snippets): the docs site's 130 code samples are now part of the test suite