Primary Concurrency Model: Threading Support with mcrfpy.lock() #219

New issue

Closed

opened 2026-01-16 19:42:31 +00:00 by john · 0 comments

john commented

2026-01-16 19:42:31 +00:00

Owner

Primary Concurrency Model: Threading Support

Enable background Python threads to run concurrently with the render loop, with a synchronization primitive for safe UI updates.

Motivation

Currently, all Python code (timers, input handlers, scene.update) runs synchronously on the render thread, blocking rendering. This prevents:

Web servers responding with game state
Package installs with progress updates
Waiting for slow API responses
Any long-running background work

Design

Core Principle

Single Python interpreter (existing)
Background Python threads for long-running work
GIL released during C++ render/display operations
mcrfpy.lock() context manager for synchronized frame-boundary updates

`mcrfpy.lock()` Semantics

import threading
import mcrfpy

def background_worker():
    while True:
        # This runs in background, GIL released during I/O
        data = requests.get("https://api.example.com/state")
        
        # Block until safe window between frames
        with mcrfpy.lock():
            # Executes sequentially with render loop
            frame.x = data.json()["x"]
            status.text = "Updated"

thread = threading.Thread(target=background_worker, daemon=True)
thread.start()

Render Loop Integration

void GameEngine::run() {
    while (running) {
        // === GIL HELD: Python callbacks ===
        testTimers();
        McRFPy_API::updatePythonScenes(frameTime);
        sUserInput();
        
        // === GIL RELEASED: Pure C++ rendering ===
        Py_BEGIN_ALLOW_THREADS
        currentScene()->render();
        window->display();
        Py_END_ALLOW_THREADS
        
        // === SAFE WINDOW: Background threads sync here ===
        if (frameLock.has_waiting()) {
            frameLock.open_window();
            // mcrfpy.lock() holders execute their blocks
            frameLock.close_window();
        }
    }
}

C++ FrameLock Implementation

class FrameLock {
    std::mutex mtx;
    std::condition_variable cv;
    bool safe_window = false;
    std::atomic<int> waiting{0};
    std::atomic<int> active{0};

public:
    void acquire() {
        waiting++;
        std::unique_lock<std::mutex> lock(mtx);
        Py_BEGIN_ALLOW_THREADS  // Release GIL while waiting
        cv.wait(lock, [this]{ return safe_window; });
        Py_END_ALLOW_THREADS
        waiting--;
        active++;
    }
    
    void release() {
        std::unique_lock<std::mutex> lock(mtx);
        active--;
        if (active == 0) cv.notify_all();
    }
    
    void open_window() {
        std::lock_guard<std::mutex> lock(mtx);
        safe_window = true;
        cv.notify_all();
    }
    
    void close_window() {
        std::unique_lock<std::mutex> lock(mtx);
        safe_window = false;
        cv.wait(lock, [this]{ return active == 0; });
    }
};

Implementation Tasks

Add FrameLock class to GameEngine
Modify render loop to release GIL during render/display
Add safe window open/close calls to render loop
Implement mcrfpy.lock() Python context manager
Consider atomic properties for simple numerics (x, y, w, h, opacity)
Document thread safety guarantees and user responsibilities
Add example: background web server exposing game state

Thread Safety Notes

User responsibilities:

Don't modify objects that animations are targeting (values will fight)
Collection modifications (adding/removing children) should use mcrfpy.lock()
Reads without lock may see intermediate states (acceptable for most use cases)

Engine guarantees:

Objects won't be destroyed while referenced (shared_ptr)
mcrfpy.lock() blocks execute sequentially with frame processing
Simple property writes won't cause segfaults (may corrupt single frame)

Example: Web Server

import threading
from http.server import HTTPServer, BaseHTTPRequestHandler
import json
import mcrfpy

class StateHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        # Read current state (eventual consistency ok)
        state = {"player_x": player.x, "player_y": player.y}
        
        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(json.dumps(state).encode())
    
    def do_POST(self):
        data = json.loads(self.rfile.read(int(self.headers['Content-Length'])))
        
        # Write requires lock
        with mcrfpy.lock():
            player.x = data['x']
            player.y = data['y']
        
        self.send_response(200)
        self.end_headers()

def run_server():
    HTTPServer(('localhost', 8080), StateHandler).serve_forever()

threading.Thread(target=run_server, daemon=True).start()

# Primary Concurrency Model: Threading Support Enable background Python threads to run concurrently with the render loop, with a synchronization primitive for safe UI updates. ## Motivation Currently, all Python code (timers, input handlers, scene.update) runs synchronously on the render thread, blocking rendering. This prevents: - Web servers responding with game state - Package installs with progress updates - Waiting for slow API responses - Any long-running background work ## Design ### Core Principle - Single Python interpreter (existing) - Background Python threads for long-running work - GIL released during C++ render/display operations - `mcrfpy.lock()` context manager for synchronized frame-boundary updates ### `mcrfpy.lock()` Semantics ```python import threading import mcrfpy def background_worker(): while True: # This runs in background, GIL released during I/O data = requests.get("https://api.example.com/state") # Block until safe window between frames with mcrfpy.lock(): # Executes sequentially with render loop frame.x = data.json()["x"] status.text = "Updated" thread = threading.Thread(target=background_worker, daemon=True) thread.start() ``` ### Render Loop Integration ```cpp void GameEngine::run() { while (running) { // === GIL HELD: Python callbacks === testTimers(); McRFPy_API::updatePythonScenes(frameTime); sUserInput(); // === GIL RELEASED: Pure C++ rendering === Py_BEGIN_ALLOW_THREADS currentScene()->render(); window->display(); Py_END_ALLOW_THREADS // === SAFE WINDOW: Background threads sync here === if (frameLock.has_waiting()) { frameLock.open_window(); // mcrfpy.lock() holders execute their blocks frameLock.close_window(); } } } ``` ### C++ FrameLock Implementation ```cpp class FrameLock { std::mutex mtx; std::condition_variable cv; bool safe_window = false; std::atomic<int> waiting{0}; std::atomic<int> active{0}; public: void acquire() { waiting++; std::unique_lock<std::mutex> lock(mtx); Py_BEGIN_ALLOW_THREADS // Release GIL while waiting cv.wait(lock, [this]{ return safe_window; }); Py_END_ALLOW_THREADS waiting--; active++; } void release() { std::unique_lock<std::mutex> lock(mtx); active--; if (active == 0) cv.notify_all(); } void open_window() { std::lock_guard<std::mutex> lock(mtx); safe_window = true; cv.notify_all(); } void close_window() { std::unique_lock<std::mutex> lock(mtx); safe_window = false; cv.wait(lock, [this]{ return active == 0; }); } }; ``` ## Implementation Tasks - [ ] Add `FrameLock` class to GameEngine - [ ] Modify render loop to release GIL during render/display - [ ] Add safe window open/close calls to render loop - [ ] Implement `mcrfpy.lock()` Python context manager - [ ] Consider atomic properties for simple numerics (x, y, w, h, opacity) - [ ] Document thread safety guarantees and user responsibilities - [ ] Add example: background web server exposing game state ## Thread Safety Notes **User responsibilities:** - Don't modify objects that animations are targeting (values will fight) - Collection modifications (adding/removing children) should use `mcrfpy.lock()` - Reads without lock may see intermediate states (acceptable for most use cases) **Engine guarantees:** - Objects won't be destroyed while referenced (shared_ptr) - `mcrfpy.lock()` blocks execute sequentially with frame processing - Simple property writes won't cause segfaults (may corrupt single frame) ## Example: Web Server ```python import threading from http.server import HTTPServer, BaseHTTPRequestHandler import json import mcrfpy class StateHandler(BaseHTTPRequestHandler): def do_GET(self): # Read current state (eventual consistency ok) state = {"player_x": player.x, "player_y": player.y} self.send_response(200) self.send_header('Content-Type', 'application/json') self.end_headers() self.wfile.write(json.dumps(state).encode()) def do_POST(self): data = json.loads(self.rfile.read(int(self.headers['Content-Length']))) # Write requires lock with mcrfpy.lock(): player.x = data['x'] player.y = data['y'] self.send_response(200) self.end_headers() def run_server(): HTTPServer(('localhost', 8080), StateHandler).serve_forever() threading.Thread(target=run_server, daemon=True).start() ```