john/McRogueFace

Fork 0

Secondary Concurrency Model: Subinterpreter Support #220

New issue

Open

opened 2026-01-16 19:42:33 +00:00 by john · 0 comments

john commented

2026-01-16 19:42:33 +00:00

Owner

Secondary Concurrency Model: Subinterpreters

Enable isolated Python subinterpreters for advanced concurrency scenarios with true parallelism (separate GILs in Python 3.12+).

Motivation

While the primary threading model (see related issue) handles most use cases, subinterpreters provide:

True parallelism (each subinterpreter has its own GIL)
Isolation for untrusted or crash-prone code
Clean separation of concerns (main game vs monitoring/tools)

Design

Command Line Interface

# Main script runs in primary interpreter
# Additional file runs in separate subinterpreter
./mcrogueface game.py --exec-subinterpreter server.py

The subinterpreter starts automatically but user code manages its own threading/event loop.

`mcrfpy.Queue` - Cross-Interpreter Communication

Thread-safe, subinterpreter-safe queue for bytes. Named queues are accessible from any interpreter.

# In main interpreter (game.py)
import mcrfpy

request_queue = mcrfpy.Queue('requests')
response_queue = mcrfpy.Queue('responses')

def process_requests(timer, runtime):
    """Timer callback in main interpreter"""
    while not request_queue.empty():
        data = request_queue.pop()  # bytes
        request = json.loads(data)
        
        # Full access to derived types here
        player.health = request['new_health']
        
        response_queue.push(json.dumps({"status": "ok"}).encode())

# In subinterpreter (server.py)
import mcrfpy
from aiohttp import web

request_queue = mcrfpy.Queue('requests')  # Same queue, different interpreter
response_queue = mcrfpy.Queue('responses')

async def handle_update(request):
    data = await request.read()
    request_queue.push(data)
    
    # Wait for main interpreter to process
    while response_queue.empty():
        await asyncio.sleep(0.01)
    
    response = response_queue.pop()
    return web.Response(body=response, content_type='application/json')

# User manages their own event loop
app = web.Application()
app.router.add_post('/update', handle_update)
web.run_app(app, port=8080)

`mcrfpy.lock()` Across Subinterpreters

Same primitive as primary model, but coordinates across all interpreters:

# In subinterpreter
import mcrfpy

scene = mcrfpy.Scene.find("game")  # Wraps existing C++ scene

with mcrfpy.lock():
    # Sequential access to C++ objects
    # NOTE: Only base mcrfpy types accessible, not user-derived classes
    for entity in scene.entities:
        print(f"Entity at ({entity.x}, {entity.y})")

Implementation

C++ Queue

class InterpreterQueue {
    std::string name;
    std::queue<std::vector<uint8_t>> data;
    mutable std::mutex mtx;
    std::condition_variable cv;
    
public:
    void push(const std::vector<uint8_t>& bytes) {
        std::lock_guard<std::mutex> lock(mtx);
        data.push(bytes);
        cv.notify_one();
    }
    
    std::vector<uint8_t> pop(bool block = false) {
        std::unique_lock<std::mutex> lock(mtx);
        if (block) {
            cv.wait(lock, [this]{ return !data.empty(); });
        }
        if (data.empty()) return {};
        auto result = std::move(data.front());
        data.pop();
        return result;
    }
    
    bool empty() const {
        std::lock_guard<std::mutex> lock(mtx);
        return data.empty();
    }
};

// Global registry
std::map<std::string, std::shared_ptr<InterpreterQueue>> queue_registry;
std::mutex registry_mutex;

Subinterpreter Lifecycle

void spawn_subinterpreter(const std::string& script_path) {
    std::thread([script_path]() {
        // Create subinterpreter with own GIL (Python 3.12+)
        PyThreadState* tstate = Py_NewInterpreterFromConfig(...);
        PyThreadState_Swap(tstate);
        
        // Import mcrfpy module (re-initialized for this interpreter)
        // Note: mcrfpy types wrap same C++ shared_ptrs
        
        // Run user script
        PyRun_SimpleFile(script_path);
        
        // Cleanup
        Py_EndInterpreter(tstate);
    }).detach();
}

Limitations

Derived Types Cannot Cross Interpreter Boundaries

# Main interpreter
class Player(mcrfpy.Entity):
    def __init__(self):
        super().__init__()
        self.health = 100  # Python attribute

# Subinterpreter
entity = scene.entities[0]  # This is a Player in main interpreter
type(entity)                # Returns mcrfpy.Entity, NOT Player
entity.health               # AttributeError - Python attrs not accessible

Workaround: Use mcrfpy.Queue for data that requires derived type access. Main interpreter processes requests with full type access.

Separate Module State

Each subinterpreter imports mcrfpy separately:

Same C++ objects (shared_ptr)
Different Python wrapper objects
Different Python object cache per interpreter

Implementation Tasks

Implement mcrfpy.Queue with named registry
Add --exec-subinterpreter command line argument
Subinterpreter spawning with Py_NewInterpreterFromConfig
Per-interpreter Python object cache
Extend mcrfpy.lock() for cross-interpreter coordination
Add mcrfpy.Scene.find(name) for accessing existing scenes
Document limitations (derived types, separate module state)
Add example: monitoring web server in subinterpreter

Dependencies

Primary concurrency model (mcrfpy.lock()) - must be implemented first
Python 3.12+ for per-interpreter GIL (optional but recommended)

Future Considerations

Hot-reload subinterpreter scripts without restarting main
Multiple subinterpreters with different roles
Shared memory regions beyond Queue (numpy arrays, etc.)

# Secondary Concurrency Model: Subinterpreters Enable isolated Python subinterpreters for advanced concurrency scenarios with true parallelism (separate GILs in Python 3.12+). ## Motivation While the primary threading model (see related issue) handles most use cases, subinterpreters provide: - True parallelism (each subinterpreter has its own GIL) - Isolation for untrusted or crash-prone code - Clean separation of concerns (main game vs monitoring/tools) ## Design ### Command Line Interface ```bash # Main script runs in primary interpreter # Additional file runs in separate subinterpreter ./mcrogueface game.py --exec-subinterpreter server.py ``` The subinterpreter starts automatically but user code manages its own threading/event loop. ### `mcrfpy.Queue` - Cross-Interpreter Communication Thread-safe, subinterpreter-safe queue for bytes. Named queues are accessible from any interpreter. ```python # In main interpreter (game.py) import mcrfpy request_queue = mcrfpy.Queue('requests') response_queue = mcrfpy.Queue('responses') def process_requests(timer, runtime): """Timer callback in main interpreter""" while not request_queue.empty(): data = request_queue.pop() # bytes request = json.loads(data) # Full access to derived types here player.health = request['new_health'] response_queue.push(json.dumps({"status": "ok"}).encode()) ``` ```python # In subinterpreter (server.py) import mcrfpy from aiohttp import web request_queue = mcrfpy.Queue('requests') # Same queue, different interpreter response_queue = mcrfpy.Queue('responses') async def handle_update(request): data = await request.read() request_queue.push(data) # Wait for main interpreter to process while response_queue.empty(): await asyncio.sleep(0.01) response = response_queue.pop() return web.Response(body=response, content_type='application/json') # User manages their own event loop app = web.Application() app.router.add_post('/update', handle_update) web.run_app(app, port=8080) ``` ### `mcrfpy.lock()` Across Subinterpreters Same primitive as primary model, but coordinates across all interpreters: ```python # In subinterpreter import mcrfpy scene = mcrfpy.Scene.find("game") # Wraps existing C++ scene with mcrfpy.lock(): # Sequential access to C++ objects # NOTE: Only base mcrfpy types accessible, not user-derived classes for entity in scene.entities: print(f"Entity at ({entity.x}, {entity.y})") ``` ## Implementation ### C++ Queue ```cpp class InterpreterQueue { std::string name; std::queue<std::vector<uint8_t>> data; mutable std::mutex mtx; std::condition_variable cv; public: void push(const std::vector<uint8_t>& bytes) { std::lock_guard<std::mutex> lock(mtx); data.push(bytes); cv.notify_one(); } std::vector<uint8_t> pop(bool block = false) { std::unique_lock<std::mutex> lock(mtx); if (block) { cv.wait(lock, [this]{ return !data.empty(); }); } if (data.empty()) return {}; auto result = std::move(data.front()); data.pop(); return result; } bool empty() const { std::lock_guard<std::mutex> lock(mtx); return data.empty(); } }; // Global registry std::map<std::string, std::shared_ptr<InterpreterQueue>> queue_registry; std::mutex registry_mutex; ``` ### Subinterpreter Lifecycle ```cpp void spawn_subinterpreter(const std::string& script_path) { std::thread([script_path]() { // Create subinterpreter with own GIL (Python 3.12+) PyThreadState* tstate = Py_NewInterpreterFromConfig(...); PyThreadState_Swap(tstate); // Import mcrfpy module (re-initialized for this interpreter) // Note: mcrfpy types wrap same C++ shared_ptrs // Run user script PyRun_SimpleFile(script_path); // Cleanup Py_EndInterpreter(tstate); }).detach(); } ``` ## Limitations ### Derived Types Cannot Cross Interpreter Boundaries ```python # Main interpreter class Player(mcrfpy.Entity): def __init__(self): super().__init__() self.health = 100 # Python attribute # Subinterpreter entity = scene.entities[0] # This is a Player in main interpreter type(entity) # Returns mcrfpy.Entity, NOT Player entity.health # AttributeError - Python attrs not accessible ``` **Workaround**: Use `mcrfpy.Queue` for data that requires derived type access. Main interpreter processes requests with full type access. ### Separate Module State Each subinterpreter imports `mcrfpy` separately: - Same C++ objects (shared_ptr) - Different Python wrapper objects - Different Python object cache per interpreter ## Implementation Tasks - [ ] Implement `mcrfpy.Queue` with named registry - [ ] Add `--exec-subinterpreter` command line argument - [ ] Subinterpreter spawning with Py_NewInterpreterFromConfig - [ ] Per-interpreter Python object cache - [ ] Extend `mcrfpy.lock()` for cross-interpreter coordination - [ ] Add `mcrfpy.Scene.find(name)` for accessing existing scenes - [ ] Document limitations (derived types, separate module state) - [ ] Add example: monitoring web server in subinterpreter ## Dependencies - Primary concurrency model (mcrfpy.lock()) - must be implemented first - Python 3.12+ for per-interpreter GIL (optional but recommended) ## Future Considerations - Hot-reload subinterpreter scripts without restarting main - Multiple subinterpreters with different roles - Shared memory regions beyond Queue (numpy arrays, etc.)

john added a new dependency

2026-01-16 19:42:41 +00:00

#219 Primary Concurrency Model: Threading Support with mcrfpy.lock()

john added the

priority:tier4-deferred

label

2026-02-07 18:57:05 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Depends on