Headless Mode: Python-Controlled Simulation

Simplified scope: Enable synchronous, Python-driven simulation in headless mode only. No changes to windowed mode.

Problem

Currently, both headless and windowed modes run the same continuous game loop:

while (running) {
    scene->update();
    testTimers();           // Python timers fire here
    animations.update();
    scene->render();
    display();
    frameTime = clock.restart();
}

This causes issues for LLM agent orchestration (#156):

Screenshots are async: automation.screenshot() captures the previous frame, requiring timer-based coordination
No simulation control: Can't advance time precisely or pause between agent turns
Wasted frames: In headless mode, we render 60fps even when nothing changes

Solution

In headless mode, stop the automatic game loop. Python explicitly controls time advancement.

New API: `mcrfpy.step(dt=None)`

def step(dt=None):
    """
    Advance simulation time (headless mode only).
    
    Args:
        dt: Time to advance in seconds.
            - float: Advance by exactly this amount
            - None: Advance to next event (timer/animation completion)
    
    Returns:
        float: Actual time advanced
    
    In windowed mode: No-op, returns 0.0
    """

Modified: `automation.screenshot()`

In headless mode, screenshot becomes synchronous:

# Current behavior (both modes):
# - Captures previous frame's render buffer

# New headless behavior:
# 1. Clear render target
# 2. Render current scene
# 3. Capture to file
# 4. Return

Usage Example

# LLM Agent Orchestrator (headless mode)

def run_agent_turn(agent):
    # Set up perspective (no time passes)
    grid.perspective = agent.entity
    agent.entity.update_visibility()
    
    # Capture what agent sees (renders synchronously)
    automation.screenshot(f"agent_{agent.id}.png")
    
    # Query LLM (wall-clock time, simulation paused)
    action = query_llm(screenshot_path)
    
    # Execute action
    if action.type == "MOVE":
        agent.entity.path_to(*action.target)
        
        # Advance until movement completes
        while agent.entity.is_moving:
            step(None)  # Jump to next event

`step(None)` - Event-Driven Advancement

Instead of stepping by fixed dt, jump directly to the next event:

# Timer A at t+100ms, Animation completes at t+300ms

dt = step(None)  # Returns 0.1, fires Timer A
dt = step(None)  # Returns 0.2, animation completes

Events considered:

Timer firing
Animation completing
Entity path completion

Implementation

GameEngine Changes

// New mode flag
bool simulation_paused = false;  // True in headless after init

void GameEngine::run() {
    if (headless) {
        // Don't run loop - wait for Python to call step()
        // Just process initial scripts and return
        executeStartupScripts();
        
        // Enter Python-driven mode
        while (running && simulation_paused) {
            // Minimal event processing (Python can call step/screenshot)
            // This might be a simple sleep or Python GIL release
        }
    } else {
        // Windowed: existing continuous loop (unchanged)
        while (running) { /* ... */ }
    }
}

// New Python-callable function
float GameEngine::step(float dt) {
    if (!headless) return 0.0f;
    
    if (dt < 0) {
        // dt=None: find next event
        dt = findNextEventTime();
    }
    
    // Advance timers
    advanceTimers(dt);
    
    // Update animations
    AnimationManager::getInstance().update(dt);
    
    return dt;
}

Screenshot Changes (Headless Only)

PyObject* McRFPy_Automation::_screenshot(...) {
    if (engine->isHeadless()) {
        auto* rt = dynamic_cast<sf::RenderTexture*>(target);
        
        // Synchronous render
        rt->clear();
        engine->currentScene()->render();
        rt->display();
        
        // Capture fresh render
        rt->getTexture().copyToImage().saveToFile(filename);
        Py_RETURN_TRUE;
    }
    
    // Windowed: existing behavior (captures buffer)
    // ...
}

What This Does NOT Include

No threading: Single-threaded, no synchronization needed
No windowed changes: Windowed mode unchanged
No background simulation: LLM calls block in windowed mode (acceptable for demos)

Benefits

Synchronous screenshots: Set perspective → screenshot captures correct state
Efficient simulation: Skip to next event, no wasted frames
Deterministic: Python controls exact time advancement
Simple: No threading, no locks, no race conditions

Enables #156 (Turn-based LLM Agent Orchestration)
Part of #154 (Grounded Multi-Agent Testbed)

# Headless Mode: Python-Controlled Simulation **Simplified scope**: Enable synchronous, Python-driven simulation in headless mode only. No changes to windowed mode. ## Problem Currently, both headless and windowed modes run the same continuous game loop: ```cpp while (running) { scene->update(); testTimers(); // Python timers fire here animations.update(); scene->render(); display(); frameTime = clock.restart(); } ``` This causes issues for LLM agent orchestration (#156): 1. **Screenshots are async**: `automation.screenshot()` captures the *previous* frame, requiring timer-based coordination 2. **No simulation control**: Can't advance time precisely or pause between agent turns 3. **Wasted frames**: In headless mode, we render 60fps even when nothing changes ## Solution In headless mode, **stop the automatic game loop**. Python explicitly controls time advancement. ### New API: `mcrfpy.step(dt=None)` ```python def step(dt=None): """ Advance simulation time (headless mode only). Args: dt: Time to advance in seconds. - float: Advance by exactly this amount - None: Advance to next event (timer/animation completion) Returns: float: Actual time advanced In windowed mode: No-op, returns 0.0 """ ``` ### Modified: `automation.screenshot()` In headless mode, screenshot becomes **synchronous**: ```python # Current behavior (both modes): # - Captures previous frame's render buffer # New headless behavior: # 1. Clear render target # 2. Render current scene # 3. Capture to file # 4. Return ``` ## Usage Example ```python # LLM Agent Orchestrator (headless mode) def run_agent_turn(agent): # Set up perspective (no time passes) grid.perspective = agent.entity agent.entity.update_visibility() # Capture what agent sees (renders synchronously) automation.screenshot(f"agent_{agent.id}.png") # Query LLM (wall-clock time, simulation paused) action = query_llm(screenshot_path) # Execute action if action.type == "MOVE": agent.entity.path_to(*action.target) # Advance until movement completes while agent.entity.is_moving: step(None) # Jump to next event ``` ### `step(None)` - Event-Driven Advancement Instead of stepping by fixed dt, jump directly to the next event: ```python # Timer A at t+100ms, Animation completes at t+300ms dt = step(None) # Returns 0.1, fires Timer A dt = step(None) # Returns 0.2, animation completes ``` Events considered: - Timer firing - Animation completing - Entity path completion ## Implementation ### GameEngine Changes ```cpp // New mode flag bool simulation_paused = false; // True in headless after init void GameEngine::run() { if (headless) { // Don't run loop - wait for Python to call step() // Just process initial scripts and return executeStartupScripts(); // Enter Python-driven mode while (running && simulation_paused) { // Minimal event processing (Python can call step/screenshot) // This might be a simple sleep or Python GIL release } } else { // Windowed: existing continuous loop (unchanged) while (running) { /* ... */ } } } // New Python-callable function float GameEngine::step(float dt) { if (!headless) return 0.0f; if (dt < 0) { // dt=None: find next event dt = findNextEventTime(); } // Advance timers advanceTimers(dt); // Update animations AnimationManager::getInstance().update(dt); return dt; } ``` ### Screenshot Changes (Headless Only) ```cpp PyObject* McRFPy_Automation::_screenshot(...) { if (engine->isHeadless()) { auto* rt = dynamic_cast<sf::RenderTexture*>(target); // Synchronous render rt->clear(); engine->currentScene()->render(); rt->display(); // Capture fresh render rt->getTexture().copyToImage().saveToFile(filename); Py_RETURN_TRUE; } // Windowed: existing behavior (captures buffer) // ... } ``` ## What This Does NOT Include - **No threading**: Single-threaded, no synchronization needed - **No windowed changes**: Windowed mode unchanged - **No background simulation**: LLM calls block in windowed mode (acceptable for demos) ## Benefits 1. **Synchronous screenshots**: Set perspective → screenshot captures correct state 2. **Efficient simulation**: Skip to next event, no wasted frames 3. **Deterministic**: Python controls exact time advancement 4. **Simple**: No threading, no locks, no race conditions ## Related Issues - Enables #156 (Turn-based LLM Agent Orchestration) - Part of #154 (Grounded Multi-Agent Testbed)

john commented

2025-11-29 21:47:41 +00:00

Author

Owner

Event Handler Complexity
Event handlers run on the main thread and block rendering

From john/McRogueFace/wiki/Input-and-Events.-

Providing consistent state is the challenge, but allowing event handlers to more thoroughly perform the simulation without losing frames is the reward.

> Event Handler Complexity > Event handlers run on the main thread and block rendering From john/McRogueFace/wiki/Input-and-Events.- Providing consistent state is the challenge, but allowing event handlers to more thoroughly perform the simulation without losing frames is the reward.