McRogueFace/tools/generate_dynamic_docs.py
John McCardle 4e94291cfb docs: Complete Phase 7 documentation system with parser fixes and man pages
Fixed critical documentation generation bugs and added complete multi-format
output support. All documentation now generates cleanly from MCRF_* macros.

## Parser Fixes (tools/generate_dynamic_docs.py)

Fixed parse_docstring() function:
- Added "Raises:" section support (was missing entirely)
- Fixed function name duplication in headings
  - Was: `createSoundBuffercreateSoundBuffer(...)`
  - Now: `createSoundBuffer(filename: str) -> int`
- Proper section separation between Returns and Raises
- Handles MCRF_* macro format correctly

Changes:
- Rewrote parse_docstring() to parse by section markers
- Fixed markdown generation (lines 514-539)
- Fixed HTML generation (lines 385-413, 446-473)
- Added "raises" field to parsed output dict

## Man Page Generation

New files:
- tools/generate_man_page.sh - Pandoc wrapper for man page generation
- docs/mcrfpy.3 - Unix man page (section 3 for library functions)

Uses pandoc with metadata:
- Section 3 (library functions)
- Git version tag in footer
- Current date in header

## Master Orchestration Script

New file: tools/generate_all_docs.sh

Single command generates all documentation formats:
- HTML API reference (docs/api_reference_dynamic.html)
- Markdown API reference (docs/API_REFERENCE_DYNAMIC.md)
- Unix man page (docs/mcrfpy.3)
- Type stubs (stubs/mcrfpy.pyi via generate_stubs_v2.py)

Includes error checking (set -e) and helpful output messages.

## Documentation Updates (CLAUDE.md)

Updated "Regenerating Documentation" section:
- Documents new ./tools/generate_all_docs.sh master script
- Lists all output files with descriptions
- Notes pandoc as system requirement
- Clarifies generate_stubs_v2.py is preferred (has @overload support)

## Type Stub Decision

Assessed generate_stubs.py vs generate_stubs_v2.py:
- generate_stubs.py has critical bugs (missing commas in method signatures)
- generate_stubs_v2.py produces high-quality manually-maintained stubs
- Decision: Keep v2, use it in master script

## Files Modified

Modified:
- CLAUDE.md (25 lines changed)
- tools/generate_dynamic_docs.py (121 lines changed)
- docs/api_reference_dynamic.html (359 lines changed)

Created:
- tools/generate_all_docs.sh (28 lines)
- tools/generate_man_page.sh (12 lines)
- docs/mcrfpy.3 (1070 lines)
- stubs/mcrfpy.pyi (532 lines)
- stubs/mcrfpy/__init__.pyi (213 lines)
- stubs/mcrfpy/automation.pyi (24 lines)
- stubs/py.typed (0 bytes)

Total: 2159 insertions, 225 deletions

## Testing

Verified:
- Man page viewable with `man docs/mcrfpy.3`
- No function name duplication in docs/API_REFERENCE_DYNAMIC.md
- Raises sections properly separated from Returns
- Master script successfully generates all formats

## Related Issues

Addresses requirements from Phase 7 documentation issues.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-30 21:20:50 -04:00

619 lines
No EOL
22 KiB
Python

#!/usr/bin/env python3
"""
Dynamic documentation generator for McRogueFace.
Extracts all documentation directly from the compiled module using introspection.
"""
import os
import sys
import inspect
import datetime
import html
import re
from pathlib import Path
def transform_doc_links(docstring, format='html', base_url=''):
"""Transform MCRF_LINK patterns based on output format.
Detects pattern: "See also: TEXT (docs/path.md)"
Transforms to appropriate format for output type.
For HTML/web formats, properly escapes content before inserting HTML tags.
"""
if not docstring:
return docstring
link_pattern = r'See also: ([^(]+) \(([^)]+)\)'
def replace_link(match):
text, ref = match.group(1).strip(), match.group(2).strip()
if format == 'html':
# Convert docs/foo.md → foo.html and escape for safe HTML
href = html.escape(ref.replace('docs/', '').replace('.md', '.html'), quote=True)
text_escaped = html.escape(text)
return f'<p class="see-also">See also: <a href="{href}">{text_escaped}</a></p>'
elif format == 'web':
# Link to hosted docs and escape for safe HTML
web_path = ref.replace('docs/', '').replace('.md', '')
href = html.escape(f"{base_url}/{web_path}", quote=True)
text_escaped = html.escape(text)
return f'<p class="see-also">See also: <a href="{href}">{text_escaped}</a></p>'
elif format == 'markdown':
# Markdown link
return f'\n**See also:** [{text}]({ref})'
else: # 'python' or default
# Keep as plain text for Python docstrings
return match.group(0)
# For HTML formats, escape the entire docstring first, then process links
if format in ('html', 'web'):
# Split by the link pattern, escape non-link parts, then reassemble
parts = []
last_end = 0
for match in re.finditer(link_pattern, docstring):
# Escape the text before this match
if match.start() > last_end:
parts.append(html.escape(docstring[last_end:match.start()]))
# Process the link (replace_link handles escaping internally)
parts.append(replace_link(match))
last_end = match.end()
# Escape any remaining text after the last match
if last_end < len(docstring):
parts.append(html.escape(docstring[last_end:]))
return ''.join(parts)
else:
# For non-HTML formats, just do simple replacement
return re.sub(link_pattern, replace_link, docstring)
# Must be run with McRogueFace as interpreter
try:
import mcrfpy
except ImportError:
print("Error: This script must be run with McRogueFace as the interpreter")
print("Usage: ./build/mcrogueface --exec generate_dynamic_docs.py")
sys.exit(1)
def parse_docstring(docstring):
"""Parse a docstring to extract signature, description, args, returns, and raises."""
if not docstring:
return {"signature": "", "description": "", "args": [], "returns": "", "raises": "", "example": ""}
lines = docstring.strip().split('\n')
result = {
"signature": "",
"description": "",
"args": [],
"returns": "",
"raises": "",
"example": ""
}
# First line often contains the signature
if lines and '(' in lines[0] and ')' in lines[0]:
result["signature"] = lines[0].strip()
lines = lines[1:] if len(lines) > 1 else []
# Parse the rest
current_section = "description"
description_lines = []
returns_lines = []
raises_lines = []
example_lines = []
for line in lines:
line_lower = line.strip().lower()
# Detect section headers
if line_lower.startswith("args:") or line_lower.startswith("arguments:"):
current_section = "args"
continue
elif line_lower.startswith("returns:") or line_lower.startswith("return:"):
current_section = "returns"
# Capture any text on the same line as "Returns:"
content_after_colon = line[line.find(':')+1:].strip()
if content_after_colon:
returns_lines.append(content_after_colon)
continue
elif line_lower.startswith("raises:") or line_lower.startswith("raise:"):
current_section = "raises"
# Capture any text on the same line as "Raises:"
content_after_colon = line[line.find(':')+1:].strip()
if content_after_colon:
raises_lines.append(content_after_colon)
continue
elif line_lower.startswith("example:") or line_lower.startswith("examples:"):
current_section = "example"
continue
elif line_lower.startswith("note:"):
# Notes go into description
if description_lines:
description_lines.append("")
description_lines.append(line)
continue
# Skip blank lines unless we're in example section
if not line.strip() and current_section != "example":
continue
# Add content to appropriate section
if current_section == "description":
description_lines.append(line)
elif current_section == "args" and line.strip():
# Parse argument lines like " filename: Path to file"
match = re.match(r'\s+(\w+):\s*(.+)', line)
if match:
result["args"].append({
"name": match.group(1),
"description": match.group(2).strip()
})
elif current_section == "returns" and line.strip():
returns_lines.append(line.strip())
elif current_section == "raises" and line.strip():
raises_lines.append(line.strip())
elif current_section == "example":
example_lines.append(line)
result["description"] = '\n'.join(description_lines).strip()
result["returns"] = ' '.join(returns_lines).strip()
result["raises"] = ' '.join(raises_lines).strip()
result["example"] = '\n'.join(example_lines).strip()
return result
def get_all_functions():
"""Get all module-level functions."""
functions = {}
for name in dir(mcrfpy):
if name.startswith('_'):
continue
obj = getattr(mcrfpy, name)
if inspect.isbuiltin(obj) or inspect.isfunction(obj):
doc_info = parse_docstring(obj.__doc__)
functions[name] = {
"name": name,
"doc": obj.__doc__ or "",
"parsed": doc_info
}
return functions
def get_all_classes():
"""Get all classes and their methods/properties."""
classes = {}
for name in dir(mcrfpy):
if name.startswith('_'):
continue
obj = getattr(mcrfpy, name)
if inspect.isclass(obj):
class_info = {
"name": name,
"doc": obj.__doc__ or "",
"methods": {},
"properties": {},
"bases": [base.__name__ for base in obj.__bases__ if base.__name__ != 'object']
}
# Get methods and properties
for attr_name in dir(obj):
if attr_name.startswith('__') and attr_name != '__init__':
continue
try:
attr = getattr(obj, attr_name)
if callable(attr):
method_doc = attr.__doc__ or ""
class_info["methods"][attr_name] = {
"doc": method_doc,
"parsed": parse_docstring(method_doc)
}
elif isinstance(attr, property):
prop_doc = (attr.fget.__doc__ if attr.fget else "") or ""
class_info["properties"][attr_name] = {
"doc": prop_doc,
"readonly": attr.fset is None
}
except:
pass
classes[name] = class_info
return classes
def get_constants():
"""Get module constants."""
constants = {}
for name in dir(mcrfpy):
if name.startswith('_') or name[0].islower():
continue
obj = getattr(mcrfpy, name)
if not (inspect.isclass(obj) or callable(obj)):
constants[name] = {
"name": name,
"value": repr(obj) if not name.startswith('default_') else f"<{name}>",
"type": type(obj).__name__
}
return constants
def generate_html_docs():
"""Generate HTML documentation."""
functions = get_all_functions()
classes = get_all_classes()
constants = get_constants()
html_content = f"""<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>McRogueFace API Reference</title>
<style>
body {{
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
line-height: 1.6;
color: #333;
max-width: 1200px;
margin: 0 auto;
padding: 20px;
background-color: #f5f5f5;
}}
.container {{
background-color: white;
padding: 30px;
border-radius: 8px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
}}
h1, h2, h3, h4, h5 {{
color: #2c3e50;
}}
.toc {{
background-color: #f8f9fa;
padding: 20px;
border-radius: 4px;
margin-bottom: 30px;
}}
.toc ul {{
list-style-type: none;
padding-left: 20px;
}}
.toc > ul {{
padding-left: 0;
}}
.toc a {{
text-decoration: none;
color: #3498db;
}}
.toc a:hover {{
text-decoration: underline;
}}
.method-section {{
margin-bottom: 30px;
padding: 20px;
background-color: #f8f9fa;
border-radius: 4px;
border-left: 4px solid #3498db;
}}
.function-signature {{
font-family: 'Consolas', 'Monaco', monospace;
background-color: #e9ecef;
padding: 10px;
border-radius: 4px;
margin: 10px 0;
}}
.class-name {{
color: #e74c3c;
font-weight: bold;
}}
.method-name {{
color: #3498db;
font-family: 'Consolas', 'Monaco', monospace;
}}
.property-name {{
color: #27ae60;
font-family: 'Consolas', 'Monaco', monospace;
}}
.arg-name {{
color: #8b4513;
font-weight: bold;
}}
.arg-type {{
color: #666;
font-style: italic;
}}
code {{
background-color: #f4f4f4;
padding: 2px 5px;
border-radius: 3px;
font-family: 'Consolas', 'Monaco', monospace;
}}
pre {{
background-color: #f4f4f4;
padding: 15px;
border-radius: 5px;
overflow-x: auto;
}}
.deprecated {{
text-decoration: line-through;
opacity: 0.6;
}}
.note {{
background-color: #fff3cd;
border-left: 4px solid #ffc107;
padding: 10px;
margin: 10px 0;
}}
.returns {{
color: #28a745;
font-weight: bold;
}}
</style>
</head>
<body>
<div class="container">
<h1>McRogueFace API Reference</h1>
<p><em>Generated on {datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')}</em></p>
<p><em>This documentation was dynamically generated from the compiled module.</em></p>
<div class="toc">
<h2>Table of Contents</h2>
<ul>
<li><a href="#functions">Functions</a></li>
<li><a href="#classes">Classes</a>
<ul>
"""
# Add classes to TOC
for class_name in sorted(classes.keys()):
html_content += f' <li><a href="#{class_name}">{class_name}</a></li>\n'
html_content += """ </ul>
</li>
<li><a href="#constants">Constants</a></li>
</ul>
</div>
<h2 id="functions">Functions</h2>
"""
# Generate function documentation
for func_name in sorted(functions.keys()):
func_info = functions[func_name]
parsed = func_info["parsed"]
# Use signature if available (already includes name), otherwise use just name
heading = parsed['signature'] if parsed['signature'] else f"{func_name}(...)"
html_content += f"""
<div class="method-section">
<h3><code class="function-signature">{heading}</code></h3>
"""
if parsed['description']:
description = transform_doc_links(parsed['description'], format='html')
html_content += f" <p>{description}</p>\n"
if parsed['args']:
html_content += " <h4>Arguments:</h4>\n <ul>\n"
for arg in parsed['args']:
html_content += f" <li><span class='arg-name'>{arg['name']}</span>: {html.escape(arg['description'])}</li>\n"
html_content += " </ul>\n"
if parsed['returns']:
html_content += f" <p><span class='returns'>Returns:</span> {html.escape(parsed['returns'])}</p>\n"
if parsed['raises']:
html_content += f" <p><span class='raises'>Raises:</span> {html.escape(parsed['raises'])}</p>\n"
if parsed['example']:
html_content += f" <h4>Example:</h4>\n <pre><code>{html.escape(parsed['example'])}</code></pre>\n"
html_content += " </div>\n"
# Generate class documentation
html_content += "\n <h2 id='classes'>Classes</h2>\n"
for class_name in sorted(classes.keys()):
class_info = classes[class_name]
html_content += f"""
<div class="method-section">
<h3 id="{class_name}"><span class="class-name">{class_name}</span></h3>
"""
if class_info['bases']:
html_content += f" <p><em>Inherits from: {', '.join(class_info['bases'])}</em></p>\n"
if class_info['doc']:
html_content += f" <p>{html.escape(class_info['doc'])}</p>\n"
# Properties
if class_info['properties']:
html_content += " <h4>Properties:</h4>\n <ul>\n"
for prop_name, prop_info in sorted(class_info['properties'].items()):
readonly = " (read-only)" if prop_info['readonly'] else ""
html_content += f" <li><span class='property-name'>{prop_name}</span>{readonly}"
if prop_info['doc']:
html_content += f": {html.escape(prop_info['doc'])}"
html_content += "</li>\n"
html_content += " </ul>\n"
# Methods
if class_info['methods']:
html_content += " <h4>Methods:</h4>\n"
for method_name, method_info in sorted(class_info['methods'].items()):
if method_name == '__init__':
continue
parsed = method_info['parsed']
# Use signature if available (already includes name), otherwise use just name
heading = parsed['signature'] if parsed['signature'] else f"{method_name}(...)"
html_content += f"""
<div style="margin-left: 20px; margin-bottom: 15px;">
<h5><code class="method-name">{heading}</code></h5>
"""
if parsed['description']:
description = transform_doc_links(parsed['description'], format='html')
html_content += f" <p>{description}</p>\n"
if parsed['args']:
html_content += " <div style='margin-left: 20px;'>\n"
for arg in parsed['args']:
html_content += f" <div><span class='arg-name'>{arg['name']}</span>: {html.escape(arg['description'])}</div>\n"
html_content += " </div>\n"
if parsed['returns']:
html_content += f" <p style='margin-left: 20px;'><span class='returns'>Returns:</span> {html.escape(parsed['returns'])}</p>\n"
if parsed['raises']:
html_content += f" <p style='margin-left: 20px;'><span class='raises'>Raises:</span> {html.escape(parsed['raises'])}</p>\n"
html_content += " </div>\n"
html_content += " </div>\n"
# Constants
html_content += "\n <h2 id='constants'>Constants</h2>\n <ul>\n"
for const_name, const_info in sorted(constants.items()):
html_content += f" <li><code>{const_name}</code> ({const_info['type']}): {const_info['value']}</li>\n"
html_content += " </ul>\n"
html_content += """
</div>
</body>
</html>
"""
# Write the file
output_path = Path("docs/api_reference_dynamic.html")
output_path.parent.mkdir(exist_ok=True)
output_path.write_text(html_content)
print(f"Generated {output_path}")
print(f"Found {len(functions)} functions, {len(classes)} classes, {len(constants)} constants")
def generate_markdown_docs():
"""Generate Markdown documentation."""
functions = get_all_functions()
classes = get_all_classes()
constants = get_constants()
md_content = f"""# McRogueFace API Reference
*Generated on {datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')}*
*This documentation was dynamically generated from the compiled module.*
## Table of Contents
- [Functions](#functions)
- [Classes](#classes)
"""
# Add classes to TOC
for class_name in sorted(classes.keys()):
md_content += f" - [{class_name}](#{class_name.lower()})\n"
md_content += "- [Constants](#constants)\n\n"
# Functions
md_content += "## Functions\n\n"
for func_name in sorted(functions.keys()):
func_info = functions[func_name]
parsed = func_info["parsed"]
# Use signature if available (already includes name), otherwise use just name
heading = parsed['signature'] if parsed['signature'] else f"{func_name}(...)"
md_content += f"### `{heading}`\n\n"
if parsed['description']:
description = transform_doc_links(parsed['description'], format='markdown')
md_content += f"{description}\n\n"
if parsed['args']:
md_content += "**Arguments:**\n"
for arg in parsed['args']:
md_content += f"- `{arg['name']}`: {arg['description']}\n"
md_content += "\n"
if parsed['returns']:
md_content += f"**Returns:** {parsed['returns']}\n\n"
if parsed['raises']:
md_content += f"**Raises:** {parsed['raises']}\n\n"
if parsed['example']:
md_content += f"**Example:**\n```python\n{parsed['example']}\n```\n\n"
# Classes
md_content += "## Classes\n\n"
for class_name in sorted(classes.keys()):
class_info = classes[class_name]
md_content += f"### {class_name}\n\n"
if class_info['bases']:
md_content += f"*Inherits from: {', '.join(class_info['bases'])}*\n\n"
if class_info['doc']:
md_content += f"{class_info['doc']}\n\n"
# Properties
if class_info['properties']:
md_content += "**Properties:**\n"
for prop_name, prop_info in sorted(class_info['properties'].items()):
readonly = " *(read-only)*" if prop_info['readonly'] else ""
md_content += f"- `{prop_name}`{readonly}"
if prop_info['doc']:
md_content += f": {prop_info['doc']}"
md_content += "\n"
md_content += "\n"
# Methods
if class_info['methods']:
md_content += "**Methods:**\n\n"
for method_name, method_info in sorted(class_info['methods'].items()):
if method_name == '__init__':
continue
parsed = method_info['parsed']
# Use signature if available (already includes name), otherwise use just name
heading = parsed['signature'] if parsed['signature'] else f"{method_name}(...)"
md_content += f"#### `{heading}`\n\n"
if parsed['description']:
description = transform_doc_links(parsed['description'], format='markdown')
md_content += f"{description}\n\n"
if parsed['args']:
md_content += "**Arguments:**\n"
for arg in parsed['args']:
md_content += f"- `{arg['name']}`: {arg['description']}\n"
md_content += "\n"
if parsed['returns']:
md_content += f"**Returns:** {parsed['returns']}\n\n"
if parsed['raises']:
md_content += f"**Raises:** {parsed['raises']}\n\n"
# Constants
md_content += "## Constants\n\n"
for const_name, const_info in sorted(constants.items()):
md_content += f"- `{const_name}` ({const_info['type']}): {const_info['value']}\n"
# Write the file
output_path = Path("docs/API_REFERENCE_DYNAMIC.md")
output_path.parent.mkdir(exist_ok=True)
output_path.write_text(md_content)
print(f"Generated {output_path}")
if __name__ == "__main__":
print("Generating dynamic documentation from mcrfpy module...")
generate_html_docs()
generate_markdown_docs()
print("Documentation generation complete!")