The generate prompt only received the analyze_requirements summary,
which often compressed the file-by-file DoD into a brief paragraph.
The LLM then produced 1 file instead of the 5-6 specified.
Fix: added {original_prompt} variable to the generate prompt template
so the LLM sees both the architecture analysis AND the original DoD
with explicit file listings. Also strengthened the instruction to
"output ALL files mentioned in the Definition of Done."
Verified: sim10 (CSV Analyzer) now produces 6 files matching DoD
(loader.py, analyzer.py, reporter.py, cli.py, requirements.txt).
First simulation testing subdirectory output support. RAG app with
separate modules: document_loader, chunker, embeddings, vector_store,
retriever, generator — plus config.py and main.py CLI entry point.
All Python files pass syntax check. 9/10 DoD checks pass.
Also adds sim_test_commands.md with 9 new simulation definitions
(SIM9-SIM17) for manual terminal testing.
Problems fixed:
- Generation prompt was vague ("produce production-ready code") with no
output format specification. LLM had to guess the multi-file format.
- Validation was fake: `is_valid = "PASS" in text or len(code) > 10`
auto-passed any code longer than 10 chars.
- Single-file fallback used keyword guessing ("test" -> test_generated.py)
instead of parsing the filename from LLM output.
Changes:
- Rewrite analyze_prompt with structured sections (Files to Create,
Dependencies, Key Requirements, Architecture Notes)
- Rewrite generate_prompt with explicit multi-file output format rules
(**filename.ext** + fenced code block) and quality constraints
- Rewrite validate_prompt with specific review checklist
- Replace fake validation with real Python syntax check (compile()) +
LLM review where first line must be PASS or FAIL
- Add _extract_single_file() for robust single-file fallback that
parses the filename from the LLM output header
- Remove keyword-based path guessing (test/error/generated.py)
- Re-ran SIM1: now produces todo.py (matching DoD) instead of main.py
_extract_files_from_markdown now builds a set of fenced-block character
ranges and filters out any FILE_HEADER_RE matches that fall within them.
This prevents dependency lines like "requests>=2.31.0" from being
misidentified as file headers, which was causing requirements.txt to be
missed in sim4's output. Re-extracted sim4 with correct scraper.py and
requirements.txt.
Each simulation ran the full legacy lifecycle (init → tell → build → apply)
with real Anthropic Claude API calls:
1. sim1-todo-cli: Click-based todo app (add/list/delete)
2. sim2-bookstore-api: FastAPI CRUD with SQLite (5 code files)
3. sim3-websocket-chat: async websocket chat server
4. sim4-hn-scraper: HN top stories scraper with JSON output
5. sim5-flask-auth: Flask + Flask-Login authentication app
All 5 generated correct, production-quality Python code matching
the requested task specifications.
aditya
