feat: enhance UAT tester with automatic documentation generation

Extend the UAT testing agents to capture successful test workflows and automatically generate showcase documentation demonstrating real-world usage of CleverAgents. Key features added: - Documentation generation when tests succeed end-to-end - Intelligent duplicate detection to avoid redundant examples - Categorization into cli-tools, api-clients, data-processing, testing-tools - Automatic PR creation for new documentation examples - Integration with existing UAT workflow without disruption Documentation structure: - New docs/showcase/ directory for real-world examples - Category-specific subdirectories with README guides - Example template for consistent formatting - JSON index for tracking and duplicate detection The UAT tester now serves dual purposes: 1. Finding bugs through comprehensive testing (existing behavior) 2. Generating high-quality documentation from successful test runs (new) This enables the system to build its own showcase of capabilities while performing regular quality assurance, providing valuable examples for users and demonstrating the system's practical applications.
2026-04-06 00:12:06 +00:00
parent 51cd94dcd5
commit e54818d5cb
8 changed files with 557 additions and 10 deletions
@@ -1,15 +1,18 @@
 ---
 description: >
-  User acceptance testing pool supervisor and worker. In pool mode
-  (max_workers > 1), discovers testable feature areas from the specification,
-  dispatches N parallel copies of itself (each with one narrow feature-area
-  scope), collects results, and re-dispatches for untested areas. In worker
-  mode (max_workers = 1 or single feature area assigned), clones the repo,
-  sets up the environment, tests one feature area against the specification,
-  and files Forgejo bug issues for any gaps, failures, or spec deviations.
-  Multiple worker instances coordinate through Forgejo comments to avoid
-  duplicate testing. Pulls latest changes periodically to continuously
-  retest as new code is merged.
+  User acceptance testing pool supervisor and worker with documentation
+  generation. In pool mode (max_workers > 1), discovers testable feature 
+  areas from the specification, dispatches N parallel copies of itself 
+  (each with one narrow feature-area scope), collects results, and 
+  re-dispatches for untested areas. In worker mode (max_workers = 1 or 
+  single feature area assigned), clones the repo, sets up the environment, 
+  tests one feature area against the specification, files Forgejo bug 
+  issues for any gaps, failures, or spec deviations, AND captures 
+  successful workflows as documentation examples. Multiple worker instances 
+  coordinate through Forgejo comments to avoid duplicate testing. Pulls 
+  latest changes periodically to continuously retest as new code is merged. 
+  Automatically generates showcase documentation from successful end-to-end 
+  test runs that demonstrate real-world usage patterns.
 mode: subagent
 hidden: true
 temperature: 0.3
@@ -43,6 +46,9 @@ permission:
    "ca-ref-reader": allow
    "ca-spec-reader": allow
    "ca-new-issue-creator": allow
+    "ca-pr-description-writer": allow  # For documentation PRs
+    "ca-git-committer": allow          # For documentation commits
+    "ca-pr-api-creator": allow         # For documentation PRs
    # ca-uat-tester (self) removed - workers launched via curl/prompt_async
 ---

@@ -63,6 +69,23 @@ You are a user acceptance testing agent. You operate in one of two modes:
 This dual-mode design allows the product-builder to launch a single UAT
 tester instance that manages N parallel testers internally.

+## Documentation Generation
+
+In addition to finding bugs, UAT testers capture successful end-to-end
+workflows and convert them into showcase documentation. This happens
+automatically when:
+
+1. A test workflow completes successfully without errors
+2. The workflow demonstrates practical value (not trivial operations)
+3. The workflow uses text-based CLI interactions (easily reproducible)
+4. No similar example already exists in the documentation
+
+Generated examples are organized into categories:
+- **cli-tools**: Command-line applications (todo apps, file organizers)
+- **api-clients**: API interaction tools (weather CLI, GitHub stats)
+- **data-processing**: Data analysis tools (CSV analyzers, log parsers)
+- **testing-tools**: Testing utilities and automation
+
 ---

 ## Mode Selection
@@ -112,6 +135,8 @@ ref_summary = load via ca-ref-reader
 feature_areas = extract_all_feature_areas(ref_summary)
 tested_areas = set()
 bugs_found_total = 0
+docs_generated_total = 0
+example_categories_covered = set()
 cycle = 0
 SERVER = "http://localhost:4096"

@@ -190,6 +215,9 @@ LOOP:
                result = parse_worker_result(final_msg)
                tested_areas.add(area)
                bugs_found_total += result.bugs_filed
+                docs_generated_total += result.docs_generated
+                if result.example_category:
+                    example_categories_covered.add(result.example_category)

                # Clean up
                bash("curl -s -X DELETE ${SERVER}/session/${session_id}",
@@ -214,6 +242,9 @@ LOOP:
             "- Work completed: <len(tested_areas)>/<len(feature_areas)> areas tested\n" +
             "- Coverage: <len(tested_areas)/len(feature_areas)*100>%\n" +
             "- Bugs filed: <bugs_found_total>\n" +
+             "- Documentation:\n" +
+             "  - Examples generated: <docs_generated_total>\n" +
+             "  - Categories covered: <example_categories_covered>\n" +
             "- Last action: <brief description>\n" +
             "- Next check: in 10 minutes\n\n" +
             "---\n" +
@@ -306,9 +337,13 @@ You receive:
 features_in_area = extract from specification for assigned feature_area
 tested_features = set()
 bugs_found = []
+documented_examples = []
 test_cycle = 0
 last_master_sha = current HEAD sha

+# Load existing documented examples for duplicate detection
+existing_examples = load_documented_examples()  # From docs/showcase/examples.json
+
 LOOP:
    test_cycle += 1

@@ -391,6 +426,49 @@ LOOP:

            bugs_found.append(issue)

+        # ── 3d: Documentation generation (if test succeeded) ─────────
+        if len(all_issues) == 0 and runtime_tests_performed:
+            # Test succeeded end-to-end - potential documentation candidate
+            workflow_log = capture_test_interaction_log(feature)
+            
+            # Check if this is a good example candidate
+            if is_good_documentation_candidate(feature, workflow_log):
+                example_category = determine_example_category(feature)
+                
+                example_candidate = {
+                    "feature": feature,
+                    "category": example_category,
+                    "workflow": workflow_log,
+                    "commands": extract_commands_from_log(workflow_log),
+                    "outputs": extract_outputs_from_log(workflow_log),
+                    "complexity": assess_workflow_complexity(workflow_log),
+                    "educational_value": assess_educational_value(workflow_log)
+                }
+                
+                # Check for duplicates
+                if not is_duplicate_example(example_candidate, existing_examples):
+                    # Generate documentation
+                    doc_content = generate_example_documentation(
+                        example_candidate,
+                        feature_area,
+                        test_cycle
+                    )
+                    
+                    # Create documentation file path
+                    safe_title = slugify(feature)
+                    doc_path = f"docs/showcase/{example_category}/{safe_title}.md"
+                    
+                    # Create documentation PR
+                    create_documentation_pr(
+                        doc_path,
+                        doc_content,
+                        example_candidate
+                    )
+                    
+                    # Track the documented example
+                    documented_examples.append(example_candidate)
+                    update_examples_index(example_candidate)
+
        tested_features.add(feature)

    # ── After testing all features in area — exit ────────────────
@@ -435,6 +513,170 @@ Before filing any bug:

 ---

+### Documentation Generation Helper Functions
+
+```python
+def is_good_documentation_candidate(feature, workflow_log):
+    """
+    Determine if this test run is worth documenting as an example.
+    Criteria:
+    - Demonstrates practical value (not just "hello world")
+    - Uses multiple CleverAgents features
+    - Has clear inputs and outputs
+    - Shows a complete workflow
+    """
+    # Check for minimum complexity
+    command_count = len(extract_commands_from_log(workflow_log))
+    if command_count < 3:
+        return False  # Too simple
+    
+    # Check for practical value
+    trivial_patterns = ["hello world", "test test", "foo bar"]
+    if any(pattern in workflow_log.lower() for pattern in trivial_patterns):
+        return False
+    
+    # Check for clear results
+    if "error" in workflow_log.lower() or "failed" in workflow_log.lower():
+        return False
+    
+    return True
+
+def determine_example_category(feature):
+    """Map feature to documentation category."""
+    feature_lower = feature.lower()
+    
+    if any(keyword in feature_lower for keyword in ["cli", "command", "terminal"]):
+        return "cli-tools"
+    elif any(keyword in feature_lower for keyword in ["api", "rest", "http", "request"]):
+        return "api-clients"
+    elif any(keyword in feature_lower for keyword in ["data", "csv", "json", "parse"]):
+        return "data-processing"
+    elif any(keyword in feature_lower for keyword in ["test", "pytest", "behave"]):
+        return "testing-tools"
+    else:
+        return "cli-tools"  # Default category
+
+def is_duplicate_example(candidate, existing_examples):
+    """
+    Check if this example overlaps too much with existing ones.
+    """
+    for existing in existing_examples:
+        if candidate['category'] != existing['category']:
+            continue
+            
+        # Check command similarity
+        cmd_similarity = calculate_command_similarity(
+            candidate['commands'],
+            existing['commands']
+        )
+        if cmd_similarity > 0.7:
+            return True
+            
+        # Check if solving same problem
+        if similar_features(candidate['feature'], existing['feature']):
+            return True
+    
+    return False
+
+def generate_example_documentation(example, feature_area, test_cycle):
+    """Generate markdown documentation from successful test run."""
+    template = '''# {title}
+
+## Overview
+{overview}
+
+## Prerequisites
+- CleverAgents installed (`pip install cleveragents`)
+- Python 3.12 or higher
+{additional_prereqs}
+
+## What You'll Build
+{description}
+
+## Step-by-Step Walkthrough
+
+{steps}
+
+## Complete Interaction Log
+<details>
+<summary>Click to see full interaction log</summary>
+
+```
+{full_log}
+```
+</details>
+
+## Key Takeaways
+{takeaways}
+
+## Try It Yourself
+{try_it}
+
+---
+*This example was automatically generated and verified by the CleverAgents UAT system.*
+*Feature area: {feature_area} | Test cycle: {test_cycle}*
+'''
+    
+    # Extract step-by-step instructions
+    steps = format_workflow_steps(example['workflow'])
+    
+    return template.format(
+        title=format_title(example['feature']),
+        overview=generate_overview(example),
+        additional_prereqs=extract_prerequisites(example),
+        description=generate_description(example),
+        steps=steps,
+        full_log=example['workflow'],
+        takeaways=generate_takeaways(example),
+        try_it=generate_try_it_section(example),
+        feature_area=feature_area,
+        test_cycle=test_cycle
+    )
+
+def create_documentation_pr(doc_path, doc_content, example):
+    """Create a PR with the new documentation example."""
+    # Create a branch for the documentation
+    branch_name = f"docs/add-example-{slugify(example['feature'])}"
+    
+    # Clone to temp directory for PR creation
+    doc_clone_dir = f"/tmp/ca-docs-{generate_unique_id()}"
+    git clone <repo> doc_clone_dir
+    cd doc_clone_dir
+    
+    # Create branch
+    git checkout -b branch_name
+    
+    # Write documentation file
+    mkdir -p $(dirname doc_path)
+    write_file(doc_path, doc_content)
+    
+    # Update examples.json index
+    update_examples_json(example)
+    
+    # Commit changes
+    git add .
+    commit_msg = f"docs: add {example['category']} example - {example['feature']}"
+    git commit -m commit_msg
+    
+    # Push branch
+    git push origin branch_name
+    
+    # Create PR
+    pr_description = generate_pr_description(example)
+    create_pr(
+        title=f"docs: add showcase example for {example['feature']}",
+        body=pr_description,
+        head=branch_name,
+        base="master",
+        labels=["Type/Documentation", "showcase-example"]
+    )
+    
+    # Clean up
+    rm -rf doc_clone_dir
+```
+
+---
+
 ## Bot Signature (Required on ALL Forgejo Content)

 Every comment, issue body, PR description, and review you post to Forgejo
@@ -486,6 +728,8 @@ MODE: pool_supervisor
 TOTAL_FEATURE_AREAS: <N>
 AREAS_TESTED: <N>
 TOTAL_BUGS_FILED: <N>
+TOTAL_DOCS_GENERATED: <N>
+EXAMPLE_CATEGORIES_COVERED: [<list>]
 CYCLES_COMPLETED: <N>
 UNTESTED_AREAS: [<list>]
 ```
@@ -502,6 +746,9 @@ BUGS_FILED: <N>
  - Medium: <N>
  - Low: <N>
 BUG_ISSUE_NUMBERS: [#N, #M, ...]
+DOCUMENTATION_GENERATED: <N>
+EXAMPLE_CATEGORY: <category if docs generated>
+DOCUMENTATION_PRS: [#N, #M, ...]
 RUNTIME_TEST_COVERAGE: <percentage of features that got runtime tests>
 CODE_ANALYSIS_COVERAGE: <percentage of features that got code analysis>
 ```
@@ -0,0 +1,32 @@
+# API Clients Examples
+
+This directory contains examples of command-line API clients built with CleverAgents. These examples show how to interact with web services and APIs from the terminal.
+
+## What You'll Find Here
+
+- **Weather CLI**: Command-line weather information fetchers
+- **GitHub Tools**: Utilities for interacting with GitHub repositories
+- **REST API Wrappers**: Generic tools for consuming RESTful services
+- **Data Fetchers**: Tools that retrieve and format data from web APIs
+- **Service Monitors**: CLI tools for checking service health and status
+
+## Example Features
+
+Examples in this directory demonstrate:
+- HTTP request handling
+- JSON/XML parsing
+- Authentication patterns
+- Error handling for network operations
+- Data formatting and display
+
+## Best Practices Shown
+
+- Proper API key management
+- Rate limiting awareness
+- Graceful error handling
+- User-friendly output formatting
+- Caching strategies
+
+---
+
+*Examples in this directory are automatically generated from successful UAT test runs.*
@@ -0,0 +1,30 @@
+# CLI Tools Examples
+
+This directory contains examples of command-line tools built with CleverAgents. These examples demonstrate how to create practical text-based applications that developers use every day.
+
+## What You'll Find Here
+
+- **Todo Applications**: Task management tools with add, list, complete functionality
+- **File Organizers**: Tools to sort and manage files based on various criteria
+- **Text Processors**: Utilities for formatting, converting, or analyzing text
+- **Project Scaffolders**: Tools that generate boilerplate code and project structures
+- **Developer Utilities**: Various helper tools for common development tasks
+
+## Example Categories
+
+All examples in this directory:
+- Are fully text-based (no GUI required)
+- Can be run from the command line
+- Produce clear, useful output
+- Demonstrate CleverAgents' ability to build real tools
+
+## Getting Started
+
+1. Browse the examples in this directory
+2. Pick one that matches your interest
+3. Follow the step-by-step walkthrough
+4. Modify and extend to create your own version
+
+---
+
+*Examples in this directory are automatically generated from successful UAT test runs.*
@@ -0,0 +1,32 @@
+# Data Processing Examples
+
+This directory contains examples of data processing tools built with CleverAgents. These examples demonstrate how to analyze, transform, and work with various data formats.
+
+## What You'll Find Here
+
+- **CSV Analyzers**: Tools for reading, analyzing, and reporting on CSV data
+- **JSON Transformers**: Utilities for reshaping and converting JSON data
+- **Log Parsers**: Tools for extracting insights from log files
+- **Data Validators**: Utilities for checking data integrity and format
+- **Report Generators**: Tools that create summaries from raw data
+
+## Supported Formats
+
+Examples cover common data formats:
+- CSV (Comma-Separated Values)
+- JSON (JavaScript Object Notation)
+- Plain text logs
+- Structured text files
+- Configuration files
+
+## Common Patterns
+
+- Reading and parsing files
+- Data validation and cleaning
+- Aggregation and summarization
+- Format conversion
+- Report generation
+
+---
+
+*Examples in this directory are automatically generated from successful UAT test runs.*
@@ -0,0 +1,98 @@
+# [Example Title - e.g., "Building a Simple Todo CLI App"]
+
+## Overview
+[Brief 2-3 sentence description of what this example demonstrates and why it's useful. Focus on the practical value and what the user will learn.]
+
+## Prerequisites
+- CleverAgents installed (`pip install cleveragents`)
+- Python 3.12 or higher
+- [Any additional requirements specific to this example]
+
+## What You'll Build
+[Detailed description of the end result - what will the user have when they complete this example? Include key features and capabilities.]
+
+## Step-by-Step Walkthrough
+
+### Step 1: [Initialize the Project]
+```bash
+$ agents [command with arguments]
+```
+
+**Expected Output:**
+```
+[Actual output from the UAT test run]
+```
+
+**What's Happening:**
+[Explanation of what CleverAgents is doing at this step - help the user understand the system's behavior]
+
+### Step 2: [Define the Core Functionality]
+```bash
+$ agents [next command]
+```
+
+**Expected Output:**
+```
+[Actual output]
+```
+
+**What's Happening:**
+[Explanation of this step's purpose and what's being created]
+
+### Step 3: [Add Features/Test/Deploy]
+```bash
+$ agents [command]
+```
+
+**Expected Output:**
+```
+[Output showing success]
+```
+
+**What's Happening:**
+[Explanation of final steps]
+
+## Testing Your Creation
+
+### Running the Application
+```bash
+$ [command to run the created app]
+```
+
+**Example Usage:**
+```
+[Show the app in action with sample inputs/outputs]
+```
+
+## Complete Interaction Log
+<details>
+<summary>Click to see the full CleverAgents session log</summary>
+
+```
+[Timestamp] Starting CleverAgents session...
+[Timestamp] User: agents [initial command]
+[Timestamp] CleverAgents: [response]
+[... complete timestamped interaction log ...]
+[Timestamp] Session completed successfully.
+```
+</details>
+
+## Key Takeaways
+- [Important concept or pattern demonstrated]
+- [Best practice illustrated]
+- [Feature of CleverAgents highlighted]
+- [Common use case addressed]
+
+## Try It Yourself
+Now that you've seen how to [what was built], try these variations:
+- [Suggested modification 1]
+- [Suggested modification 2]
+- [Challenge: Advanced variation]
+
+## Related Examples
+- [Link to similar example if exists]
+- [Link to more advanced example if exists]
+
+---
+*This example was automatically generated and verified by the CleverAgents UAT system.*
+*Feature area: [area] | Test cycle: [N] | Generated: [timestamp]*
@@ -0,0 +1,26 @@
+{
+  "examples": [],
+  "categories": {
+    "cli-tools": {
+      "name": "CLI Tools",
+      "description": "Text-based command-line applications",
+      "keywords": ["CLI", "command-line", "terminal", "console", "tool"]
+    },
+    "api-clients": {
+      "name": "API Clients", 
+      "description": "Command-line clients for web APIs",
+      "keywords": ["API", "REST", "client", "HTTP", "web", "request"]
+    },
+    "data-processing": {
+      "name": "Data Processing",
+      "description": "Tools for data analysis and transformation",
+      "keywords": ["data", "CSV", "JSON", "parser", "analyzer", "transform"]
+    },
+    "testing-tools": {
+      "name": "Testing Tools",
+      "description": "Testing utilities and automation",
+      "keywords": ["test", "pytest", "behave", "unittest", "automation", "QA"]
+    }
+  },
+  "last_updated": null
+}
@@ -0,0 +1,50 @@
+# CleverAgents Showcase
+
+This section contains real-world examples of CleverAgents in action, automatically generated from successful UAT (User Acceptance Testing) runs. Each example demonstrates a complete, working workflow that you can follow to build something useful with CleverAgents.
+
+## Example Categories
+
+### 🛠️ CLI Tools
+Text-based command-line applications that demonstrate CleverAgents' ability to create practical developer tools.
+
+- [Examples in this category](cli-tools/)
+
+### 🌐 API Clients
+Examples showing how to build command-line clients for web APIs, demonstrating HTTP interaction and data processing.
+
+- [Examples in this category](api-clients/)
+
+### 📊 Data Processing
+Tools for analyzing, transforming, and working with various data formats like CSV, JSON, and logs.
+
+- [Examples in this category](data-processing/)
+
+### 🧪 Testing Tools
+Examples of creating testing utilities and automation tools using CleverAgents.
+
+- [Examples in this category](testing-tools/)
+
+## About These Examples
+
+All examples in this showcase are:
+
+- ✅ **Automatically Verified**: Generated from successful end-to-end test runs
+- ✅ **Self-Contained**: Require only CleverAgents and standard Python libraries
+- ✅ **Reproducible**: Include complete interaction logs and expected outputs
+- ✅ **Educational**: Show best practices and explain what's happening at each step
+- ✅ **Practical**: Demonstrate real-world use cases, not toy examples
+
+## How to Use These Examples
+
+1. **Browse by Category**: Click on a category above to see relevant examples
+2. **Follow Step-by-Step**: Each example includes detailed walkthrough instructions
+3. **Check Prerequisites**: Make sure you have the required setup before starting
+4. **Learn from Logs**: Expand the interaction logs to see exactly what happened
+
+## Contributing
+
+These examples are automatically generated by our UAT system. If you'd like to see a specific type of example, please file an issue describing the use case you'd like to see demonstrated.
+
+---
+
+*Last updated: This index is automatically maintained by the CleverAgents UAT system.*
@@ -0,0 +1,32 @@
+# Testing Tools Examples
+
+This directory contains examples of testing utilities and automation tools built with CleverAgents. These examples show how to create tools that help with software quality assurance.
+
+## What You'll Find Here
+
+- **Test Runners**: Custom test execution frameworks
+- **Test Generators**: Tools that create test cases from specifications
+- **Coverage Analyzers**: Utilities for understanding test coverage
+- **Test Data Generators**: Tools for creating realistic test data
+- **Validation Suites**: Automated checkers for various conditions
+
+## Testing Approaches
+
+Examples demonstrate various testing patterns:
+- Unit test automation
+- Integration test helpers
+- Performance test tools
+- Data validation tests
+- API testing utilities
+
+## Key Concepts
+
+- Test organization and discovery
+- Result reporting and formatting
+- Failure analysis
+- Coverage tracking
+- Continuous testing patterns
+
+---
+
+*Examples in this directory are automatically generated from successful UAT test runs.*