test(integration): workflow example 6 — documentation generation from codebase analysis (trusted profile) #770

Closed
opened 2026-03-12 19:38:59 +00:00 by freemo · 6 comments
Owner

Metadata

  • Commit Message: test(integration): workflow example 6 — documentation generation from codebase analysis (trusted profile)
  • Branch: test/int-wf06-doc-generation

Background

Integration test for Specification Workflow Example 6: Writing Technical Documentation from Codebase Analysis. Exercises the trusted automation profile with context policy configuration, code intelligence, and documentation-generation invariants using mocked LLM providers.

Runs within the standard nox -s integration_tests session using mocked LLM providers.

Expected Behavior

The integration test validates documentation generation with mocked LLM responses. Context policies are configured, the mocked LLM produces documentation content, and invariants (no source modification) are enforced.

Acceptance Criteria

  • Robot Framework test suite in robot/ directory (standard integration tests)
  • Test configures context policy with view-specific settings
  • Test uses integration-appropriate mocking (mocked LLM providers)
  • Test verifies new documentation files are created
  • Test verifies no source files are modified (invariant)
  • Test passes via nox -s integration_tests
  • Coverage >=97% maintained

Subtasks

  • Write Robot Framework integration test suite for workflow example 6
  • Configure mocked LLM responses for documentation generation
  • Create temp project with source code fixture
  • Implement trusted-profile documentation workflow
  • Verify via nox -s integration_tests
  • Verify coverage >=97% via nox -s coverage_report
  • Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
## Metadata - **Commit Message**: `test(integration): workflow example 6 — documentation generation from codebase analysis (trusted profile)` - **Branch**: `test/int-wf06-doc-generation` ## Background Integration test for Specification Workflow Example 6: Writing Technical Documentation from Codebase Analysis. Exercises the `trusted` automation profile with context policy configuration, code intelligence, and documentation-generation invariants using mocked LLM providers. Runs within the standard `nox -s integration_tests` session using mocked LLM providers. ## Expected Behavior The integration test validates documentation generation with mocked LLM responses. Context policies are configured, the mocked LLM produces documentation content, and invariants (no source modification) are enforced. ## Acceptance Criteria - [x] Robot Framework test suite in `robot/` directory (standard integration tests) - [x] Test configures context policy with view-specific settings - [x] Test uses integration-appropriate mocking (mocked LLM providers) - [x] Test verifies new documentation files are created - [x] Test verifies no source files are modified (invariant) - [x] Test passes via `nox -s integration_tests` - [x] Coverage >=97% maintained ## Subtasks - [x] Write Robot Framework integration test suite for workflow example 6 - [x] Configure mocked LLM responses for documentation generation - [x] Create temp project with source code fixture - [x] Implement trusted-profile documentation workflow - [x] Verify via `nox -s integration_tests` - [x] Verify coverage >=97% via `nox -s coverage_report` - [x] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done.
freemo added this to the v3.1.0 milestone 2026-03-12 19:39:00 +00:00
freemo modified the milestone from v3.1.0 to v3.4.0 2026-03-16 00:32:01 +00:00
Member

Started work on #770 in isolated workspace /tmp/cleveragents-770 on branch test/int-wf06-doc-generation.

Preparation log:

  • Verified issue is open.
  • Verified and corrected assignee to brent.edwards (was previously assigned to hurui200320).
  • Transitioned state label to State/In Progress.
  • Read docs/specification.md (WF06 section), CONTRIBUTING.md (project-specific test/tooling/coverage rules), and docs/timeline.md for milestone context.

Implementation plan:

  1. Add Robot integration suite for WF06 under robot/ (non-E2E).
  2. Add a dedicated helper script to exercise trusted-profile workflow semantics with mocked providers and deterministic assertions.
  3. Validate context policy view-specific settings, documentation artifact creation under output directory, and no source-file modification invariant.
  4. Run required nox gates and coverage checks before commit.
Started work on #770 in isolated workspace `/tmp/cleveragents-770` on branch `test/int-wf06-doc-generation`. Preparation log: - Verified issue is open. - Verified and corrected assignee to `brent.edwards` (was previously assigned to `hurui200320`). - Transitioned state label to `State/In Progress`. - Read `docs/specification.md` (WF06 section), `CONTRIBUTING.md` (project-specific test/tooling/coverage rules), and `docs/timeline.md` for milestone context. Implementation plan: 1. Add Robot integration suite for WF06 under `robot/` (non-E2E). 2. Add a dedicated helper script to exercise trusted-profile workflow semantics with mocked providers and deterministic assertions. 3. Validate context policy view-specific settings, documentation artifact creation under output directory, and no source-file modification invariant. 4. Run required nox gates and coverage checks before commit.
Member

Implementation Notes — WF06 Integration Test

Design Decisions

  1. Followed the WF05 pattern exactly: Robot Framework .robot file + Python helper script with command dispatch. This ensures consistency across workflow integration tests.

  2. Six test cases covering the acceptance criteria:

    • context-policy-views — Context policy with strategize/execute view-specific settings and inheritance resolution
    • budget-enforcement — Size budget enforcement filtering ContextFragments by max_file_size and max_total_size
    • trusted-profile-behavior — Trusted profile auto-progress (Strategize → Execute automatic; Execute → Apply gated)
    • action-with-doc-args — Action creation with doc_types/output_dir args and 3 invariants per spec
    • doc-generation-sandbox — Temp project + sandbox with documentation generation and source-code invariant
    • trusted-doc-lifecycle — Full plan lifecycle with trusted profile, args, invariants
  3. Automation profile binding: PlanLifecycleService.use_action() does not propagate action.automation_profile to plan.automation_profile (the AutomationProfileRef). The profile must be explicitly bound on the plan. The test sets it via plan.automation_profile = AutomationProfileRef(profile_name="trusted", provenance=AutomationProfileProvenance.ACTION).

  4. Auto-progress behavior: complete_strategize() internally calls auto_progress(), which checks should_auto_progress(). With the trusted profile (create_tool=0.0), this returns True and the plan auto-advances to Execute/QUEUED. The test validates this by asserting the returned plan is in Execute phase rather than Strategize/COMPLETE.

Key Code Locations

  • robot/int_wf06_doc_generation.robot — Robot Framework test suite
  • robot/helper_int_wf06_doc_generation.py — Python helper with 6 subcommands
  • Spec reference: docs/specification.md ~lines 38700-39039 (Example 6)

Test Coverage

  • ContextView / ProjectContextPolicy (view resolution with inheritance chain)
  • enforce_size_budget (max_file_size + max_total_size filtering)
  • PlanLifecycleService (trusted profile auto-progress, complete_strategize auto-advance, gated apply)
  • Action with args (ArgumentType.STRING, REQUIRED/OPTIONAL, default_value) and invariants
  • SandboxManager / SandboxFactory (copy_on_write sandbox for doc generation)
  • Source-code invariant verification (auth.py, models.py, utils.py unchanged in sandbox)

All 6 tests pass via nox -s integration_tests. Proceeding to run quality gates.

## Implementation Notes — WF06 Integration Test ### Design Decisions 1. **Followed the WF05 pattern exactly**: Robot Framework `.robot` file + Python helper script with command dispatch. This ensures consistency across workflow integration tests. 2. **Six test cases** covering the acceptance criteria: - `context-policy-views` — Context policy with strategize/execute view-specific settings and inheritance resolution - `budget-enforcement` — Size budget enforcement filtering ContextFragments by `max_file_size` and `max_total_size` - `trusted-profile-behavior` — Trusted profile auto-progress (Strategize → Execute automatic; Execute → Apply gated) - `action-with-doc-args` — Action creation with `doc_types`/`output_dir` args and 3 invariants per spec - `doc-generation-sandbox` — Temp project + sandbox with documentation generation and source-code invariant - `trusted-doc-lifecycle` — Full plan lifecycle with trusted profile, args, invariants 3. **Automation profile binding**: `PlanLifecycleService.use_action()` does not propagate `action.automation_profile` to `plan.automation_profile` (the `AutomationProfileRef`). The profile must be explicitly bound on the plan. The test sets it via `plan.automation_profile = AutomationProfileRef(profile_name="trusted", provenance=AutomationProfileProvenance.ACTION)`. 4. **Auto-progress behavior**: `complete_strategize()` internally calls `auto_progress()`, which checks `should_auto_progress()`. With the trusted profile (`create_tool=0.0`), this returns True and the plan auto-advances to Execute/QUEUED. The test validates this by asserting the returned plan is in Execute phase rather than Strategize/COMPLETE. ### Key Code Locations - `robot/int_wf06_doc_generation.robot` — Robot Framework test suite - `robot/helper_int_wf06_doc_generation.py` — Python helper with 6 subcommands - Spec reference: `docs/specification.md` ~lines 38700-39039 (Example 6) ### Test Coverage - ContextView / ProjectContextPolicy (view resolution with inheritance chain) - enforce_size_budget (max_file_size + max_total_size filtering) - PlanLifecycleService (trusted profile auto-progress, complete_strategize auto-advance, gated apply) - Action with args (ArgumentType.STRING, REQUIRED/OPTIONAL, default_value) and invariants - SandboxManager / SandboxFactory (copy_on_write sandbox for doc generation) - Source-code invariant verification (auth.py, models.py, utils.py unchanged in sandbox) All 6 tests pass via `nox -s integration_tests`. Proceeding to run quality gates.
Member

Quality Gates — Final Results

All nox sessions pass on commit be685010:

Session Result
lint Pass
format Pass
typecheck Pass (17s)
security_scan Pass (15s)
dead_code Pass (2s)
unit_tests Pass (5 min)
integration_tests Pass (7 min)
e2e_tests Pass
docs Pass (2 min)
build Pass (2s)
benchmark Pass (24 min)
coverage_report Pass — 97%

PR #1230 created → test/int-wf06-doc-generationmaster

Issue transitioned to State/In Review.

## Quality Gates — Final Results All nox sessions pass on commit `be685010`: | Session | Result | |---|---| | lint | ✅ Pass | | format | ✅ Pass | | typecheck | ✅ Pass (17s) | | security_scan | ✅ Pass (15s) | | dead_code | ✅ Pass (2s) | | unit_tests | ✅ Pass (5 min) | | integration_tests | ✅ Pass (7 min) | | e2e_tests | ✅ Pass | | docs | ✅ Pass (2 min) | | build | ✅ Pass (2s) | | benchmark | ✅ Pass (24 min) | | coverage_report | ✅ Pass — **97%** | PR #1230 created → `test/int-wf06-doc-generation` → `master` Issue transitioned to `State/In Review`.
Author
Owner

PR #1230 reviewed, approved, and merged.

PR #1230 reviewed, approved, and merged.
freemo self-assigned this 2026-04-02 06:13:54 +00:00
Author
Owner

PR #1230 reviewed, approved, and merged.

Integration test suite for Workflow Example 6 (documentation generation from codebase analysis with trusted profile) is now on master. The PR added 6 Robot Framework test cases covering context policy views, budget enforcement, trusted profile auto-progress behavior, action creation with doc args/invariants, sandbox doc generation with source invariant verification, and full trusted lifecycle.

PR #1230 reviewed, approved, and merged. Integration test suite for Workflow Example 6 (documentation generation from codebase analysis with trusted profile) is now on `master`. The PR added 6 Robot Framework test cases covering context policy views, budget enforcement, trusted profile auto-progress behavior, action creation with doc args/invariants, sandbox doc generation with source invariant verification, and full trusted lifecycle.
Author
Owner

PR #1230 reviewed, approved, and merged.

The integration test suite for Workflow Example 6 (documentation generation from codebase analysis with trusted profile) has been squash-merged into master. All 6 test cases exercise meaningful spec behavior and quality gates pass.

PR #1230 reviewed, approved, and merged. The integration test suite for Workflow Example 6 (documentation generation from codebase analysis with trusted profile) has been squash-merged into `master`. All 6 test cases exercise meaningful spec behavior and quality gates pass.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#770
No description provided.