5 Commits

Author	SHA1	Message	Date
brent.edwards	d31d9226f9	test(wf10_batch.robot): request that LLM not create files (#11133) ... Closes: #11007 ## Description Asks LLM to not create files. ## Type of Change - [ X ] Bug fix (non-breaking change which fixes an issue) - [ X ] Test improvements ## Quality Checklist - [ ] Code follows the project's coding standards (see CONTRIBUTING.md) - [ ] All public/protected methods have argument validation - [ ] Static typing is complete (no `Any` unless justified) - [ ] `nox -s typecheck` passes with no errors - [ ] `nox -s lint` passes with no errors - [ ] Unit tests written/updated (Behave scenarios in `features/`) - [ ] Integration tests written/updated (Robot suites in `robot/`) if applicable - [ ] Coverage remains above 85% (`nox -s coverage_report`) - [ ] No security issues introduced (`nox -s security_scan`) - [ ] No dead code introduced (`nox -s dead_code`) - [ ] Documentation updated if behavior changed ## Testing Running in git.cleverthis.com is the only real test possible. ### Test Commands Run ```bash nox -s unit_tests # Behave tests nox -s typecheck # Type checking nox -s lint # Linting nox -s coverage_report # Coverage ``` ## Related Issues Closes #11007 Reviewed-on: #11133 Reviewed-by: Hamza Khyari <hamza.khyari@cleverthis.com>	2026-05-13 23:39:15 +00:00
brent.edwards	57930c9fb3	fix(wf10): fixing more of the add/add problems ... Closes #10861	2026-05-05 00:38:44 +00:00
brent.edwards	e249afa30e	fix(wf10_batch): fix add/add conflict ... Closes: #10861	2026-05-04 15:54:22 -07:00
brent.edwards	56e9f86749	fix(e2e): relax wf10 batch apply assertion from == 3 to >= 2 ... The CI run showed that one of the three healthy plans occasionally fails during the apply phase (LLM-generated changes can cause merge conflicts or produce empty changesets). The assertion `success_count == 3` was therefore too strict for a real-LLM E2E test. Change to `>= 2`: still verifies the batch apply mechanism works (at least two of three plans reach `applied` state) while tolerating one transient LLM-related apply failure. ISSUES CLOSED: #756	2026-04-22 02:35:13 +00:00
hurui200320	66b42eed0e	test(e2e): workflow example 10 — full-auto batch formatting and linting (full-auto profile) ... Add E2E Robot Framework test for Workflow Example 10: Full-Auto Batch Operations. The test creates a temporary monorepo with 3 badly-formatted Python packages (pkg_auth, pkg_common, pkg_billing), registers a formatting action with full-auto automation profile and whitespace-only invariants, creates git-checkout resources and projects for each package, launches plans in full-auto mode, executes plans through the strategize+execute pipeline, applies changes via `plan apply --yes` (which runs and completes the apply phase), and verifies batch completion via `plan list --state applied` and `--state errored` filters. Error handling is demonstrated via a deliberately broken action using a non-existent LLM actor (nonexistent/model-xyz-404). Plans created with this action fail during execution, and the test verifies the specific broken plan ID is absent from successfully-executed plans or present in the errored list. Key design decisions: - Explicit execution pipeline: plan use creates plans, plan execute auto-advances strategize+execute with full-auto profile, and `plan apply --yes` performs the actual application and completes the apply phase. Note: `plan lifecycle-apply` only transitions the plan INTO the apply phase (apply/queued) without completing it. - Dynamic actor selection: uses ANTHROPIC_API_KEY if available, falls back to openai/gpt-4o. - Canonical ULID regex with Crockford Base32 for plan ID extraction. - Per-test teardown resets automation profile to manual. - 25-minute timeout covers worst-case 4-plan batch execution. Zero mocking — exercises the real CLI with real LLM API keys. Tagged E2E and skips gracefully when API keys are absent. ISSUES CLOSED: #756	2026-04-22 02:35:13 +00:00