hurui200320
317d015260
test(e2e): validate M4 acceptance criteria for v3.3.0 milestone closure
...
Strengthen M4 E2E CLI acceptance tests and address all review findings
from the self-review on PR #681 (3 blocking, 7 non-blocking issues).
Blocking fixes:
- Remove tautological domain assertions from cli_plan_execute() that
verified values the test itself constructed via _mock_parent_plan and
could never fail. Replace with a comment documenting that subplan
info cannot be verified from CLI output alone.
- Fix unguarded positional arg access in plan_diff() that would raise
IndexError if the production CLI passes plan_id as a keyword argument.
Now uses kwargs-with-positional-fallback pattern.
- Split the 1074-line helper_m4_e2e_verification.py into four focused
modules under the CONTRIBUTING.md 500-line limit:
helper_m4_e2e_common.py (237 lines) — constants, helpers, factories
helper_m4_e2e_domain.py (495 lines) — domain model tests
helper_m4_e2e_cli.py (426 lines) — CLI integration tests
helper_m4_e2e_verification.py (81 lines) — thin dispatcher
Non-blocking improvements:
- Add ProjectLink isinstance type guard before iterating project_links.
- Wrap all assert_called_once*() calls in try/except with _fail()
pattern to produce standardised FAIL: diagnostics instead of raw
AssertionError.
- Make project_links extraction symmetric with action_name (kwargs +
positional fallback at index 1).
- Replace broad "execute" substring match with word-boundary regex
r"\bexecute\b" to avoid false positives from incidental substrings.
- Extract _make_subplan_status() factory, _assert_exit_code() and
_assert_mock_called_once*() wrappers to eliminate DRY violations
(SubplanStatus construction repeated 3x, exit-code checks 4x).
- Replace non-deterministic datetime.now() calls with frozen constant
_FROZEN_NOW = datetime(2026, 3, 1, 12, 0, 0) for deterministic
fixtures.
- Rename generic _DECISION_ULID_N constants to descriptive names
(_DECISION_PROMPT_DEF, _DECISION_STRATEGY, _DECISION_PARALLEL_SPAWN,
_DECISION_SPAWN_API, _DECISION_SPAWN_UI).
Quality gates:
- lint: PASS
- format: PASS (1404 files unchanged)
- typecheck: PASS (0 errors)
- unit_tests: 10,431 scenarios, 0 failures
- integration_tests: 1,452 tests, 0 failures
- coverage_report: PASS (98%, threshold 97%)
- security_scan: PASS
- dead_code: PASS
- docs: PASS
- build: PASS
ISSUES CLOSED : #495
2026-03-13 14:03:37 +08:00
d88dad94f6
fix(test): resolve race condition in M4 validation integration test
...
Three-pronged fix for intermittent pabot-parallel race condition in
M4 validation integration tests:
1. Composable Setup Database Isolation keyword in common.resource
gives each suite a unique CLEVERAGENTS_DATABASE_URL so concurrent
pabot workers never contend on the same SQLite file.
2. Per-suite CLEVERAGENTS_HOME directories prevent shared temp
directory cleanup from racing between workers.
3. Centralised reset_global_state() in robot/helpers_common.py clears
Settings singleton, DI container, provider registry, and engine
cache between chained CLI invocations in helper processes.
Also:
- Setup Test Environment now accepts optional mock_ai and
auto_apply_migrations arguments (default TRUE) for backward
compatibility while allowing suites to opt out.
- Added Suite Teardown to cli_plan_context_commands.robot.
- Fixed _COMMANDS typing in two helpers to eliminate type: ignore.
- Updated docs/development/testing.md to reflect helpers_common
delegation pattern.
- Added timeout=30s to all Run Process calls in
m4_e2e_verification.robot.
Fixes : #563
2026-03-10 19:04:57 +00:00
hurui200320
09d92ac67b
test(e2e): validate M4 acceptance criteria for v3.3.0 milestone closure ( #560 )
...
## Summary
Validates all M4 acceptance criteria for v3.3.0 milestone closure. Adds CLI-exercising integration tests for the three subplan commands that lacked actual CLI invocation, fixes a pre-existing unit test failure, and corrects CONTRIBUTORS.md ordering.
## Changes
### New CLI Integration Tests (Robot Framework)
Three new test cases in `robot/m4_e2e_verification.robot` with helper functions in `robot/helper_m4_e2e_verification.py`:
| Test | CLI Command | Mocking Approach |
|------|-------------|-----------------|
| `CLI Plan Use Creates Plan With Subplan Config` | `plan use local/refactor-action local/monorepo` | Patches `_get_lifecycle_service`; mocks `get_action_by_name` + `use_action` |
| `CLI Plan Execute Transitions With Subplans` | `plan execute <plan_id>` | Patches `_get_lifecycle_service`; mocks `get_plan` + `execute_plan` |
| `CLI Plan Tree Displays Subplan Hierarchy` | `plan tree <plan_id> --format json` | Patches `get_container`; real `Decision` objects with `SUBPLAN_SPAWN`/`SUBPLAN_PARALLEL_SPAWN` types |
All three follow the same pattern as the existing `plan-diff` test: Typer `CliRunner.invoke()` with mocked services.
### Pre-existing Unit Test Fix
**File:** `features/steps/repositories_uncovered_branches_steps.py`
**Scenario:** `repo branch cov upsert profile with schema version mismatch` (line 110)
**Root cause:** Plain `sessionmaker` returns a new session per call. The `Given` step inserted via session A and committed session B (different session), so the `When` step's session C couldn't see the uncommitted data.
**Fix:** Changed to `scoped_session(sessionmaker(...))` so all factory calls return the same thread-local session.
### Other Fixes
- **CONTRIBUTORS.md**: Moved "Rui Hu" to correct alphabetical position (between Freeman and Khyari)
- **CHANGELOG.md**: Updated #495 entry to document CLI test additions
## M4 Acceptance Criteria Verification
All 7 criteria exercised by 10 E2E tests (7 existing + 3 new) + 8 smoke tests:
| # | Criterion | Test(s) | Status |
|---|-----------|---------|--------|
| 1 | Subplans spawned during Execute via SubplanConfig | `spawn-subplans`, **`cli-plan-use`**, **`cli-plan-execute`** | PASS |
| 2 | Parallel execution with max_parallel bounds | `parallel-exec` | PASS |
| 3 | Three-way merge combines non-conflicting changes | `merge-clean` | PASS |
| 4 | Merge conflicts surfaced with git markers | `merge-conflict` | PASS |
| 5 | Parent plan tracks subplan statuses | `parent-tracking` | PASS |
| 6 | Plan tree displays subplan hierarchy | `plan-tree`, **`cli-plan-tree`** | PASS |
| 7 | Plan diff shows merged results | `plan-diff` (already uses CLI) | PASS |
## Quality Gates
| Stage | Result |
|-------|--------|
| lint | pass |
| format | pass (1074 files unchanged) |
| typecheck | pass (0 errors) |
| unit_tests | 8524 scenarios, 0 failures |
| integration_tests | 1110/1118 pass (8 pre-existing failures in `cli_plan_context_commands.robot`, identical on master) |
| coverage_report | 97% (threshold: 97%) |
| security_scan | pass |
| dead_code | pass |
| docs | pass |
| build | pass |
| benchmark | pass |
## Files Changed
- `CHANGELOG.md` — Updated #495 entry
- `CONTRIBUTORS.md` — Fixed alphabetical ordering
- `features/steps/repositories_uncovered_branches_steps.py` — `scoped_session` fix
- `robot/helper_m4_e2e_verification.py` — 3 new helper functions + imports
- `robot/m4_e2e_verification.robot` — 3 new test cases
ISSUES CLOSED : #495
Reviewed-on: cleveragents/cleveragents-core#560
Reviewed-by: Luis Mendes <luis.mendes@cleverthis.com >
Co-authored-by: Rui Hu <rui.hu@cleverthis.com >
Co-committed-by: Rui Hu <rui.hu@cleverthis.com >
2026-03-06 03:22:45 +00:00
brent.edwards
hurui200320
Brent E. Edwards