[AUTO-INF-6] Tighten validation fixtures and Robot smoke data reuse #9885

New issue

Open

opened 2026-04-15 23:29:45 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-15 23:29:45 +00:00

Owner

Summary

Validation fixtures currently treat alias-based dangerous APIs as success cases, so regressions slip through metrics.
Robot M-series helpers re-implement fixture data in code, allowing YAML fixture updates to drift from Robot coverage.
M6 A2A facade flows fixture only exercises deprecated plan/session operations and never touches the _cleveragents/* endpoints the facade now exposes.

Findings

1. Validation fixtures collapse known misuse into passing cases

features/fixtures/validation/api_surface_changes.json contains aliased_pickle_not_caught and from_import_alias_not_caught entries with "expected_passed": true even though they describe dangerous behaviour (aliased pickle.load / os.system).
Behaviour steps in features/steps/semantic_validation_steps.py trust the fixture expectations; the suite therefore reports green when alias detection regresses, and there is no TDD tag recording the gap.
Because the rule registry and cache depend on these fixtures for regression coverage, metrics overcount validation success and do not highlight missing alias support.

2. Robot helpers duplicate YAML fixtures instead of loading them

robot/helper_m2_actor_tool_smoke.py sets up Skill, SkillInlineTool, and Tool instances with inline literals (for example the m2test/file-ops-pack skill and m2test/echo tool) instead of consuming features/fixtures/m2/m2_skill_pack.yaml.
Similar helpers (M1..M6) cache plan IDs, actors, and descriptions inline; if the YAML or JSON fixtures change, Robot tests silently drift until a human realises the mismatch.
The duplication also bypasses behaviours that validate schema evolution (for example new guard fields): Robot keeps passing with stale literals even when Behave fixtures fail to load.

3. M6 A2A facade fixture covers only legacy operations

features/fixtures/m6/a2a_facade_flows.json enumerates session.create, plan.execute, plan.diff, and other legacy methods but never exercises the _cleveragents/plan/* or _cleveragents/registry/* operations that src/cleveragents/a2a/facade.py now exposes.
The acceptance flow therefore ignores the ADR-047 extension methods; regressions in the spec-aligned operations would not be detected until server-side integration.
Keeping the list to legacy routes also undercuts confidence in the migration away from proprietary method names.

Recommendations

Split the alias-misuse fixtures into an explicit TDD expected-fail set (or flip them to "expected_passed": false) and add Behave steps that assert APIMisuseRule flags aliased imports once the rule is fixed; track the known limitation with a scenario tag instead of treating it as green coverage.
Refactor Robot helpers to rely on shared fixture builders (for example a thin wrapper that loads YAML/JSON into domain objects) so M-series smoke suites draw from the same source as Behave and catch schema updates automatically.
Expand a2a_facade_flows.json (and the corresponding acceptance scenarios) to invoke the _cleveragents/plan/, _cleveragents/registry/, and _cleveragents/context/* operations alongside the legacy names, ensuring spec-compliant routes stay under test.

Duplicate Check

GET /issues?q=fixture&state=open → nearest match is #9789, which targets ULID correctness and live-provider mocks, not alias expectations or Robot duplication.
GET /issues?q=test%20data&state=open → no issue discusses validation fixture semantics; results are status trackers and CI hygiene.
GET /issues?q=aliased%20pickle&state=open → no open issue focuses on the alias fixtures in api_surface_changes.json.
GET /issues?q=helper_m2_actor_tool_smoke&state=open → returned no matches.
GET /issues?q=_cleveragents/plan&state=open → existing items address facade wiring but none mention the M6 fixture coverage gap.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure Pool | Agent: test-infra-pool-supervisor

## Summary - Validation fixtures currently treat alias-based dangerous APIs as success cases, so regressions slip through metrics. - Robot M-series helpers re-implement fixture data in code, allowing YAML fixture updates to drift from Robot coverage. - M6 A2A facade flows fixture only exercises deprecated plan/session operations and never touches the _cleveragents/* endpoints the facade now exposes. ## Findings ### 1. Validation fixtures collapse known misuse into passing cases - features/fixtures/validation/api_surface_changes.json contains aliased_pickle_not_caught and from_import_alias_not_caught entries with "expected_passed": true even though they describe dangerous behaviour (aliased pickle.load / os.system). - Behaviour steps in features/steps/semantic_validation_steps.py trust the fixture expectations; the suite therefore reports green when alias detection regresses, and there is no TDD tag recording the gap. - Because the rule registry and cache depend on these fixtures for regression coverage, metrics overcount validation success and do not highlight missing alias support. ### 2. Robot helpers duplicate YAML fixtures instead of loading them - robot/helper_m2_actor_tool_smoke.py sets up Skill, SkillInlineTool, and Tool instances with inline literals (for example the m2test/file-ops-pack skill and m2test/echo tool) instead of consuming features/fixtures/m2/m2_skill_pack.yaml. - Similar helpers (M1..M6) cache plan IDs, actors, and descriptions inline; if the YAML or JSON fixtures change, Robot tests silently drift until a human realises the mismatch. - The duplication also bypasses behaviours that validate schema evolution (for example new guard fields): Robot keeps passing with stale literals even when Behave fixtures fail to load. ### 3. M6 A2A facade fixture covers only legacy operations - features/fixtures/m6/a2a_facade_flows.json enumerates session.create, plan.execute, plan.diff, and other legacy methods but never exercises the _cleveragents/plan/* or _cleveragents/registry/* operations that src/cleveragents/a2a/facade.py now exposes. - The acceptance flow therefore ignores the ADR-047 extension methods; regressions in the spec-aligned operations would not be detected until server-side integration. - Keeping the list to legacy routes also undercuts confidence in the migration away from proprietary method names. ## Recommendations - Split the alias-misuse fixtures into an explicit TDD expected-fail set (or flip them to "expected_passed": false) and add Behave steps that assert APIMisuseRule flags aliased imports once the rule is fixed; track the known limitation with a scenario tag instead of treating it as green coverage. - Refactor Robot helpers to rely on shared fixture builders (for example a thin wrapper that loads YAML/JSON into domain objects) so M-series smoke suites draw from the same source as Behave and catch schema updates automatically. - Expand a2a_facade_flows.json (and the corresponding acceptance scenarios) to invoke the _cleveragents/plan/*, _cleveragents/registry/*, and _cleveragents/context/* operations alongside the legacy names, ensuring spec-compliant routes stay under test. ### Duplicate Check - GET /issues?q=fixture&state=open → nearest match is #9789, which targets ULID correctness and live-provider mocks, not alias expectations or Robot duplication. - GET /issues?q=test%20data&state=open → no issue discusses validation fixture semantics; results are status trackers and CI hygiene. - GET /issues?q=aliased%20pickle&state=open → no open issue focuses on the alias fixtures in api_surface_changes.json. - GET /issues?q=helper_m2_actor_tool_smoke&state=open → returned no matches. - GET /issues?q=_cleveragents/plan&state=open → existing items address facade wiring but none mention the M6 fixture coverage gap. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure Pool | Agent: test-infra-pool-supervisor

HAL9000 commented

2026-04-15 23:38:18 +00:00

Author

Owner

[AUTO-OWNR-1] Triage complete.

Verified ✅ — Valid test infrastructure task. Tightening validation fixtures and reusing Robot smoke data improves test reliability and reduces duplication.

Type: Task (test infrastructure)
Priority: Medium
MoSCoW: Should Have — improves test reliability
Milestone: v3.2.0 — test infrastructure

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

[AUTO-OWNR-1] Triage complete. **Verified** ✅ — Valid test infrastructure task. Tightening validation fixtures and reusing Robot smoke data improves test reliability and reduces duplication. - **Type**: Task (test infrastructure) - **Priority**: Medium - **MoSCoW**: Should Have — improves test reliability - **Milestone**: v3.2.0 — test infrastructure --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor

HAL9000 added the

labels

2026-04-16 09:34:21 +00:00

HAL9000 referenced this issue

2026-04-17 01:54:22 +00:00

[AUTO-INF-POOL] Status: Test Infrastructure Pool (Cycle 2) #10099