Commit Graph

17 Commits

Author SHA1 Message Date
CoreRasurae 007af498b8 refactor(autonomy): rename automation profile task flags to spec names
Renamed all 11 task-type confidence threshold fields in AutomationProfile
from phase-transition semantics to spec-defined task-type semantics.
Updated all 8 built-in profiles, CLI formatting, YAML schema, services,
and all Behave/Robot tests referencing the old field names.

Post-review fixes:
- Fixed 24 stale old field names in M6 fixture files
  (automation_profiles.json, autonomy_guardrails.json)
- Added model_validator(mode='before') to detect legacy field names
  and raise actionable ValueError with rename mapping
- Added semantic bridge comments in PlanLifecycleService mapping
  task-type thresholds to phase-transition gates
- Added threshold_field to structured log messages for observability
- Restored categorised CLI automation-profile show output to match
  spec (Phase Transitions / Decision Automation / Self-Repair /
  Execution Controls) instead of flat list
- Added missing access_network field to spec show output examples
  (Rich, Plain, JSON, YAML variants)
- Aligned ADR-017 profile fields table to all 11 fields with
  descriptions matching spec Automatable Tasks table
- Aligned automation_profiles.md threshold descriptions with spec
- Added spec section references in phase_reversion.md, error_recovery.md,
  and plan_execute.md for field naming context
- Extended repository roundtrip test to assert all 11 threshold fields
- Fixed benchmark _make_profile() passing safety fields as top-level
  kwargs instead of via SafetyProfile sub-model (incompatible with
  extra="forbid")
- Aligned CLI JSON/YAML output structure for automation-profile show
  with the specification grouped format (phase_transitions,
  decision_automation, self_repair, execution_controls)
- Moved safety boolean fields into the Execution Controls section
  of Rich output per spec examples
- Reverted auto profile description to "Fully automatic except apply"
  per specification (line 16703, line 28406)
- Improved bridge comments in test steps with semantic context for
  threshold-to-gate mappings

ISSUES CLOSED: #902
2026-03-30 13:18:07 +01:00
brent.edwards b88bc0ec1b feat(perf): large project scaling tests (#984)
## Summary

Add large project scaling benchmarks and tests at production scale (10K–100K files).

### New ASV Benchmarks

**IndexingScalingSuite** (`large_project_scaling_bench.py`):
- `time_walk_and_index` at 1K/10K/50K/100K files
- `time_incremental_refresh` (1% modified files)
- `track_indexed_file_count`, `track_tokens_per_second`

**ContextAssemblyScalingSuite** (`context_assembly_scaling_bench.py`):
- `time_full_pipeline` at 100/1K/5K/10K fragments
- `time_tiered_strategy`, `time_recency_strategy`
- `track_assembled_tokens`, `track_fragments_per_second`

**ExecutionThroughputSuite** (`execution_throughput_bench.py`):
- `time_sequential_plans` at 10/50/100 plans
- `time_executor_construction`, `time_decision_tree_scaling`

### Scale Fixture Updates

- Added `xlarge` (50K files) and `xxlarge` (100K files) profiles to `scale_metadata.json`
- Added 50K/100K thresholds to `baseline_thresholds.json`
- Added `context_assembly` and `execution_throughput` threshold sections

### Tests & Documentation

- 15 Behave scenarios validating profiles, thresholds, monotonicity, memory budgets
- 6 Robot integration tests including live 1K-file indexing throughput check
- `docs/reference/scaling_baselines.md` documenting all baseline metrics

### Quality Gates

| Session | Result |
|---|---|
| `nox -s lint` | PASS |
| `nox -s typecheck` | PASS (0 errors) |
| `nox -s unit_tests` | PASS (10,910 scenarios) |
| `nox -s integration_tests` | PASS (1,526 tests) |
| `nox -s coverage_report` | 97% (>= 97%) |

Closes #859

Reviewed-on: cleveragents/cleveragents-core#984
Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
2026-03-21 04:46:45 +00:00
freemo ec0b7631d0 refactor(a2a): rename ACP module and symbols to A2A standard
Renamed src/cleveragents/acp/ to src/cleveragents/a2a/ and all 13
Acp* classes to A2a* per ADR-047 (A2A Standard Adoption). Updated
all imports, structlog event names (acp.* → a2a.*), field names
(acp_version → a2a_version), and test references across the entire
codebase. This is a cosmetic rename only — no behavioral changes.

ISSUES CLOSED: #688
2026-03-12 14:38:57 +00:00
CoreRasurae 7e6f6fae37 test(validation): add semantic validation suites
Add comprehensive semantic validation test suites covering five
new fixture categories: language porting mismatches, dependency
graph violations, API surface changes, cross-file symbols, and
circular import detection.

New BDD scenarios (38) exercise all six built-in rules against
fixture-driven inputs.  Robot Framework integration tests (11)
validate end-to-end rule execution via the SemanticValidationService.
ASV benchmarks (6 suites) establish performance baselines for
batch validation throughput and per-rule latency.

Files added:
- features/fixtures/validation/{language_porting_mismatches,
  dependency_graph_violations, api_surface_changes,
  cross_file_symbols, circular_import_detection}.json
- features/semantic_validation_suite.feature
- features/steps/semantic_validation_suite_steps.py
- robot/semantic_validation_suite.robot
- robot/helper_semantic_validation_suite.py
- benchmarks/semantic_validation_suite_bench.py
- docs/reference/semantic_validation_coverage.md

All 11 nox sessions pass (lint, format, typecheck, security_scan,
dead_code, unit_tests, integration_tests, docs, build, benchmark,
coverage_report).  Coverage remains at 97%.

ISSUES CLOSED: #316
2026-03-02 22:05:53 +00:00
brent.edwards 38e75342a9 refactor: merge from master
This code is a merge from master.
2026-02-28 05:49:26 +00:00
brent.edwards 348c230bc5 test(e2e): add M6 autonomy acceptance suite
Add comprehensive M6 autonomy acceptance test suites covering the ACP
local-mode facade, autonomy guardrails, automation profile resolution,
event queue pub/sub, HTTP transport stub, and version negotiation.

Behave suite (52 scenarios):
- ACP facade dispatch for all 11 operations
- Guard enforcement (denylist, allowlist, budget, call limit, write/apply)
- Automation profile built-in validation and custom creation
- Profile resolution precedence (plan > action > project > global)
- Event queue lifecycle (publish, subscribe, unsubscribe, close)
- HTTP transport stub rejection in local mode
- ACP version negotiation (accept/reject)
- Model validation (AcpRequest, AcpResponse, AcpEvent, AcpErrorDetail)

Robot integration suite (11 tests):
- Facade session/plan lifecycle, unknown operation error
- Event queue publish/subscribe, transport stub, version negotiation
- Guard denylist/budget enforcement, profile resolution
- Fixture loading, full end-to-end flow

ASV benchmarks (5 suites):
- Facade dispatch, guard evaluation, profile resolution
- Event queue operations, fixture loading

Fixtures: acp_facade_flows.json, autonomy_guardrails.json,
automation_profiles.json

Closes #211
2026-02-27 20:31:15 +00:00
brent.edwards ece5e61725 test(e2e): add M5 ACMS + context suites 2026-02-27 20:20:13 +00:00
brent.edwards c80cda2590 Merge feature/m4-correction-subplan-smoke into develop-brent-5 2026-02-25 22:11:33 +00:00
brent.edwards 917c2bc546 test(e2e): add M4 correction + subplan suites 2026-02-25 21:51:26 +00:00
brent.edwards ace7311de4 test(e2e): add M3 decision + validation suites 2026-02-25 21:26:19 +00:00
brent.edwards e230700857 Merge feature/m1-e2e-sourcecode into develop-brent-4 2026-02-25 06:36:45 +00:00
brent.edwards 0397a00eb6 test(e2e): add M1 source-code plan lifecycle suite 2026-02-25 04:50:43 +00:00
brent.edwards 093a74953f test(e2e): add M2 actor + tool source smoke suite
Add comprehensive E2E test suite for M2 (Actor Graphs + Tool Sources) epic:

- Behave BDD: 10 scenarios covering actor YAML loading, skill registry,
  tool lifecycle (discover/activate/execute/deactivate), and MCP stub
- Robot Framework: 6 integration tests via CLI helper script
- ASV benchmarks: 12 benchmarks for actor loading, skill registry,
  tool lifecycle, and MCP stub performance baselines
- MCP stub server mock: in-process fake with 3 tools (search/fetch/transform)
- Fixtures: hierarchical graph actor YAML + skill pack with tool refs and
  inline tools
- Docs: updated testing.md with M2 smoke suite section

Closes #169
2026-02-24 21:55:24 +00:00
brent.edwards a181b774c9 Merge branch 'feature/m6-perf-scale' into develop-brent-2
# Conflicts:
#	docs/development/testing.md
2026-02-20 16:31:01 +00:00
brent.edwards 0dfdc02564 test(perf): add scale test fixtures 2026-02-20 03:31:43 +00:00
brent.edwards 558bcaf586 test(validation): add edge case suites 2026-02-20 02:54:08 +00:00
freemo 3cf6a812ae feat: ported in run behavior as a subcommand of actor 2026-02-04 14:43:08 -05:00