12 Commits

Author	SHA1	Message	Date
freemo	b122ec7ed5	fix(test-infra): remove redundant ${PYTHON} variable definitions from robot files ... Remove the local ${PYTHON} python (and python3) variable definitions from the *** Variables *** sections of all affected robot files. These local definitions were overriding the pabot-injected venv Python path passed via --variable PYTHON:/path/to/venv/python, causing tests to use the system Python (which lacks required packages like structlog, sqlalchemy, etc.) instead of the nox venv Python. The correct ${PYTHON} value is already set by Setup Test Environment in common.resource via sys.executable, and pabot passes it via --variable. The local fallback definitions are redundant and harmful in parallel runs. Audit found 56 robot files with the pattern (more than the 9 originally identified in the issue). All occurrences have been removed. ISSUES CLOSED: #1309	2026-04-14 14:50:55 +00:00
freemo	4d31f0ed02	fix(cli): promote --format to global CLI callback option per spec ... Add `fmt: OutputFormat` parameter to `main_callback()` in `src/cleveragents/cli/main.py` and store the selected format in `ctx.obj["format"]` so all subcommands can read it without needing their own per-command `--format` flag. Remove per-command `--format` / `fmt` parameters from `version()`, `info()`, and `diagnostics()` commands. These commands now read the format from `ctx.obj.get("format", OutputFormat.RICH.value)`. The specification states: "The framework supports six distinct output formats, selectable via the global `--format` flag." This change aligns the implementation with the spec by making `--format` a global option on the root `agents` command (via the Typer callback). All six formats (json, yaml, plain, rich, table, color) are supported via the global flag and the `-f` shorthand. Add Behave BDD scenarios covering global `--format` flag propagation to subcommands for all six formats. Update Robot Framework integration tests to exercise the global `--format` flag. Update existing tests that used per-command `--format` for version/info/diagnostics to use the global flag instead. ISSUES CLOSED: #2908	2026-04-05 08:39:14 +00:00
brent.edwards	01b6eb1804	feat(autonomy): parallel execution scales to 10+ concurrent subplans (#1201) ... ## Summary Add M6 parallel-scaling coverage for 10+ concurrent subplans: - **15-subplan parallel scenario** with explicit peak-concurrency bound checks (`max_parallel=10`) and thread-safe concurrency tracking via `_build_executor()`. - **Deep hierarchical decomposition** coverage (4+ levels) with adjusted leaf condition that only stops early when hitting `max_depth` or when the workset is trivially small (`min_files_per_subplan`). - **Non-progress guard** in `_build_hierarchy` to prevent pathological recursion when clustering cannot meaningfully split the file set. - **Small-project regression test** (< 50 files) verifying decomposition depth does not increase unexpectedly with the relaxed leaf condition. - **ASV benchmark** for 15-subplan parallel execution with `max_parallel=10` to track scaling behavior. ### Removed from this PR The `_build_hierarchy` child-linkage correctness fix (returning `node_id` from recursive calls instead of using `nodes[-1].node_id`) has been **removed** per review feedback — it is a separate bug fix and will be submitted as an independent issue/PR per CONTRIBUTING.md §Atomic Commits. ## Approach - **Concurrency tracking:** The `_build_executor()` closure in step definitions detects `context.concurrency_counter` / `context.concurrency_lock` and performs thread-safe peak tracking in a try/finally block. - **Leaf condition:** Replaced the `max_files_per_subplan` / `max_tokens_per_subplan` leaf check with a `min_files_per_subplan` check to allow deeper decomposition for large projects. Added a non-progress guard so clustering that cannot split the file set terminates immediately rather than recursing to `max_depth`. - **Deterministic IDs:** `_ids_for_count()` preserves legacy fixed IDs for the first 5 subplans and generates additional deterministic IDs for scale scenarios. ## Validation ### Passing - `nox -s lint` — all checks passed - `nox -s typecheck` — 0 errors, 0 warnings - `nox -s unit_tests` — 12,988 scenarios passed, 0 failed - `nox -s coverage_report` — 97% (passes `--fail-under=97`) Closes #855 Reviewed-on: #1201 Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com> Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>	2026-03-31 23:57:39 +00:00
hamza.khyari	3f66221938	test(integration): workflow example 5 — database schema migration with safety nets (review profile) ... Robot Framework integration test suite for Specification Workflow Example 5: Database Schema Migration with Safety Nets. Exercises the review automation profile with custom resource types, custom skills with database tools, phased child plan execution, checkpointing, and rollback. - 7 test cases covering: custom resource type registration, review profile behavior, custom skill with 3 database tools, action creation with review profile, checkpoint/rollback via managers, 5-phase sequential SubplanService.spawn with fail-fast, and plan lifecycle through strategize-to-execute - Uses mocked LLM providers (CLEVERAGENTS_TESTING_USE_MOCK_AI=true) - All Run Process calls have timeout=60s on_timeout=kill - Fix environment-dependent test failures: mock shutil.disk_usage in disk space and diagnostics --check Behave steps so tests pass regardless of CI runner disk space; Robot diagnostics test tolerates disk-space errors ISSUES CLOSED: #769	2026-03-31 11:03:20 +00:00
brent.edwards	4f950a1600	feat(cli): repo indexing CLI functional (#982) ... ## Summary Add `agents repo index` and `agents repo status` CLI commands, wiring the existing `RepoIndexingService` backend to the CLI layer. ### Commands - **`agents repo index <resource_name>`** — Triggers full or incremental indexing of a repository resource. Options: `--full` (force full re-index), `--format text|json` - **`agents repo status <resource_name>`** — Displays indexing metadata: status, file count, token estimate, primary language, last indexed timestamp. Options: `--format text|json` ### Implementation - New CLI module `src/cleveragents/cli/commands/repo.py` (238 lines) - Registered as `repo` subcommand group in `cli/main.py` - Resolves resource by name via `ResourceRegistryService` - Calls `RepoIndexingService.index_resource()` / `refresh_index()` / `get_index_status()` - Both text (Rich panel) and JSON output formats ### Quality Gates | Session | Result | |---|---| | `nox -s lint` | PASS | | `nox -s typecheck` | PASS (0 errors) | | `nox -s unit_tests` | PASS (10,819 scenarios) | | `nox -s coverage_report` | 98% (>= 97%) | | Integration tests | 5/5 PASS | Closes #856 Reviewed-on: #982 Co-authored-by: Brent Edwards <brent.edwards@cleverthis.com> Co-committed-by: Brent Edwards <brent.edwards@cleverthis.com>	2026-03-20 22:28:32 +00:00
brent.edwards	3eecb79003	test(plan): TDD failing tests for checkpoint real rollback (bug #822) ... TDD expected-fail tests proving bug #822 exists: CheckpointService.rollback_to_checkpoint() returns a successful RollbackResult but does not execute git reset --hard. Files modified after the checkpoint remain unchanged after rollback. Also fixes Robot Framework timeout robustness across the entire test suite: all Run Process calls now use on_timeout=kill (prevents SIGTERM-induced -15 exit codes under CI load) and timeouts increased to 120s (prevents premature kills during heavy parallel execution). ISSUES CLOSED: #839	2026-03-16 01:45:50 +00:00
freemo	77a7813f0e	fix(test): tighten diagnostics check exit code assertion in CLI core test ... Replace the tolerant "rc == 0 or rc == 1" assertion with a strict "rc == 0" check for the diagnostics --check command. In a clean test environment, diagnostics should complete without finding errors, and the previous assertion could not distinguish a legitimate diagnostic failure from a command crash.	2026-03-14 00:37:34 +00:00
freemo	3f14cbbf7e	feat(observability): add LLMTrace model and operational metrics ... Add LLMTrace Pydantic v2 domain model with all required fields (trace_id, plan_id, decision_id, actor, provider, model, prompt_tokens, completion_tokens, cost_usd, latency_ms, tool_calls, context_hash, streaming, retry_count, error). Define 14 OperationalMetricKey values (PLAN_DURATION_MS, PLAN_TOTAL_COST_USD, PLAN_DECISION_COUNT, ACTOR_INVOCATION_COUNT, ACTOR_LATENCY_MS, TOOL_INVOCATION_COUNT, TOOL_ERROR_RATE, CONTEXT_BUILD_TIME_MS, CONTEXT_TOKEN_COUNT, LLM_CALL_COUNT, LLM_TOTAL_TOKENS, LLM_TOTAL_COST_USD, LLM_AVG_LATENCY_MS, SUBPLAN_COUNT) with MetricEntry model and MetricCollector. Add llm_traces database table (LLMTraceModel) with LLMTraceRepository for persistence. TraceService provides recording, querying, metric computation, plan lifecycle hooks, and optional LangSmith forwarding when LANGCHAIN_TRACING_V2=true. Wired into DI container as trace_service. Includes 28 Behave BDD scenarios, 6 Robot Framework smoke tests, 3 ASV benchmark suites, and reference documentation. Fix pre-existing cli_core server_mode test flake by mocking resolve_server_mode in test steps to avoid stale config file interference. Closes #500	2026-03-03 17:04:17 -05:00
freemo	738c3b5eda	feat(plan): add multi-project subplan support ... Add domain models, application service, CLI integration, and full test coverage for multi-project subplan orchestration. New components: - MultiProjectMetadata, ProjectScope, ProjectDependency domain models - MultiProjectService for creating/managing multi-project plans - CLI display of multi-project metadata in plan commands - Behave BDD tests (20 scenarios), Robot Framework integration tests, ASV benchmarks, and reference documentation Also fixes pre-existing test flakiness in cli_core, core_cli_commands, cli_plan_context_commands, and helper_server_stubs caused by environment leakage and path-with-spaces issues. ISSUES CLOSED: #199	2026-03-03 14:55:58 -05:00
brent.edwards	e4705822b1	fix(test): increase subprocess timeouts in cli.robot and cli_core.robot ... - Add timeout=60s to all Run Process calls in cli.robot (previously had no timeout, causing hangs under CI parallel load) - Increase cli_core.robot timeouts from 30s to 60s to match other robot files - Relax performance assertions from <30s to <60s to account for Python cold-start overhead under pabot parallel execution	2026-02-19 01:35:09 +00:00
khyari hamza	12845d3a3c	fix(test): make diagnostics --check robot test CI-safe ... The test now accepts both exit code 0 (no errors) and 1 (has errors) since CI environments may have low disk space or missing tools that trigger error-level diagnostic checks.	2026-02-18 15:28:48 +00:00
khyari hamza	02c75fb4bb	feat(cli): enhance version/info/diagnostics with --format and --check ... Add structured data builders and Rich renderers for version, info, and diagnostics commands. Support --format rich/plain/json/yaml output parity and --check flag for diagnostics (exits non-zero on errors). Includes Behave scenarios, Robot smoke tests, ASV benchmarks, and updates existing tests to match the new Rich panel output format.	2026-02-18 12:18:03 +00:00