Add TDD regression tests for bug #570 where `_get_session_service()`
calls `container.db()` but the DI `Container` class has no `db`
provider, raising `AttributeError`. Same root cause as bug #554.
Includes 4 Behave BDD scenarios tagged `@tdd_bug @tdd_bug_570
@tdd_expected_fail`, Robot Framework integration smoke tests with
`--format plain`, and ASV service-layer benchmarks. Tests exercise the
real DI path by resetting `_service = None` and using a file-based
SQLite database.
Implements the `@tdd_expected_fail` inversion infrastructure:
- Behave: `after_scenario` hook in `features/environment.py` inverts
pass/fail for scenarios tagged `@tdd_expected_fail`
- Robot: `robot/tdd_expected_fail_listener.py` listener (API v3)
performs the same inversion for Robot test cases
- `noxfile.py`: registers the listener via `--listener` in both the
`integration_tests` and `slow_integration_tests` sessions
Migrates 18 existing TDD scenarios across 5 feature files from the old
`@tdd @bugNNN` convention to the standardised `@tdd_bug @tdd_bug_NNN`
tags per CONTRIBUTING.md § TDD Bug Test Tags.
Refs: #570
Add 10 Behave BDD scenarios (@tdd_bug @tdd_bug_554 @tdd_expected_fail)
for the session list DI wiring bug where _get_session_service() calls
container.db() but the Container has no db provider (AttributeError).
Scenarios cover empty list, format validation (JSON/YAML/plain/rich),
init-then-list lifecycle, and post-create list paths.
Implement @tdd_expected_fail infrastructure: Behave after_scenario hook
inverts FAIL→PASS (expected) and PASS→FAIL (unexpected fix), plus Robot
Framework listener (Listener API v3) with identical semantics registered
via --listener in both integration_tests and slow_integration_tests nox
sessions.
Migrate 18 existing TDD scenarios across 5 feature files from legacy
@tdd @bugNNN convention to @tdd_bug @tdd_bug_NNN per CONTRIBUTING.md
§ TDD Bug Test Tags.
Includes Robot Framework integration smoke tests and ASV service-layer
benchmarks.
Refs: #554
Implement TDD bug-capture tests for bug #570 where `agents session create`
fails because `_get_session_service()` calls `container.db()` which does
not exist on the DI Container class (AttributeError). Same root cause as
bug #554.
Behave BDD scenarios tagged @tdd_bug @tdd_bug_570 @tdd_expected_fail
exercise the real DI path (no mocks). Includes Robot Framework integration
smoke tests with self-inverting helper and ASV benchmark baseline.
ISSUES CLOSED: #631
Implement TDD bug-capture tests for bug #554 where `agents session list`
fails because `_get_session_service()` calls `container.db()` which does
not exist on the DI Container class (AttributeError).
Behave BDD scenarios tagged @tdd_bug @tdd_bug_554 @tdd_expected_fail
exercise the real DI path (no mocks) and assert correct behavior. The
@tdd_expected_fail handler in environment.py inverts failed→passed while
the bug is present, keeping CI green.
Also adds:
- @tdd_expected_fail infrastructure in features/environment.py
(tag validation + status inversion in after_scenario hook)
- behave-parallel exit logic fix to use summary-based failure
detection (compatible with TDD status inversion)
- Robot Framework integration smoke tests with self-inverting helper
- ASV benchmark baseline for session list command throughput
ISSUES CLOSED: #630
Three-pronged fix for intermittent pabot-parallel race condition in
M4 validation integration tests:
1. Composable Setup Database Isolation keyword in common.resource
gives each suite a unique CLEVERAGENTS_DATABASE_URL so concurrent
pabot workers never contend on the same SQLite file.
2. Per-suite CLEVERAGENTS_HOME directories prevent shared temp
directory cleanup from racing between workers.
3. Centralised reset_global_state() in robot/helpers_common.py clears
Settings singleton, DI container, provider registry, and engine
cache between chained CLI invocations in helper processes.
Also:
- Setup Test Environment now accepts optional mock_ai and
auto_apply_migrations arguments (default TRUE) for backward
compatibility while allowing suites to opt out.
- Added Suite Teardown to cli_plan_context_commands.robot.
- Fixed _COMMANDS typing in two helpers to eliminate type: ignore.
- Updated docs/development/testing.md to reflect helpers_common
delegation pattern.
- Added timeout=30s to all Run Process calls in
m4_e2e_verification.robot.
Fixes: #563
Implement the structured metrics collection framework covering 14
operational metric types with proper Histogram, Counter, and Gauge
semantics per the spec (Architecture > Observability > Metrics
Collection, lines ~43805-43825).
Domain layer:
- Add MetricType enum (HISTOGRAM, COUNTER, GAUGE) to metrics.py
- Add MetricDefinition model and METRIC_DEFINITIONS registry mapping
all 14 OperationalMetricKey values to their metric types
- Extend MetricCollector with typed factory methods (histogram, counter,
gauge) and 14 convenience methods (plan_duration, plan_cost,
plan_decision_count, subplan_count, actor_invocation_count,
actor_latency, tool_invocation_count, tool_error_rate,
context_build_time, context_token_count, llm_call_count,
llm_total_tokens, llm_total_cost, llm_avg_latency)
- Extend MetricEntry with optional metric_type field that auto-resolves
from METRIC_DEFINITIONS via MetricCollector.record()
Infrastructure layer:
- Add MetricsEmitter (infrastructure/observability/metrics_emitter.py)
with emit(), emit_batch(), from_settings(), and enabled/disabled
support for structured log emission in local mode
- Add metrics_log_processor (config/metrics_processor.py) for structlog
integration
Configuration:
- Add metrics_enabled and metrics_export_prometheus settings
- Register MetricsEmitter as DI Singleton in application container
Instrumentation:
- Add best-effort metric emission in PlanExecutor for plan_duration
(both runtime and stub execute paths) and plan_decision_count
(strategize path) via _try_emit_metric helper that tolerates invalid
plan IDs in test fixtures
Testing:
- 34 Behave BDD scenarios (features/observability/metrics_collection.feature)
- 8 Robot Framework integration tests (robot/metrics_collection.robot)
- ASV benchmark suite (benchmarks/bench_metrics_collection.py)
Closes#579
Implement StrategyCoordinator and FusionEngine as named facades over
the existing ACMS pipeline components, providing clean public APIs
for parallel strategy execution with proportional budget allocation
and fragment fusion with dedup/conflict resolution/knapsack packing.
Key changes:
- Add StrategyCoordinator with parallel execution and confidence-based budget allocation
- Add FusionEngine with URI+hash dedup, max-depth conflict resolution, greedy knapsack packing
- Add budget overage guard with lowest-relevance fragment dropping
- Add per-strategy max caps enforcement
- Wire into existing ContextAssemblyPipeline
- Add Behave BDD tests, Robot integration tests, ASV benchmarks
- Add docs/reference/acms_fusion.md
ISSUES CLOSED: #192
Implemented lazy container activation for devcontainer-instance resources
with ContainerLifecycleState enum tracking six states (inactive, starting,
active, stopping, stopped, error) with validated transitions. Extended
DevcontainerHandler with devcontainer up CLI integration and JSON output
parsing for container start. Added periodic health checking via
devcontainer exec ping with configurable interval. Added agents resource
stop and agents resource rebuild CLI commands for manual lifecycle
control. Wired session close and plan completion hooks to automatic
container cleanup. Includes lifecycle state persistence in resource
registry with timestamped transitions. Added Behave BDD tests, Robot
integration tests, and ASV activation latency benchmarks.
- Added remoteWorkspaceFolder absolute-path validation
- Aligned spec: handler name, rebuild types, --yes flag on stop/rebuild
- Added registry re-read in stop_container success path for consistency
- Added session_id field to ContainerLifecycleTracker for scoped cleanup
- Scoped stop_all_active_containers to session_id when provided
- Wired _cleanup_devcontainers into fail_apply and fail_execute
- Wired start_health_check into activate_container success path
- Restructured facade session close to always run container cleanup
even without session service (F4)
- Re-read tracker from registry in activate_container success path
- Added evict_terminal_trackers to cap registry growth
- Updated devcontainer_resources.md: health check auto-start, scoped
cleanup hooks, known limitations for eviction and sandbox_strategy
- Wired evict_terminal_trackers into stop_all_active_containers so
terminal-state trackers are actually evicted in production
- Made stop_container idempotent: returns early when container is
already in a terminal state instead of raising ValueError
- Fixed benchmark health check thread leak in TimeActivationLatency
by clearing registry after each timing loop
- Added rebuild pass-through (--reset-container flag to devcontainer up)
- Added host_workspace_path field on ContainerLifecycleTracker so
health probes use the host-side path for devcontainer exec
- Wired lazy activation into DevcontainerHandler.resolve() for
devcontainer-instance resources in non-running states
- Changed _default_strategy from SNAPSHOT to NONE (container
itself provides isolation; SandboxFactory raises NotImplementedError
for snapshot)
- Restricted _STOPPABLE_TYPES to devcontainer-instance only
(container-instance is not directly stoppable via CLI)
ISSUES CLOSED: #514
m6_004_resource_type_inherits and m7_001_repo_indexing_tables both had
down_revision='m6_003_async_jobs_table', creating two Alembic heads.
Chain m6_004 after m7_001 so the migration graph is linear.
Complete F24 fix: capabilities, auto_discovery, and equivalence dicts
were returned as direct references. Now wrapped with dict() to
prevent callers from mutating internal state.
Extract data helpers (_resource_registry_data.py, 389 lines) and
operation mixins (_resource_registry_ops.py, 497 lines) from the
1247-line resource_registry_service.py, reducing it to 467 lines.
All three modules stay under the 500-line CONTRIBUTING limit.
Backward-compat aliases (_spec_to_db, _db_to_spec) are re-exported
from the main module so existing test imports continue to work.
Addresses review finding F8 for PR #618.
Add RepoIndexingService with incremental refresh, language detection,
SHA-256 hashing, and policy enforcement. Includes domain models,
DB persistence, DI wiring, Behave/Robot/ASV tests, and reference docs.
ISSUES CLOSED: #195
## Summary
Fix `agents project show` not finding a project immediately after creation. Extends the `session.commit()` fix from #589 to also cover `update()` and `delete()` in `NamespacedProjectRepository`.
## Changes
**Production fix** (`src/cleveragents/infrastructure/database/repositories.py`):
- Add `session.commit()` to `create()`, `update()`, and `delete()` methods
- Add `finally: session.close()` guard to all three methods
- Update class docstring to reflect commit-per-method pattern
**Tests & benchmarks**:
- 3 Behave BDD regression scenarios (`features/project_show_after_create.feature`)
- Robot Framework integration smoke tests with "not found" assertion (`robot/project_show_after_create.robot`)
- ASV benchmarks for create-then-show round-trip (`benchmarks/project_show_after_create_bench.py`)
## Review feedback addressed
- **F1**: Removed unrelated em-dash CHANGELOG edits — wrote clean entry from scratch
- **F2**: Kept Suite Setup/Teardown (required for `${PYTHON}` variable); updated stale docs
- **F3**: Added "not found" string assertion to Robot negative test case
- **F4**: Removed redundant `Base.metadata.create_all()` from `_make_fresh_repo()` helper
- Updated all stale TDD "expected to fail" comments — this PR includes the fix
## Process
- Single squashed commit, rebased onto `master` (no merge commits)
- Prescribed commit message from issue #590 metadata
ISSUES CLOSED: #590
Reviewed-on: cleveragents/cleveragents-core#593
Reviewed-by: Rui Hu <rui.hu@cleverthis.com>
Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
## Summary
Fix `agents project create` not persisting projects to the database. `NamespacedProjectRepository.create()` called `session.flush()` but never `session.commit()`, so projects were invisible to subsequent `agents project list` invocations that open a separate session.
## Changes
**Production fix** (`src/cleveragents/infrastructure/database/repositories.py`):
- Replace `session.flush()` with `session.commit()` in `NamespacedProjectRepository.create()`
- Add `finally: session.close()` guard for proper session lifecycle
**Tests & benchmarks**:
- 4 Behave BDD regression scenarios (`features/project_create_persist.feature`)
- Robot Framework integration smoke tests (`robot/project_create_persist.robot`)
- ASV benchmarks for create-then-list round-trip (`benchmarks/project_create_persist_bench.py`)
## Review feedback addressed
- **H3**: Added `finally: session.close()` to `create()` method
- **M1**: Removed redundant `session.flush()` before `session.commit()`
- **M2**: Updated stale TDD "expected to fail" comments — this PR includes the fix
- **M4**: Rewrote CHANGELOG entry to describe the fix, not just tests
- **F1**: Updated Robot documentation (Suite Setup/Teardown kept — needed for `${PYTHON}`)
- **F2**: Fixed bare assertion `"my-app"` to `"local/my-app"` in namespace scenario
- **F3**: Removed redundant `Base.metadata.create_all()` from `_make_fresh_repo()`
## Process
- Single squashed commit, rebased onto `master` (no merge commits)
- Prescribed commit message from issue #589 metadata
ISSUES CLOSED: #589
Reviewed-on: cleveragents/cleveragents-core#591
Reviewed-by: Rui Hu <rui.hu@cleverthis.com>
Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
## Summary
Fixes the `plan correct` CLI handler which never passes the decision tree or influence edges to `CorrectionService`, producing single-node impact analysis regardless of the plan's actual decision structure.
Closes#606
## Root Cause
The `correct_decision()` handler in `plan.py` created a bare `CorrectionService()` and called `analyze_impact()` and `execute_correction()` without passing `decision_tree` or `influence_edges`. Both parameters default to `None` -> empty dicts, causing `_compute_affected_subtree()` BFS to return only the single target decision, ignoring all descendants and influence-DAG dependents.
## Changes
### Production Fix (`src/cleveragents/cli/commands/plan.py`)
- Resolve `DecisionService` via `get_container()` (following the pattern used by `plan explain` and `plan tree`)
- Build structural tree adjacency list from `decision_svc.list_decisions(plan_id)` using `parent_decision_id` relationships
- Fetch influence edges from `decision_svc.get_influence_edges(plan_id)`
- Pass both `decision_tree` and `influence_edges` to `svc.analyze_impact()` and `svc.execute_correction()`
### Existing Test Fixups
- Updated 4 step definition files that mock `CorrectionService` to also mock the new `DecisionService` resolution path
### New Tests
- **Behave BDD**: 3 scenarios verifying tree/edge forwarding (dry-run subtree, execution subtree, leaf node)
- **Robot Framework**: 3 integration smoke tests
- **ASV Benchmark**: Tree building and analyze_impact overhead benchmarks
## Quality Gates
- `nox -s lint` — PASSED
- `nox -s typecheck` — 0 errors
- `nox -s unit_tests` — 9,109 scenarios, 0 failures
- `nox -s coverage_report` — 97%
ISSUES CLOSED: #606
Reviewed-on: cleveragents/cleveragents-core#639
Co-authored-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com>
Co-committed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com>
Implement spec-mandated pre-flight guardrail checks that validate plan
readiness before entering the Strategize phase:
1. Action schema validation — verifies action exists and is well-formed.
2. Actor availability — confirms all 4 actor roles registered.
3. Skill/tool existence — transitively resolves all tools/skills.
4. Automation policy — verifies profile permits execution.
5. Rollback feasibility — ensures all tools checkpointable if required.
6. Resource accessibility — shallow connectivity check on linked resources.
7. Validation attachment resolution — pre-resolves all applicable validations.
PlanPreflightGuardrail service runs all 7 checks via run_all_checks().
On first failure, raises PreflightRejection with check name and message.
Wired into plan_lifecycle_service before Strategize phase.
Behave BDD: 18 scenarios covering all 7 checks (positive + negative).
Robot Framework: 3 integration smoke tests.
ASV benchmarks: pre-flight check execution time.
ISSUES CLOSED: #582
ToolRegistryRepository.create(), .update(), and .delete() called
session.flush() but never session.commit(). The CLI factory creates a
raw sessionmaker without a UnitOfWork wrapper, so the transaction was
never committed and SQLAlchemy performed an implicit rollback when the
session was garbage-collected.
The same bug existed in ValidationAttachmentRepository.attach() and
.detach().
Changes:
- Add session.commit() after session.flush() in all five mutating
methods across ToolRegistryRepository and
ValidationAttachmentRepository.
- Add finally: session.close() to guarantee session cleanup regardless
of success or failure.
- Update class docstrings to reflect the new commit-on-write semantics.
- Add Behave BDD feature (tool_add_persist.feature) with scenarios for
single-tool round-trip, multi-tool persistence, and duplicate
rejection, using file-based SQLite to reproduce the cross-session
issue.
- Add Robot Framework integration test (tool_add_persist.robot) with
add-then-list and fresh-list-empty scenarios.
- Add ASV benchmark (tool_add_persist_bench.py) with
track_list_after_add_count metric.
Key decisions:
- File-based SQLite (not in-memory) is used in tests because the bug
only manifests when the session/engine is fully disposed between add
and list, simulating separate CLI invocations.
- Step patterns are prefixed with "tool-persist" to avoid AmbiguousStep
collisions with existing tool_registry_steps.py.
- The commit-in-repository approach was chosen over adding a UnitOfWork
to the CLI factory because the CLI commands are simple CRUD operations
that should auto-persist without requiring callers to remember to
commit.
ISSUES CLOSED: #621
Wire SkillService to use SkillRepository for database persistence,
fixing the bug where `agents skill add` stored skills only in an
in-memory OrderedDict that was lost when the CLI process exited.
Changes:
- SkillService now accepts optional skill_repo and session_factory
parameters. When provided, add_skill() persists to the database
and the constructor pre-loads existing skills from DB rows.
- _get_skill_service() in the CLI now creates a DB-backed service
following the same engine/sessionmaker/repository pattern used by
the tool CLI (tool.py).
- _reset_skill_service() now installs a fresh in-memory SkillService
(instead of setting None) to avoid DB side-effects during unit
testing with parallel runners.
- remove_skill() also persists the deletion to the database.
Test coverage:
- Behave BDD: features/skill_add_persist.feature (4 scenarios)
- Robot Framework: robot/skill_add_persist.robot (3 smoke tests)
- ASV benchmark: benchmarks/skill_add_persist_bench.py
ISSUES CLOSED: #620
Add a call to bootstrap_builtin_types() in init_command() (project.py)
immediately after initialize_project() returns. This seeds the built-in
resource types (fs-directory, git-checkout, etc.) into the database so
that "resource add" commands succeed without "Resource type not found"
errors.
The call is idempotent — invoking it multiple times will not create
duplicate types.
Also fix the TDD robot test (resource_type_bootstrap_git.robot) to
initialize a project before running "resource add", and fix a
pre-existing parallel test failure in plan_commands_new_coverage where
unittest.mock.patch could not reliably intercept PlanApplyService under
behave-parallel fork() workers.
ISSUES CLOSED: #523, #524
Add TDD-style failing tests that verify the built-in git-checkout resource
type is available after initialization. Tests assert the correct expected
behavior: after agents init, the git-checkout type should exist in the
registry and 'agents resource add git-checkout' should succeed.
Tests are expected to fail until bug #524 is fixed, because
bootstrap_builtin_types() is never called during initialization. The fix
branch should be based on this branch so the fix commit inherits these tests.
Files added:
- features/resource_type_bootstrap_git.feature (2 Behave scenarios)
- features/steps/resource_type_bootstrap_git_steps.py (step definitions)
- robot/resource_type_bootstrap_git.robot (Robot Framework smoke test)
ISSUES CLOSED: #553
Added --yes/-y flag to the agents init CLI command to support non-interactive
initialization as specified in the documentation. When --yes is passed, all
interactive prompts are skipped and default values are applied. The command
produces the expected initialization summary output including data directory,
config file, database, and created directories.
The flag was added to both the top-level "agents init" command in main.py and
the "agents project init" subcommand in project.py. Both delegate to the shared
init_command() function which now accepts a yes parameter and produces
spec-aligned Rich panel output (Data Dir, Config, Database, Directories) with
the "Initialized (non-interactive)" status message when yes=True. When
yes=False (default), existing interactive behavior is preserved unchanged.
ISSUES CLOSED: #522
Implemented the remaining ACMS pipeline components and advanced context
strategies:
Pipeline Phase 3:
- FragmentOrdererProtocol + RelevanceCoherenceOrderer: orders fragments
by relevance while maintaining narrative coherence via UKO node prefix
grouping. Groups related fragments together, sorts groups by max
relevance, and within groups orders by relevance desc / depth asc.
- PreambleGeneratorProtocol + ProvenancePreambleGenerator: generates
provenance preamble with strategy contributions (fragment counts and
token percentages), confidence indicators (avg/min/max), tier and
depth distribution, UKO node coverage, and coverage gap detection.
Advanced Strategies:
- ArceStrategy (quality 0.95): adaptive recursive context expansion with
iterative multi-backend refinement and configurable iteration limit
(default 5) to prevent unbounded refinement. Uses composite scoring
(relevance + depth + diversity) with contextual boosting for fragments
related to the current top-ranked anchor set.
- TemporalArchaeologyStrategy (quality 0.5): historical context retrieval
from graph+cold backends. Prioritises cold-tier fragments using a
temporal scoring model (tier bonus + relevance + depth).
- PlanDecisionContextStrategy (quality 0.7): decision history retrieval
from warm/cold backends. Prioritises warm then cold tier fragments
for correction and retry scenarios.
All strategies registered in strategy registry with correct quality scores
and backend requirements. All components implement their respective
Protocol interfaces and can be injected into the ContextAssemblyPipeline
via constructor dependency injection.
Tests:
- 33 BDD scenarios in features/acms_pipeline_phase3.feature
- Robot Framework integration tests in robot/acms_pipeline_phase3.robot
- ASV performance benchmarks in benchmarks/acms_pipeline_phase3_bench.py
ISSUES CLOSED: #545
The "Resource Add Git Checkout Should Not Fail With Type Not Found"
Robot test was calling the CLI process directly, which hits the real
database file that does not exist in CI — causing
sqlite3.OperationalError. Rewrite the test to use an in-memory
SQLite database with register_resource(), matching the pattern used
by the fs-directory Robot tests.
Also fix common.resource path to use ${CURDIR}/common.resource.
Refs: #524
Brent E. Edwards
brent.edwards
freemo
CoreRasurae
hamza.khyari
