Implement spec-mandated pre-flight guardrail checks that validate plan
readiness before entering the Strategize phase:
1. Action schema validation — verifies action exists and is well-formed.
2. Actor availability — confirms all 4 actor roles registered.
3. Skill/tool existence — transitively resolves all tools/skills.
4. Automation policy — verifies profile permits execution.
5. Rollback feasibility — ensures all tools checkpointable if required.
6. Resource accessibility — shallow connectivity check on linked resources.
7. Validation attachment resolution — pre-resolves all applicable validations.
PlanPreflightGuardrail service runs all 7 checks via run_all_checks().
On first failure, raises PreflightRejection with check name and message.
Wired into plan_lifecycle_service before Strategize phase.
Behave BDD: 18 scenarios covering all 7 checks (positive + negative).
Robot Framework: 3 integration smoke tests.
ASV benchmarks: pre-flight check execution time.
ISSUES CLOSED: #582
ToolRegistryRepository.create(), .update(), and .delete() called
session.flush() but never session.commit(). The CLI factory creates a
raw sessionmaker without a UnitOfWork wrapper, so the transaction was
never committed and SQLAlchemy performed an implicit rollback when the
session was garbage-collected.
The same bug existed in ValidationAttachmentRepository.attach() and
.detach().
Changes:
- Add session.commit() after session.flush() in all five mutating
methods across ToolRegistryRepository and
ValidationAttachmentRepository.
- Add finally: session.close() to guarantee session cleanup regardless
of success or failure.
- Update class docstrings to reflect the new commit-on-write semantics.
- Add Behave BDD feature (tool_add_persist.feature) with scenarios for
single-tool round-trip, multi-tool persistence, and duplicate
rejection, using file-based SQLite to reproduce the cross-session
issue.
- Add Robot Framework integration test (tool_add_persist.robot) with
add-then-list and fresh-list-empty scenarios.
- Add ASV benchmark (tool_add_persist_bench.py) with
track_list_after_add_count metric.
Key decisions:
- File-based SQLite (not in-memory) is used in tests because the bug
only manifests when the session/engine is fully disposed between add
and list, simulating separate CLI invocations.
- Step patterns are prefixed with "tool-persist" to avoid AmbiguousStep
collisions with existing tool_registry_steps.py.
- The commit-in-repository approach was chosen over adding a UnitOfWork
to the CLI factory because the CLI commands are simple CRUD operations
that should auto-persist without requiring callers to remember to
commit.
ISSUES CLOSED: #621
Wire SkillService to use SkillRepository for database persistence,
fixing the bug where `agents skill add` stored skills only in an
in-memory OrderedDict that was lost when the CLI process exited.
Changes:
- SkillService now accepts optional skill_repo and session_factory
parameters. When provided, add_skill() persists to the database
and the constructor pre-loads existing skills from DB rows.
- _get_skill_service() in the CLI now creates a DB-backed service
following the same engine/sessionmaker/repository pattern used by
the tool CLI (tool.py).
- _reset_skill_service() now installs a fresh in-memory SkillService
(instead of setting None) to avoid DB side-effects during unit
testing with parallel runners.
- remove_skill() also persists the deletion to the database.
Test coverage:
- Behave BDD: features/skill_add_persist.feature (4 scenarios)
- Robot Framework: robot/skill_add_persist.robot (3 smoke tests)
- ASV benchmark: benchmarks/skill_add_persist_bench.py
ISSUES CLOSED: #620
Add a call to bootstrap_builtin_types() in init_command() (project.py)
immediately after initialize_project() returns. This seeds the built-in
resource types (fs-directory, git-checkout, etc.) into the database so
that "resource add" commands succeed without "Resource type not found"
errors.
The call is idempotent — invoking it multiple times will not create
duplicate types.
Also fix the TDD robot test (resource_type_bootstrap_git.robot) to
initialize a project before running "resource add", and fix a
pre-existing parallel test failure in plan_commands_new_coverage where
unittest.mock.patch could not reliably intercept PlanApplyService under
behave-parallel fork() workers.
ISSUES CLOSED: #523, #524
Add TDD-style failing tests that verify the built-in git-checkout resource
type is available after initialization. Tests assert the correct expected
behavior: after agents init, the git-checkout type should exist in the
registry and 'agents resource add git-checkout' should succeed.
Tests are expected to fail until bug #524 is fixed, because
bootstrap_builtin_types() is never called during initialization. The fix
branch should be based on this branch so the fix commit inherits these tests.
Files added:
- features/resource_type_bootstrap_git.feature (2 Behave scenarios)
- features/steps/resource_type_bootstrap_git_steps.py (step definitions)
- robot/resource_type_bootstrap_git.robot (Robot Framework smoke test)
ISSUES CLOSED: #553
Implemented the remaining ACMS pipeline components and advanced context
strategies:
Pipeline Phase 3:
- FragmentOrdererProtocol + RelevanceCoherenceOrderer: orders fragments
by relevance while maintaining narrative coherence via UKO node prefix
grouping. Groups related fragments together, sorts groups by max
relevance, and within groups orders by relevance desc / depth asc.
- PreambleGeneratorProtocol + ProvenancePreambleGenerator: generates
provenance preamble with strategy contributions (fragment counts and
token percentages), confidence indicators (avg/min/max), tier and
depth distribution, UKO node coverage, and coverage gap detection.
Advanced Strategies:
- ArceStrategy (quality 0.95): adaptive recursive context expansion with
iterative multi-backend refinement and configurable iteration limit
(default 5) to prevent unbounded refinement. Uses composite scoring
(relevance + depth + diversity) with contextual boosting for fragments
related to the current top-ranked anchor set.
- TemporalArchaeologyStrategy (quality 0.5): historical context retrieval
from graph+cold backends. Prioritises cold-tier fragments using a
temporal scoring model (tier bonus + relevance + depth).
- PlanDecisionContextStrategy (quality 0.7): decision history retrieval
from warm/cold backends. Prioritises warm then cold tier fragments
for correction and retry scenarios.
All strategies registered in strategy registry with correct quality scores
and backend requirements. All components implement their respective
Protocol interfaces and can be injected into the ContextAssemblyPipeline
via constructor dependency injection.
Tests:
- 33 BDD scenarios in features/acms_pipeline_phase3.feature
- Robot Framework integration tests in robot/acms_pipeline_phase3.robot
- ASV performance benchmarks in benchmarks/acms_pipeline_phase3_bench.py
ISSUES CLOSED: #545
Add TDD-style Behave BDD tests for the built-in fs-directory resource type
bootstrap (bug #523). Three Gherkin scenarios: one failing TDD test
reproducing the bug (no bootstrap called during init, tagged @wip), and
two regression tests verifying bootstrap_builtin_types() seeds correct
data and resource add fs-directory succeeds after bootstrap. Includes
Robot Framework regression tests.
Review feedback addressed:
- Removed all 21 unnecessary # type: ignore comments (hurui200320 M1)
- Fixed is not True to is False for clarity (Aditya F2)
- Fixed Robot common.resource path to ${CURDIR}/common.resource (hurui200320 L1)
- Squashed all commits into one and rebased onto master (C1, C2)
- Added CHANGELOG entry with correct scenario count
Closes#537
Added minimal LSP server entrypoint supporting initialize/shutdown/exit
handshake over JSON-RPC stdin/stdout transport with Content-Length
framing. Unsupported methods return MethodNotFound error with descriptive
message. Wired LSP requests through ACP facade in local mode. Added
agents lsp serve CLI command with --log-level flag, PID output, and
startup banner. Created reference documentation for the stub server.
Includes Behave BDD tests for protocol handshake, Robot smoke test, and
ASV startup latency benchmark.
ISSUES CLOSED: #203
Core domain types (FragmentProvenance, ContextFragment, ContextBudget,
ContextPayload) now extend their CRP counterparts via Pydantic v2
inheritance, ensuring isinstance compatibility across the model
hierarchy.
Key changes:
- CRP base types made frozen=True (no consumer mutates them)
- CRP AssembledContext fields changed from list to tuple (frozen consistency)
- Core types extend CRP bases: FragmentProvenance(CRPFragmentProvenance),
ContextFragment(CRPContextFragment), ContextBudget(CRPContextBudget),
ContextPayload(CRPAssembledContext)
- Removed duplicate ContextFragment dataclass from skeleton_compressor
- Updated project_context.py to pass tuples to frozen AssembledContext
- Added Behave tests (10 scenarios), Robot integration tests (3 cases),
and ASV benchmarks for the unified hierarchy
- Updated Known Limitations table in docs/reference/acms.md
ISSUES CLOSED: #569
- SPEC-1: Added genuine TDD failing Scenario 1 (@wip) that creates registry
WITHOUT bootstrap and asserts git-checkout exists — reproduces bug #524.
Existing scenarios retained as regression tests (no @wip since they pass).
Added NOTE FOR FIX AUTHOR comment documenting fix-path expectations.
- BUG-1: Removed colliding @when('I run "agents resource add..."') step.
Replaced with uniquely-prefixed bootstrap-git step pattern that invokes
resource_add() directly with mocked DI, avoiding AmbiguousStep collision
with wildcard @when('I run "{command}"') in cli_plan_context_commands_steps.
- BUG-2: Removed duplicate @then('the CLI exit code should be {code:d}').
Replaced with prefixed bootstrap-git assertion steps.
- BUG-3: Removed duplicate @then('the CLI output should not contain...").
Replaced with prefixed bootstrap-git assertion steps.
- TEST-1: Replaced bare MagicMock() with direct service patching via
_PATCH_SERVICE, consistent with PR #567 pattern.
- TEST-2: Updated Robot docs from 'expected to FAIL' to 'regression tests'
since both Robot tests call bootstrap explicitly and pass.
- CODE-1: Simplified hasattr guards on enum fields — removed redundant
hasattr checks, using .value directly since ResourceKind and
SandboxStrategy are always enums.
- TEST-3: Added assertion on bootstrap_builtin_types() return value via
new Then step 'the bootstrap-git registered types should include'.
- Updated CHANGELOG from 'Two scenarios' to 'Three scenarios'.
Refs: #553
Implemented the analyzer plugin framework with AnalyzerProtocol,
AnalyzerRegistry for registration/discovery by file extension,
PythonAnalyzer (AST-based extraction of modules, classes, functions,
imports, docstrings), and MarkdownAnalyzer (section, code block, and
link extraction). Both analyzers produce well-formed UKO triples with
proper URI schemes.
ISSUES CLOSED: #551
Add the Depth/Breadth Projection System and Skeleton Context
Propagation as specified in docs/specification.md §25265-25340
and §43057-43128:
- ProjectionSpec: frozen Pydantic model capturing a projection
request (focus, breadth, depth, gradient, domain)
- ProjectedNode: frozen model for materialized graph nodes with
resolved depth and distance
- DepthBreadthProjector: stateless BFS projector over UKO graph
adjacency with depth gradient (linear reduction by distance)
- PlanContextInheritance: service computing child plan context
from parent assembled context with skeleton injection
- ChildContextResult: frozen result model with request and skeleton
- InheritanceConfig: frozen config for skeleton_ratio (default 0.2)
- Built-in DetailLevelMap presets for code, docs, and database
Includes 27 Behave BDD scenarios, 9 Robot Framework integration
tests, and ASV benchmarks for all components.
ISSUES CLOSED: #544
Add production-grade Phase 2 (Fragment Fusion) components for the ACMS
context assembly pipeline, replacing the no-op defaults:
- ContentHashDeduplicator: Groups fragments by UKO node URI, hashes
content to detect duplicates, retains highest relevance_score.
- MaxDepthResolver: Resolves depth conflicts by keeping the highest
detail depth per UKO node, with relevance tiebreaking.
- WeightedCompositeScorer: Computes composite score from configurable
weighted factors (relevance=0.4, hierarchy=0.3, quality=0.2,
recency=0.1). Stores component breakdown in metadata.
- GreedyKnapsackPacker: Greedy knapsack selection with depth fallback
(tries depths [9,4,2,0] for oversized fragments) and minimum
fragment token threshold (10).
Also adds:
- ScoredFragment frozen Pydantic model (spec §42825) with
composite_score, score_components, and fragment reference
- score_detailed() method on WeightedCompositeScorer returning
ScoredFragment objects for callers needing full breakdowns
- All components implement v1 Protocol signatures from acms_service.py
and can be DI-injected into ACMSPipeline constructor
Testing:
- 31 Behave BDD scenarios in acms_pipeline_phase2.feature covering
deduplication, depth resolution, scoring, packing, depth fallback,
budget constraints, pipeline integration, and ScoredFragment model
- 6 Robot Framework integration smoke tests
- ASV benchmark suites for all 4 components and ScoredFragment
Quality gates: lint, typecheck (0 errors), unit_tests (8555 scenarios),
coverage (97.0%), dead_code — all passing.
ISSUES CLOSED: #540
Add production-quality Phase 1 pipeline components for the ACMS
Context Assembly Pipeline:
- ConfidenceWeightedSelector: strategy selection with preference
boosting and confidence-based ranking
- ProportionalBudgetAllocator: proportional token budget distribution
with min_useful_budget enforcement and largest-remainder rounding
- ParallelStrategyExecutor: concurrent strategy execution via
ThreadPoolExecutor with per-strategy timeouts and circuit breaking
- CircuitBreaker: per-strategy failure tracking with configurable
threshold and explicit reset
- ContextAssemblyPipeline: extends ACMSPipeline with Phase 1
production components and per-stage timing (StageTimings)
Includes 28 Behave BDD scenarios, 9 Robot Framework integration tests,
and ASV benchmarks for all components.
ISSUES CLOSED: #539
Created ScoredFragment frozen model wrapping ContextFragment with
composite_score, score_breakdown, and rank fields. Added pipeline-
specific fragment models in domain/contexts/ with proper equality
based on uko_uri + detail_depth for deduplication support.
ISSUES CLOSED: #538
Implement the first three built-in context strategies for the ACMS v1
context assembly pipeline:
1. SimpleKeywordStrategy (quality 0.3) - Keyword matching on fragment
content with word-density fallback. Universal fallback strategy.
2. SemanticEmbeddingStrategy (quality 0.6) - Jaccard word-overlap
similarity scoring between query and fragment content.
3. BreadthDepthNavigatorStrategy (quality 0.85) - UKO node hierarchy
navigation prioritising fragments near focus nodes with higher
detail depths. Primary strategy for code projects.
All strategies implement the v1 ContextStrategy Protocol from
acms_service.py and can be registered with ACMSPipeline via
register_strategy().
Includes:
- 28 Behave BDD scenarios covering ranking, budget, capabilities,
can_handle confidence, explain, empty input, and pipeline
registration
- 9 Robot Framework integration tests
- ASV benchmarks at 10/100/1000 fragment scales for all 3 strategies
- Vulture whitelist entries for public API symbols
- 100% coverage on context_strategies.py
ISSUES CLOSED: #541
- Add AsyncJob domain model with status state machine and Pydantic validation
- Add AsyncWorker service with configurable concurrency and job store
- Add CancellationToken, WorkerHealthReport, InMemoryJobStore
- Add AsyncJobModel SQLAlchemy model and Alembic migration (m6_003)
- Add 5 async config keys to Settings (worker_id, concurrency, poll_interval, max_retries, timeout)
- Add _check_async_worker_health diagnostic check in system.py
- Add comprehensive Behave BDD tests (~60 scenarios) with full step definitions
- Add Robot Framework integration tests (6 smoke tests)
- Add ASV benchmark suite for async execution
- Add architecture documentation
- Update vulture_whitelist with new public API symbols
- All quality gates pass: lint, typecheck, unit_tests, integration_tests, coverage_report (97%)
1. Wire async job creation into PlanLifecycleService:
- Add optional job_store parameter to __init__
- Add _maybe_enqueue_async_job() helper that checks settings.async_enabled
and job store presence before creating and enqueuing an AsyncJob
- Call helper from execute_plan() (phase="execute") and apply_plan()
(phase="apply") after phase transitions
- When async is disabled or no job store is configured, behaviour is
unchanged (silent no-op)
2. Redact secrets in failed job error messages:
- Apply shared.redaction.redact_value() to the error string before
persisting to AsyncJob.error_message, preventing accidental secret
leakage (e.g. API keys in exception text) into the audit trail
Documentation:
- S1: Added specification reconciliation note (ADR-style) to
async_architecture.md addressing tension between "No Plan Queuing"
clause and the async subsystem authorised by issue #312
ISSUES CLOSED: #312
- Add @tdd @bug524 tags to both feature scenarios for selective execution
- Move module-level CliRunner singleton to per-step instantiation for
consistency with PR #566 pattern
Refs: #553
Add CrossPlanCorrectionService that implements the four child-plan-state-
dependent behaviours from the specification when a correction's affected
subtree includes child plans:
- Not yet started → cancel the child plan
- In progress → cancel + rollback sandbox to pre-child-plan state
- Completed but not applied → cancel + rollback sandbox
- Already applied → reject the correction (CorrectionRejection)
Key additions:
- ChildPlanState enum classifying child plans into 4 states
- CorrectionRejection result type with reason and affected applied plan IDs
- CascadeAction/CascadeResult models for cascade operation tracking
- CorrectionStatus.REJECTED for rejected corrections
- Atomic cascade-or-rollback: all child plan actions succeed or the
entire cascade is rolled back
- Protocol-based dependency injection (ChildPlanLookup, ChildPlanCanceller,
SandboxRollbacker) for testability
- execute_correction_with_cascade() integrates with CorrectionService flow
Testing:
- 24 Behave BDD scenarios in cross_plan_correction.feature
- 8 Robot Framework end-to-end smoke tests
- ASV benchmarks for cascade performance with varying child plan counts
ISSUES CLOSED: #547
Add InvariantReconciliationActor that runs at the start of the Strategize
phase to reconcile invariants from four scopes (global, project, action,
plan). The actor detects conflicts, resolves them using specificity-based
precedence (plan > action > project > global), honours non_overridable
global invariants, records invariant_enforced decisions, and produces a
reconciled InvariantSet.
Changes:
- New: src/cleveragents/actor/reconciliation.py
- InvariantReconciliationActor class with collect_invariants() and run()
- reconcile_invariants() pure function
- ScopeInvariants, ConflictRecord, ReconciliationResult dataclasses
- Modified: src/cleveragents/domain/models/core/invariant.py
- Added non_overridable: bool field to Invariant model
- New: features/invariant_reconciliation_actor.feature (26 BDD scenarios)
- New: features/steps/invariant_reconciliation_actor_steps.py
- New: robot/invariant_reconciliation_actor.robot
- New: robot/helper_invariant_reconciliation.py
- New: benchmarks/invariant_reconciliation_bench.py
Closes#549
Add TDD-style failing tests that verify the built-in git-checkout resource
type is available after initialization. Tests assert the correct expected
behavior: after agents init, the git-checkout type should exist in the
registry and 'agents resource add git-checkout' should succeed.
Tests are expected to fail until bug #524 is fixed, because
bootstrap_builtin_types() is never called during initialization. The fix
branch should be based on this branch so the fix commit inherits these tests.
Files added:
- features/resource_type_bootstrap_git.feature (2 Behave scenarios)
- features/steps/resource_type_bootstrap_git_steps.py (step definitions)
- robot/resource_type_bootstrap_git.robot (Robot Framework smoke test)
ISSUES CLOSED: #553
Implement the runtime execution engine for validation tool wrapping,
as specified in docs/specification.md § Tool Wrapping.
WrappedToolExecutor resolves wraps references and delegates execution
to wrapped tools, supporting composable wrapping chains with cycle
detection and depth limiting (max 10 levels).
ArgumentMapper translates arguments between wrapper and wrapped tool
schemas using the argument_mapping configuration. Supports both
forwarded parameter names and literal fixed values.
TransformExecutor runs user-supplied transform functions in a
sandboxed Python environment with restricted builtins (no imports,
no filesystem, no network access). Validates that transforms return
proper validation-format dicts with a passed boolean.
Wired into the tool package public API via tool/__init__.py exports.
All new error types (WrappedToolNotFoundError, WrappingCycleError,
WrappingDepthExceededError, TransformExecutionError) provide clear
diagnostic messages.
Tests: 20 Behave scenarios covering argument mapping, transform
execution, simple/chained delegation, error handling, and sandbox
restrictions. 8 Robot Framework integration smoke tests. ASV
benchmarks for delegation overhead measurement.
ISSUES CLOSED: #543
Implement sandbox_boundary(r) function that walks up containment edges
in the resource DAG to the nearest sandboxable ancestor, enabling
resources sharing a boundary to share one sandbox instance.
Changes:
- Add boundary.py: is_sandbox_boundary(), sandbox_boundary(),
compute_sandbox_domains(), BoundaryCache (thread-safe, per-execution)
- Update SandboxManager: resolve_sandbox_key() and
get_or_create_sandbox_for_resource() key by (plan_id, boundary_id)
instead of (plan_id, resource_id); boundary cache lifecycle methods
- Define "sandboxable" via ResourceCapabilities.sandboxable + non-none
sandbox_strategy as per specification section 24659-24674
- Export new symbols from sandbox __init__.py
- Add vulture whitelist entries for new public API
Tests:
- 26 Behave BDD scenarios (features/sandbox_boundary_algebra.feature)
- 5 Robot Framework integration tests (robot/sandbox_boundary_algebra.robot)
- ASV benchmarks for boundary walk, domain grouping, and cache performance
ISSUES CLOSED: #548
Extended _compute_affected_subtree() to BFS over both the structural tree
(parent-child plan relationships) and decision_dependencies edges (influence
DAG). The algorithm performs a single O(V+E) BFS pass that unions neighbors
from both edge sources, using a visited set for cycle detection to guard
against data corruption.
Decision creation now supports dependency_decision_ids parameter in
record_decision() which populates the in-memory influence DAG store.
get_influence_edges() returns the adjacency list format consumed by
CorrectionService.
All public CorrectionService methods (analyze_impact, execute_revert,
execute_correction, generate_dry_run_report) accept an optional
influence_edges parameter while remaining backward-compatible (defaults
to None, preserving structural-only traversal when not provided).
Key design decisions:
- Single BFS pass over union of structural + influence edges rather than
separate traversals, ensuring O(V+E) complexity and consistent visit order
- Cycle detection via visited set with warning log (not an error) since
cycles indicate data corruption, not a programming error
- Influence edge logging: traversal emits count of influence edges processed
for observability
- Backward-compatible API: existing callers that only pass decision_tree
continue to work identically
ISSUES CLOSED: #542
Wire all four project context CLI commands (inspect, simulate, set, show)
to live ACMS pipeline services via ContextTierService and CRP models.
- context inspect: queries ContextTierService for tier metrics and
per-project fragments with filtering by strategy/focus/breadth/depth
- context simulate: dry-run context assembly using CRP models with
configurable token budget and assembly strategies
- context set: 12 new ACMS pipeline options (hot_max_tokens,
warm_max_decisions, cold_max_decisions, summary_max_tokens,
temporal_scope, auto_refresh, focus_area, breadth, depth,
assembly_strategy, retrieval_strategy, summary_strategy)
- context show: displays ACMS pipeline configuration alongside policy
- `context inspect` displayed global tier fragment counts (hot/warm/cold)
across all projects instead of counts for the target project only.
Add `ContextTierService.get_scoped_metrics(project_names)` which uses
`ScopedBackendView` to filter fragment counts to the specified projects
while keeping hit/miss counters as global service-level cache metrics.
Update `context_inspect()` to call `get_scoped_metrics([project])`
instead of `get_metrics()`.
- `context simulate --focus` accepted focus URIs and passed them to the
`ContextRequest` model but never used them to filter the fragment list,
making the `--focus` flag a no-op.
Add focus URI filtering in `_simulate_context_assembly()` after
`get_scoped_view()` — filters `project_fragments` by matching each
fragment's `resource_id` against the supplied focus URIs.
Also fixes redaction false positives for hot_max_tokens and
summary_max_tokens keys, and Rich Console line-wrapping in test
output that caused JSON parse failures.
Includes 28 new Behave BDD scenarios for wiring coverage, updated
Robot Framework integration tests, and reference documentation.
ISSUES CLOSED: #499
Implement safety profile resolution and enforcement in the tool
execution pipeline, replacing the NotImplementedError stub with
working precedence logic and runtime safety checks.
Core changes:
- resolve_safety_profile() now resolves plan > action > project >
global precedence, returning the highest-priority non-None profile
(or DEFAULT_SAFETY_PROFILE with GLOBAL provenance when all None)
- ToolExecutionContext gains an optional safety_profile field
- ToolRuntime._enforce_capabilities() extended with three new checks:
* Unsafe tool gating: blocks tools with unsafe=True when profile
has allow_unsafe_tools=False (ToolSafetyViolationError)
* Skill category allow-list: blocks tools whose skill category
is not in allowed_skill_categories (ToolSafetyViolationError)
* Checkpoint requirement: OR-combines ctx.require_checkpoints
with safety_profile.require_checkpoints
- New ToolSafetyViolationError in tool error hierarchy
Test coverage:
- 30 updated BDD scenarios in safety_profile.feature (resolve
precedence replaces NotImplementedError stub test)
- 24 new BDD scenarios in safety_profile_enforcement.feature
- 9 Robot Framework integration smoke tests
- 4 ASV benchmark suites (construction, serialization, resolution,
provenance enum)
All nox sessions pass (typecheck 0 errors, unit_tests 7735 scenarios
0 failures, coverage 97%, integration_tests 9/9 passed, benchmarks
complete).
ISSUES CLOSED: #345
Add LLMTrace Pydantic v2 domain model with all required fields (trace_id,
plan_id, decision_id, actor, provider, model, prompt_tokens, completion_tokens,
cost_usd, latency_ms, tool_calls, context_hash, streaming, retry_count, error).
Define 14 OperationalMetricKey values (PLAN_DURATION_MS, PLAN_TOTAL_COST_USD,
PLAN_DECISION_COUNT, ACTOR_INVOCATION_COUNT, ACTOR_LATENCY_MS,
TOOL_INVOCATION_COUNT, TOOL_ERROR_RATE, CONTEXT_BUILD_TIME_MS,
CONTEXT_TOKEN_COUNT, LLM_CALL_COUNT, LLM_TOTAL_TOKENS, LLM_TOTAL_COST_USD,
LLM_AVG_LATENCY_MS, SUBPLAN_COUNT) with MetricEntry model and MetricCollector.
Add llm_traces database table (LLMTraceModel) with LLMTraceRepository for
persistence. TraceService provides recording, querying, metric computation,
plan lifecycle hooks, and optional LangSmith forwarding when
LANGCHAIN_TRACING_V2=true.
Wired into DI container as trace_service. Includes 28 Behave BDD scenarios,
6 Robot Framework smoke tests, 3 ASV benchmark suites, and reference
documentation.
Fix pre-existing cli_core server_mode test flake by mocking
resolve_server_mode in test steps to avoid stale config file interference.
Closes#500
freemo
brent.edwards
hamza.khyari
CoreRasurae
aditya
hurui200320
khyari hamza