Compare commits

...

14 Commits

Author SHA1 Message Date
HAL9000 5b73be9f43 docs: fix quickstart plan/apply command signatures
CI / benchmark-publish (pull_request) Has been skipped
CI / helm (pull_request) Successful in 37s
CI / build (pull_request) Successful in 44s
CI / lint (pull_request) Successful in 1m0s
CI / quality (pull_request) Successful in 1m8s
CI / push-validation (pull_request) Successful in 29s
CI / typecheck (pull_request) Successful in 1m19s
CI / benchmark-regression (pull_request) Failing after 51s
CI / security (pull_request) Successful in 1m34s
CI / integration_tests (pull_request) Successful in 4m4s
CI / e2e_tests (pull_request) Successful in 4m12s
CI / unit_tests (pull_request) Successful in 4m40s
CI / docker (pull_request) Successful in 1m28s
CI / coverage (pull_request) Successful in 10m39s
CI / status-check (pull_request) Successful in 3s
Update docs/quickstart.md to use correct CLI command signatures:
- Replace 'cleveragents plan --project' with 'cleveragents plan use <action> --project'
- Replace 'cleveragents apply --project' with 'cleveragents plan list' + 'cleveragents plan apply <plan-id>'

Addresses reviewer non-blocking suggestion to verify CLI commands match
actual signatures before merging.
2026-05-05 14:30:56 +00:00
HAL9000 f179b57e18 docs: add quick start guide (PR #9245)
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 45s
CI / typecheck (pull_request) Successful in 1m6s
CI / quality (pull_request) Successful in 46s
CI / security (pull_request) Successful in 1m15s
CI / build (pull_request) Successful in 31s
CI / helm (pull_request) Successful in 24s
CI / push-validation (pull_request) Successful in 20s
CI / unit_tests (pull_request) Successful in 4m5s
CI / integration_tests (pull_request) Successful in 3m58s
CI / e2e_tests (pull_request) Successful in 4m23s
CI / benchmark-regression (pull_request) Failing after 1m29s
CI / docker (pull_request) Successful in 1m41s
CI / coverage (pull_request) Successful in 11m5s
CI / status-check (pull_request) Successful in 4s
Add docs/quickstart.md with end-to-end quick start guide covering
prerequisites, installation, project creation, resource registration,
plan/apply workflow, and troubleshooting.

Update mkdocs.yml navigation to include Quick Start page and rename
ADR-028 title to Skill/Tool Abstraction Definition.

Update CHANGELOG.md with quick start guide entry.

Closes #9245
2026-05-05 10:14:20 +00:00
HAL9000 90b06e6308 fix(db): remove redundant a5_007 merge migration (m4_004 already merged into m8_002 on master)
CI / benchmark-publish (pull_request) Has been skipped
CI / helm (pull_request) Successful in 46s
CI / build (pull_request) Successful in 52s
CI / lint (pull_request) Successful in 1m2s
CI / benchmark-regression (pull_request) Failing after 1m9s
CI / quality (pull_request) Successful in 1m13s
CI / security (pull_request) Successful in 1m31s
CI / typecheck (pull_request) Successful in 1m36s
CI / push-validation (pull_request) Successful in 24s
CI / integration_tests (pull_request) Successful in 3m43s
CI / e2e_tests (pull_request) Successful in 4m29s
CI / unit_tests (pull_request) Successful in 4m42s
CI / docker (pull_request) Successful in 1m52s
CI / coverage (pull_request) Successful in 11m35s
CI / coverage (push) Blocked by required conditions
CI / docker (push) Blocked by required conditions
CI / push-validation (push) Waiting to run
CI / status-check (push) Blocked by required conditions
CI / benchmark-publish (push) Has started running
CI / benchmark-regression (push) Has been skipped
CI / unit_tests (push) Has started running
CI / integration_tests (push) Has started running
CI / e2e_tests (push) Has started running
CI / status-check (pull_request) Successful in 3s
CI / helm (push) Successful in 35s
CI / build (push) Successful in 55s
CI / lint (push) Successful in 1m5s
CI / quality (push) Successful in 1m17s
CI / typecheck (push) Successful in 1m36s
CI / security (push) Successful in 1m43s
2026-05-05 09:37:22 +00:00
HAL9000 beaeac165d fix(db): add merge migration to resolve multiple Alembic heads
m4_004_schema_parity_resource_decision_checkpoint was added to master
after this branch was created, creating a second Alembic head alongside
a5_006_action_invariants_unique_constraint. The CI unit_tests job runs
on the merge commit (PR branch + master), so both migration files are
present, causing create_template_db.py to fail with MultipleHeads.

Add a5_007_merge_m4_004_and_a5_006 no-op merge migration to resolve
the two heads into a single head.
2026-05-05 09:37:22 +00:00
HAL9000 2240a028c8 fix(tests): replace ThreadPoolExecutor with threading.Thread in RLock concurrency steps
ThreadPoolExecutor holds internal locks (e.g. _global_shutdown_lock) that
can be in a locked state after fork(). The behave-parallel runner uses
multiprocessing.Pool with the fork start method, so forked worker processes
inherit these locked states and deadlock when trying to create or shut down
a ThreadPoolExecutor.

Replace both concurrent step implementations with direct threading.Thread
usage, following the pattern established in context_tier_thread_safety_steps.py
and other step files throughout the codebase. This eliminates the fork+lock
deadlock while preserving the deterministic concurrency-counter approach for
proving the RLock is released during transport calls.
2026-05-05 09:37:22 +00:00
HAL9000 236499f086 fix(tests): replace timing-based concurrency assertions with deterministic counter approach 2026-05-05 09:37:22 +00:00
HAL9000 60671306ea fix(tests): fix format and reduce timing thresholds in RLock concurrency tests
Fixed ruff format violation (single quotes to double quotes) in step definitions file.  Reduced SlowMCPTransport delay from 0.3s to 0.15s and tightened the wall-clock assertion from 0.9s to 0.40s to improve CI reliability while still proving concurrent execution.

ISSUES CLOSED: #10512
2026-05-05 09:37:22 +00:00
HAL9000 79ecaea18f fix(tests): use getattr for time._original_sleep fallback in RLock concurrency tests 2026-05-05 09:37:22 +00:00
HAL9000 69b7cd3cb2 fix(mcp): release RLock before transport call in MCPToolAdapter
MCPToolAdapter.invoke() and discover_tools() held the RLock during the
entire transport.call(), blocking all concurrent operations on the adapter.

The fix splits both methods into three phases:
1. Acquire lock — validate connection state and inputs
2. Release lock — make the transport call without holding the lock
3. Re-acquire lock (discover_tools only) — update shared state

Added Behave BDD tests verifying:
- Concurrent invocations complete in parallel (wall-clock < serial time)
- Concurrent discoveries complete in parallel
- Lock is provably not held during transport calls (cross-thread check)
- Validation still occurs under lock before transport call
- Disconnected adapter still raises RuntimeError under lock

ISSUES CLOSED: #10512
2026-05-05 09:37:22 +00:00
HAL9000 908e53ea85 fix(mcp): release RLock before transport call in MCPToolAdapter
MCPToolAdapter.invoke() and discover_tools() held the RLock during the
entire transport.call(), blocking all concurrent operations on the adapter.

The fix splits both methods into three phases:
1. Acquire lock — validate connection state and inputs
2. Release lock — make the transport call without holding the lock
3. Re-acquire lock (discover_tools only) — update shared state

Added Behave BDD tests verifying:
- Concurrent invocations complete in parallel (wall-clock < serial time)
- Concurrent discoveries complete in parallel
- Lock is provably not held during transport calls (cross-thread check)
- Validation still occurs under lock before transport call
- Disconnected adapter still raises RuntimeError under lock

ISSUES CLOSED: #10512
2026-05-05 09:37:22 +00:00
hurui200320 7164b04019 refactor(providers): unify provider factory behind single source of truth
CI / lint (push) Successful in 1m1s
CI / build (push) Successful in 53s
CI / quality (push) Successful in 1m23s
CI / typecheck (push) Successful in 1m24s
CI / benchmark-regression (push) Has been skipped
CI / security (push) Successful in 1m42s
CI / push-validation (push) Successful in 21s
CI / helm (push) Successful in 25s
CI / e2e_tests (push) Successful in 4m6s
CI / integration_tests (push) Successful in 4m12s
CI / unit_tests (push) Successful in 6m39s
CI / docker (push) Successful in 1m44s
CI / coverage (push) Successful in 10m51s
CI / status-check (push) Successful in 2s
CI / benchmark-publish (push) Successful in 1h17m10s
CI / benchmark-publish (pull_request) Has been skipped
CI / helm (pull_request) Successful in 33s
CI / build (pull_request) Successful in 50s
CI / lint (pull_request) Successful in 1m7s
CI / quality (pull_request) Successful in 1m9s
CI / typecheck (pull_request) Successful in 1m20s
CI / security (pull_request) Successful in 1m25s
CI / push-validation (pull_request) Successful in 19s
CI / integration_tests (pull_request) Successful in 3m20s
CI / e2e_tests (pull_request) Failing after 5m3s
CI / unit_tests (pull_request) Successful in 6m40s
CI / docker (pull_request) Successful in 1m30s
CI / coverage (pull_request) Successful in 11m9s
CI / status-check (pull_request) Failing after 3s
CI / benchmark-regression (pull_request) Failing after 1m11s
Extract _get_api_key() and _create_provider_instance() as a single internal
factory that handles all provider types, including MOCK. Both create_llm() and
create_ai_provider() now delegate to _create_provider_instance(), eliminating
the duplicated if/elif provider-switching chains and the divergence between the
two code paths. Adding a new provider now requires touching exactly one place.

Key changes:
- New _get_api_key(provider_type) consolidates API key lookup and validation
  that was previously duplicated across create_llm(), _create_provider_llm(),
  and every branch of create_ai_provider().
- New _create_provider_instance() is the unified raw-LLM factory, replacing
  _create_provider_llm() and the specialised adapter constructors in
  create_ai_provider(). MOCK is now handled here via FakeListLLM.
- create_ai_provider() wraps all providers uniformly in LangChainChatProvider
  instead of using specialised adapters (OpenAIChatProvider, AnthropicChatProvider,
  GoogleChatProvider, OpenRouterChatProvider) for the four primary providers.
- create_llm() drops its own API key validation block; validation now happens
  inside _create_provider_instance() via _get_api_key().
- All BDD scenarios updated: _create_provider_llm references become
  _create_provider_instance, and provider-type assertions on create_ai_provider
  results are updated to LangChainChatProvider.

Review fixes (Cycle 2):
- Add _coerce_env_bool() for consistent boolean env-var coercion in
  CLEVERAGENTS_ALLOW_MOCK_PROVIDER checks.
- Introduce _ApiKeyMissing sentinel to distinguish 'not provided' from None
  in provider configuration.
- Set MOCK supports_streaming=False in DEFAULT_CAPABILITIES to align with
  FakeListLLM semantics.
- Make MOCK is_configured conditional on CLEVERAGENTS_ALLOW_MOCK_PROVIDER.
- Use FakeListLLM(responses=['mock response'] * 10) to prevent IndexError
  in multi-step workflows.
- Strip max_retries from kwargs before passing to LangChain constructors.
- Extract _resolve_provider_type() helper shared by create_llm() and
  create_ai_provider() to reduce duplication and improve readability.
- Ensure __api_key_sentinel flows through factory_kwargs even when api_key
  is None, avoiding double _get_api_key() calls.
- Add CHANGELOG env-var documentation and providers.md env-var table.
- Resolve template DB lock issues by persisting in-memory SQLite via
  VACUUM INTO instead of direct file creation on tmpfs.

ISSUES CLOSED: #10949
2026-05-05 06:16:46 +00:00
HAL9000 988a169831 docs: add InvariantReconciliationActor API docs, devcontainer discovery module guide, and mkdocs nav
CI / benchmark-regression (push) Has been skipped
CI / benchmark-publish (push) Failing after 41s
CI / quality (push) Failing after 1m4s
CI / security (push) Failing after 1m4s
CI / unit_tests (push) Failing after 1m3s
CI / build (push) Failing after 45s
CI / push-validation (push) Successful in 47s
CI / helm (push) Successful in 42s
CI / lint (push) Successful in 1m33s
CI / e2e_tests (push) Failing after 45s
CI / integration_tests (push) Failing after 47s
CI / typecheck (push) Successful in 1m43s
CI / docker (push) Has been skipped
CI / coverage (push) Has been skipped
CI / status-check (push) Failing after 6s
CI / benchmark-publish (pull_request) Has been skipped
CI / helm (pull_request) Successful in 50s
CI / build (pull_request) Successful in 57s
CI / lint (pull_request) Successful in 1m9s
CI / quality (pull_request) Successful in 1m18s
CI / benchmark-regression (pull_request) Failing after 1m16s
CI / typecheck (pull_request) Successful in 1m30s
CI / security (pull_request) Successful in 1m36s
CI / e2e_tests (pull_request) Successful in 3m52s
CI / integration_tests (pull_request) Successful in 4m30s
CI / push-validation (pull_request) Successful in 20s
CI / unit_tests (pull_request) Successful in 6m31s
CI / docker (pull_request) Successful in 1m36s
CI / coverage (pull_request) Successful in 12m19s
CI / status-check (pull_request) Successful in 4s
- docs/api/actor.md: Add InvariantReconciliationActor section documenting
  the built-in reconciliation actor introduced in v3.8.0, including the
  reconciliation algorithm, ReconciliationResult/ConflictRecord/ScopeInvariants
  data classes, standalone reconcile_invariants() function, DI registration,
  and failure behaviour.

- docs/modules/devcontainer-discovery.md: New module guide for the
  devcontainer auto-discovery system (v3.8.0 fix #2615), covering
  DevcontainerDiscoveryResult, discover_devcontainers(), is_trigger_type(),
  monorepo named-config support, and gotchas.

- mkdocs.yml: Add 'Devcontainer Auto-Discovery' to the Modules nav section,
  alongside shell-safety, uko-provenance, invariant-reconciliation,
  context-hydration, and git-worktree-sandbox module docs.

- robot/coverage_threshold.robot: Add tdd_issue and tdd_issue_4305 tags
  to the Noxfile Contains Coverage Threshold Constant test case.

ISSUES CLOSED: #4485
2026-05-05 06:15:57 +00:00
HAL9000 39175dd284 fix(git_tools): eliminate TOCTOU race in _get_base_env() with double-checked locking
CI / benchmark-publish (push) Has started running
CI / push-validation (push) Successful in 43s
CI / helm (push) Successful in 44s
CI / build (push) Successful in 1m22s
CI / benchmark-regression (push) Has been skipped
CI / lint (push) Successful in 1m49s
CI / quality (push) Successful in 2m2s
CI / typecheck (push) Successful in 2m5s
CI / security (push) Successful in 2m26s
CI / integration_tests (push) Successful in 5m1s
CI / e2e_tests (push) Successful in 6m11s
CI / unit_tests (push) Successful in 11m38s
CI / docker (push) Successful in 2m20s
CI / coverage (push) Successful in 16m24s
CI / status-check (push) Successful in 4s
CI / benchmark-publish (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Failing after 1m8s
CI / coverage (pull_request) Successful in 12m53s
CI / unit_tests (pull_request) Successful in 8m11s
CI / quality (pull_request) Successful in 1m35s
CI / e2e_tests (pull_request) Successful in 4m58s
CI / helm (pull_request) Successful in 40s
CI / push-validation (pull_request) Successful in 21s
CI / build (pull_request) Successful in 1m4s
CI / lint (pull_request) Successful in 1m10s
CI / typecheck (pull_request) Successful in 1m35s
CI / security (pull_request) Successful in 2m19s
CI / integration_tests (pull_request) Successful in 4m12s
CI / docker (pull_request) Successful in 2m15s
CI / status-check (pull_request) Successful in 3s
Add module-level _BASE_ENV_LOCK: threading.Lock and replace the bare
if _BASE_ENV is None assignment with double-checked locking. The outer
check keeps the warm-cache path lock-free; the inner check inside
with _BASE_ENV_LOCK prevents duplicate initialisation when two threads
race on the very first call.

Add three BDD scenarios in features/git_tools.feature (step definitions
in features/steps/git_tools_thread_safety_steps.py) verifying caching
identity, content correctness, and thread safety under 20 concurrent
threads using threading.Barrier.

Update CHANGELOG.md and CONTRIBUTORS.md per contribution guidelines.

ISSUES CLOSED: #7619
2026-05-05 05:25:28 +00:00
HAL9000 81935a9cf7 chore(agents): improve ca-test-infra-improver — strengthen duplicate avoidance
CI / coverage (push) Blocked by required conditions
CI / docker (push) Blocked by required conditions
CI / status-check (push) Blocked by required conditions
CI / benchmark-regression (push) Waiting to run
CI / unit_tests (push) Has started running
CI / benchmark-publish (push) Has started running
CI / lint (push) Successful in 1m10s
CI / quality (push) Successful in 1m29s
CI / typecheck (push) Successful in 1m36s
CI / security (push) Successful in 1m39s
CI / helm (push) Successful in 35s
CI / push-validation (push) Successful in 50s
CI / build (push) Successful in 57s
CI / e2e_tests (push) Successful in 4m0s
CI / integration_tests (push) Successful in 4m20s
CI / lint (pull_request) Successful in 1m4s
CI / quality (pull_request) Successful in 1m6s
CI / typecheck (pull_request) Successful in 1m30s
CI / security (pull_request) Successful in 1m31s
CI / push-validation (pull_request) Successful in 37s
CI / helm (pull_request) Successful in 39s
CI / build (pull_request) Successful in 58s
CI / benchmark-publish (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Failing after 1m22s
CI / integration_tests (pull_request) Successful in 4m35s
CI / e2e_tests (pull_request) Failing after 5m18s
CI / unit_tests (pull_request) Successful in 7m26s
CI / docker (pull_request) Successful in 3m9s
CI / coverage (pull_request) Successful in 14m19s
CI / status-check (pull_request) Failing after 3s
Expand the Duplicate Avoidance section from a 3-step placeholder into a
comprehensive 6-step process to prevent the 48+ duplicate TEST-INFRA issues
observed across 8 topic clusters.

Key additions:
- CRITICAL blockquote: when in doubt, skip
- Keyword extraction before filing
- Keyword-based search (open AND closed issues, not just TEST-INFRA: prefix)
- Semantic comparison with concrete examples
- Cross-agent prefix checking (TEST-INFRA:, BUG-HUNT:, UAT:)
- Post-filing verification with 5-second delay for race condition handling
- Duplicate Check audit section requirement in every filed issue body

ISSUES CLOSED: #2370
2026-05-05 04:53:48 +00:00
21 changed files with 1679 additions and 426 deletions
+39 -4
View File
@@ -415,10 +415,45 @@ For each finding, invoke `ca-new-issue-creator` with:
### Duplicate Avoidance
Before filing any issue:
1. Search Forgejo for existing issues with "TEST-INFRA:" prefix
2. Check for similar titles/descriptions
3. If potential duplicate found, skip
> **CRITICAL: When in doubt, SKIP — it is better to miss an improvement
> suggestion than to create noise that wastes groomer and implementor time.**
Before filing ANY issue, you MUST perform rigorous duplicate checking:
1. **Extract keywords**: From your proposed issue title, extract 3-5 key
technical terms (e.g., "parallelize", "E2E", "coverage", "Docker",
"nox", "cache").
2. **Search by keywords**: Search Forgejo for open AND closed issues containing ANY
of those key terms. Use the search API, not just the "TEST-INFRA:"
prefix.
3. **Compare semantically**: For each search result, compare your proposed
improvement with the existing issue's description. Two issues are
**duplicates** if they propose the same optimization for the same
component, even if the wording differs. Examples of duplicates:
- "Parallelize E2E tests" and "Run E2E tests in parallel"
- "Create custom Docker image for CI" and "Pre-built CI base image"
- "Cache nox environments" and "Persist nox virtualenvs between runs"
4. **Check all prefixes**: Search for issues with `TEST-INFRA:`, `BUG-HUNT:`,
and `UAT:` prefixes — other agents may have already identified the same
improvement opportunity from a different angle.
5. **If ANY existing issue proposes the same or substantially similar
improvement**: **SKIP** — do not file. It is far better to miss one
improvement suggestion than to file the 7th duplicate of the same idea.
Previous sessions created 48+ TEST-INFRA issues with significant overlap
across 8 topic clusters.
6. **Post-filing verification**: After filing an issue, wait 5 seconds and
re-check for duplicates (another parallel worker may have filed the same
issue simultaneously). If a duplicate appeared, close your issue as a
duplicate of the earlier one.
When filing, include a `### Duplicate Check` section in the issue body
listing the search queries used, result counts, and your justification for
why this is not a duplicate.
---
+37 -8
View File
@@ -50,6 +50,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
- `agents actor context clear` command to reset actor message history and
state while preserving the underlying context directory via `ContextManager`
(#6370).
- **Quick Start Guide** (PR #9245): Added `docs/quickstart.md` with an end-to-end quick start guide covering prerequisites, installation, project creation, resource registration, plan/apply workflow, and troubleshooting. Updated `mkdocs.yml` navigation to include the Quick Start page.
- **Plan checkpoint management CLI commands** (#8683): Added `agents plan checkpoint-list <plan-id>` and `agents plan checkpoint-delete <checkpoint-id>` commands. Listing output now highlights checkpoint ID, type, created timestamp, reason, phase, and decision linkage with a concise field summary footer across rich/table/json/yaml formats. Deletion supports batch IDs, interactive confirmation (skip with `--yes`), and structured JSON/YAML responses for automation-friendly scripting.
- **Invariant Remove CLI Command** (#8530): Implemented `agents invariant remove <id>` command that soft-deletes an invariant by ID. The command displays a confirmation prompt before removal (bypassable with `--yes`/`-y`), outputs the removed invariant ID on success, and shows a clear error message when the invariant ID does not exist. Supports `--format` flag for JSON and YAML output. Full BDD test coverage and Robot Framework integration tests included.
@@ -78,6 +79,42 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
### Fixed
- **git_tools._get_base_env() TOCTOU Race Condition** (#7619): Fixed a
Time-Of-Check-To-Time-Of-Use race condition in `git_tools._get_base_env()`
where two concurrent threads could both observe `_BASE_ENV is None`, both
snapshot `os.environ`, and write potentially different snapshots. The fix
adds a module-level `_BASE_ENV_LOCK: threading.Lock` and replaces the bare
`if _BASE_ENV is None` assignment with double-checked locking: the outer
check keeps the warm-cache path lock-free; the inner check inside
`with _BASE_ENV_LOCK` prevents duplicate initialisation on the very first
concurrent call. Three new BDD scenarios in `features/git_tools.feature`
(with step definitions in
`features/steps/git_tools_thread_safety_steps.py`) verify caching identity,
content correctness, and thread safety under 20 concurrent threads.
- **Unified provider factory: eliminate divergence between `create_llm()` and `create_ai_provider()`** (#10949):
Introduced `_create_provider_instance()` as the single internal factory so that
both public methods delegate to one place. Creating a new provider now
requires changes in exactly one method.
- **Fixed API key regression**: the unified factory now explicitly passes
the validated API key to all LangChain constructors (OpenAI, Anthropic,
Google / Gemini, Azure, Groq, Together, Cohere, and OpenRouter). Users
who configure providers via `CLEVERAGENTS_`-prefixed variables are no
longer silently failed when LangChain falls back to raw environment
variable lookup. Pre-validated keys are forwarded through the
`api_key` kwarg to avoid a second settings lookup in the factory
closure. (Closes #10949)
- **Fixed mock provider accessibility in production**: `ProviderType.MOCK`
is now gated by the `CLEVERAGENTS_ALLOW_MOCK_PROVIDER=true` sentinel
environment variable. Without this flag, both `create_llm()` and
`create_ai_provider()` raise `ValueError` when MOCK is requested,
preventing accidental or malicious use of the fake LLM in production.
`resolve_provider_by_name("mock")` now also respects the guard and
`is_provider_configured(ProviderType.MOCK)` returns `True` as expected.
- **Fixed type annotation**: `create_llm()` now declares `**kwargs: Any`
instead of `**kwargs: object`, restoring correct Pyright inference for
forwarded keyword arguments.
- **`create_llm()` raises `Unsupported provider type: openrouter`** (#10948): Fixed
`ProviderRegistry._create_provider_llm()` missing an `OPENROUTER` branch, which
caused `agents actor run openrouter/<model>` to fail with `ValueError`. Added a
@@ -87,14 +124,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
`default_headers` kwarg with automatic string coercion for non-string keys/values.
- **LoadingThrobber Widget Restored** (#6357): Restored `LoadingThrobber` widget
(`src/cleveragents/tui/widgets/throbber.py`) and its Robot Framework integration
tests (`robot/tui_throbber.robot`) that were missing from master. Also restored
supporting modules `src/cleveragents/tui/quotes.py` and
`src/cleveragents/tui/data/throbber_quotes.txt`. Fixed the `Throbber Rejects
Invalid Styles` integration test by using `Fix Python Indentation` to correctly
reconstruct indentation stripped by Robot Framework's `Catenate` keyword.
Narrowed exception handling in `throbber.py` to specific types
(`ImportError`, `AttributeError`) per coding standards.
- **Built-in actors v3 YAML format** (#10883): Fixed `agents actor run` failing for
built-in actors (e.g., `openai/gpt-4`, `anthropic/claude-3-opus`) due to missing
+3
View File
@@ -25,4 +25,7 @@ Below are some of the specific details of various contributions.
* This project was made possible thanks to considerable donation of time, money, and resources by CleverThis, Inc.
* HAL 9000 has contributed automated bug fixes, CLI output formatting improvements, and ongoing maintenance as part of the CleverAgents automation system.
* HAL 9000 has contributed the file edit encoding parameter fix (PR #8258 / issue #7559).
* HAL 9000 has contributed the architecture-pool-supervisor milestone assignment feature (PR #8188 / issue #7521): added `forgejo_update_pull_request` permission and documented the PR workflow for major spec changes, enabling automatic milestone assignment for specification PRs.
* HAL 9000 has contributed the git worktree TOCTOU race condition fix (PR #8178 / issue #7507): replaced the unsafe mkdtemp() + rmdir() pattern with a parent-directory approach to eliminate the race window in concurrent git worktree operations.
* HAL 9000 has contributed the git_tools TOCTOU race condition fix (PR #8255 / issue #7619): eliminated the Time-Of-Check-To-Time-Of-Use race in `_get_base_env()` by adding double-checked locking with a module-level `threading.Lock`, preventing concurrent threads from writing conflicting environment snapshots.
* HAL 9000 has contributed comprehensive milestone documentation for v3.6.0 (Advanced Concepts & Deferred Features) and v3.7.0 (TUI Implementation) (PR #9903): split into sub-documents covering context strategies, LLM backends, resource types, A2A rename, container tool execution, scope chain resolution, cost/safety budgets, E2E workflow tests, code review examples, plugin architecture, TUI layout, persona system, reference/command input, session management, configuration, and TuiMaterializer integration.
+124
View File
@@ -100,6 +100,130 @@ nodes, and any warnings.
---
## `InvariantReconciliationActor`
**Introduced:** v3.8.0 | **Module:** `cleveragents.actor.reconciliation`
The built-in `InvariantReconciliationActor` (`builtin/invariant-reconciliation`)
is invoked by `PlanLifecycleService` when a plan enters the Strategize phase.
This gate must succeed before Strategize work begins; the lifecycle service
persists the reconciled invariant view and reuses it for later Execute or Apply
transitions instead of re-running the actor. The actor collects invariants
from four scopes, resolves conflicts using specificity-based precedence,
records `invariant_enforced` decisions, and returns a reconciled
`InvariantSet`.
See [ADR-016 — Invariant System](../adr/ADR-016-invariant-system.md) for the
design rationale behind invariant reconciliation.
```python
from cleveragents.actor.reconciliation import InvariantReconciliationActor
actor = InvariantReconciliationActor(
invariant_service=container.invariant_service(),
decision_service=container.decision_service(),
)
result = actor.run(plan_id="01HXYZ...", project_name="local/my-app")
# result.reconciled_set — effective InvariantSet
# result.conflicts — list[ConflictRecord] with resolution details
# result.enforced_decision_ids — list of ULID strings
```
### Reconciliation Algorithm
1. Collect invariants from four scopes: **global**, **project**, **action**, **plan**.
2. Group by normalised text (case-insensitive, stripped).
3. Detect conflicts between invariants at different scopes.
4. Resolve using specificity: `plan > action > project > global`.
Exception: `non_overridable` global invariants always win.
5. Record an `invariant_enforced` decision for each active invariant.
6. Return a reconciled `InvariantSet`.
### `InvariantReconciliationActor` Methods
| Method signature | Description |
|------------------|-------------|
| `collect_invariants(*, plan_id: str \| None = None, project_name: str \| None = None, action_name: str \| None = None) -> ScopeInvariants` | Collect invariants from all four scopes and return them as a `ScopeInvariants` container |
| `run(*, plan_id: str, project_name: str \| None = None, action_name: str \| None = None, parent_decision_id: str \| None = None) -> ReconciliationResult` | Execute the reconciliation lifecycle (collect → reconcile → record) and return a `ReconciliationResult` |
### `ReconciliationResult`
```python
from cleveragents.actor.reconciliation import ReconciliationResult
@dataclass(frozen=True)
class ReconciliationResult:
reconciled_set: InvariantSet # effective invariants
conflicts: list[ConflictRecord] # detected conflicts with resolution details
enforced_decision_ids: list[str] # ULIDs of invariant_enforced decisions
```
### `ConflictRecord`
```python
from cleveragents.actor.reconciliation import ConflictRecord
@dataclass(frozen=True)
class ConflictRecord:
key: str # normalised invariant text used for grouping
winner: Invariant # invariant that prevailed
losers: list[Invariant] # invariants that were overridden
reason: str # human-readable explanation of the resolution
```
### `ScopeInvariants`
Convenience container grouping invariants by scope tier:
```python
from cleveragents.actor.reconciliation import ScopeInvariants
scope_invs = ScopeInvariants(
global_invariants=[...],
project_invariants=[...],
action_invariants=[...],
plan_invariants=[...],
)
all_invs = scope_invs.all_invariants() # flat list across all scopes
```
### `reconcile_invariants` (standalone function)
```python
from cleveragents.actor.reconciliation import reconcile_invariants
reconciled, conflicts = reconcile_invariants(scope_invariants)
# reconciled — list[Invariant] (winners only)
# conflicts — list[ConflictRecord]
```
### Constructing the actor via dependency injection
`InvariantReconciliationActor` is instantiated directly with application
services resolved from the DI container:
```python
from cleveragents.application.container import Container
from cleveragents.actor.reconciliation import InvariantReconciliationActor
container = Container()
actor = InvariantReconciliationActor(
invariant_service=container.invariant_service(),
decision_service=container.decision_service(),
)
```
### Failure Behaviour
Reconciliation failures block the phase transition with
`ReconciliationBlockedError` and emit an `INVARIANT_VIOLATED` event. The
exception is raised by `PlanLifecycleService`, importable via
`from cleveragents.application.services.plan_lifecycle_service import ReconciliationBlockedError`.
Post-correction reconciliation runs via `CORRECTION_APPLIED` event subscription
(best-effort; does not block correction completion).
---
## Example: Loading and Compiling an Actor
```python
+7 -4
View File
@@ -72,7 +72,7 @@ gate features (e.g., tool calls, vision inputs) at runtime.
| Cohere | ✓ | ✓ | ✗ | 128 000 | ✗ |
| Groq | ✓ | ✓ | ✗ | 32 000 | ✓ |
| Together | ✓ | ✓ | ✗ | 32 000 | ✗ |
| Mock | | ✗ | ✗ | 4 096 | ✗ |
| Mock | | ✗ | ✗ | 4 096 | ✗ |
---
@@ -175,9 +175,11 @@ unknown.
#### `create_ai_provider(provider_type=None, model_id=None, max_retries=3) → AIProviderInterface`
Creates an `AIProviderInterface` implementation. For most providers this
returns a `LangChainChatProvider`; Google and OpenRouter return their own
specialized implementations.
Creates an `AIProviderInterface` implementation. **All** providers are now
wrapped uniformly in `LangChainChatProvider`; the raw LLM instantiation is
delegated to `_create_provider_instance()` internally. Previously, some
providers (Google, OpenRouter) returned specialised adapter classes, but the
unified factory now handles every provider type consistently.
```python
provider = registry.create_ai_provider("openai", model_id="gpt-4o-mini")
@@ -266,6 +268,7 @@ provider = resolve_provider_by_name("anthropic")
| `GROQ_API_KEY` | Groq credentials |
| `TOGETHER_API_KEY` | Together AI credentials |
| `CLEVERAGENTS_TESTING_USE_MOCK_AI` | Force mock provider in tests |
| `CLEVERAGENTS_ALLOW_MOCK_PROVIDER` | Enable MOCK provider for testing (set to true) |
---
+175
View File
@@ -0,0 +1,175 @@
# Devcontainer Auto-Discovery
**Package:** `cleveragents.resource.handlers.discovery`
**Introduced:** v3.8.0 (issue #2615)
This module guide covers the devcontainer auto-discovery system, which
automatically detects `devcontainer.json` configurations when a
`git-checkout` or `fs-directory` resource is linked to a project.
It supports both root-level configurations and named configurations
for monorepo projects.
---
## Purpose
When a `git-checkout` or `fs-directory` resource is registered, the
`discover_devcontainers()` function scans the resource location for
`devcontainer.json` files. Each valid configuration is returned as a
`DevcontainerDiscoveryResult`, which the resource registry uses to
create child `devcontainer-instance` resources automatically.
---
## Scan Paths
The scanner checks three categories of paths:
| Category | Path | `config_name` |
|----------|------|---------------|
| Root-level (default) | `.devcontainer/devcontainer.json` | `None` |
| Root-level (flat) | `.devcontainer.json` | `None` |
| Named configuration | `.devcontainer/<name>/devcontainer.json` | `"<name>"` |
Named configurations are discovered by iterating one subdirectory level
inside `.devcontainer/`. Each subdirectory that contains a
`devcontainer.json` produces a separate result with `config_name` set to
the subdirectory name (e.g. `"api"`, `"frontend"`).
---
## Key Classes and Functions
### `DevcontainerDiscoveryResult`
```python
from cleveragents.resource.handlers.discovery import DevcontainerDiscoveryResult
result = DevcontainerDiscoveryResult(
config_path=Path("/workspace/.devcontainer/api/devcontainer.json"),
config_data={"name": "API Dev Container", "image": "mcr.microsoft.com/devcontainers/python:3.12"},
parent_location="/workspace",
config_name="api",
)
```
| Attribute | Type | Description |
|-----------|------|-------------|
| `config_path` | `Path` | Absolute path to the `devcontainer.json` file |
| `config_data` | `dict[str, object]` | Parsed JSON content of the devcontainer config |
| `parent_location` | `str` | Filesystem path of the parent resource |
| `config_name` | `str \| None` | Named configuration identifier, or `None` for root-level configs |
---
### `discover_devcontainers`
```python
from cleveragents.resource.handlers.discovery import discover_devcontainers
results = discover_devcontainers(
resource_location="/workspace",
resource_type="git-checkout",
)
for result in results:
print(result.config_name, result.config_path)
```
Scans a resource location for all valid devcontainer configurations.
Invalid JSON files are skipped with a warning log rather than raising.
**Parameters:**
| Parameter | Type | Description |
|-----------|------|-------------|
| `resource_location` | `str` | Filesystem path of the parent resource |
| `resource_type` | `str` | Type name of the parent resource |
**Returns:** `list[DevcontainerDiscoveryResult]` — one entry per valid config.
**Raises:**
- `ValueError` — if `resource_location` is empty or `resource_type` is empty/blank.
---
### `is_trigger_type`
```python
from cleveragents.resource.handlers.discovery import is_trigger_type
is_trigger_type("git-checkout") # True
is_trigger_type("sqlite") # False
```
Returns `True` if the given resource type triggers devcontainer auto-discovery.
Currently only `"git-checkout"` and `"fs-directory"` are trigger types.
---
## Monorepo Example
Given a monorepo with the following layout:
```
/workspace/
├── .devcontainer/
│ ├── api/
│ │ └── devcontainer.json → config_name="api"
│ └── frontend/
│ └── devcontainer.json → config_name="frontend"
└── src/
```
`discover_devcontainers("/workspace", "git-checkout")` returns two results:
```python
results = discover_devcontainers("/workspace", "git-checkout")
# results[0].config_name == "api"
# results[1].config_name == "frontend"
```
Each result produces a distinct `devcontainer-instance` resource in the
registry, allowing plans to target individual containers by name.
---
## Root-Level Example
For a single-container project:
```
/workspace/
├── .devcontainer/
│ └── devcontainer.json → config_name=None
└── src/
```
```python
results = discover_devcontainers("/workspace", "git-checkout")
# results[0].config_name is None
```
Root-level configs retain `config_name=None` for full backward compatibility.
---
## Gotchas
- **Non-trigger types return an empty list.** Calling `discover_devcontainers()`
with a resource type that is not `"git-checkout"` or `"fs-directory"` returns
`[]` immediately without scanning the filesystem.
- **Invalid JSON is silently skipped.** Files that cannot be parsed as JSON
objects are logged at `WARNING` level and excluded from results.
- **Only one subdirectory level is scanned.** Named configurations must be
directly inside `.devcontainer/<name>/devcontainer.json`. Deeper nesting
is not supported.
- **Sorted order.** Named configurations are returned in lexicographic order
of their subdirectory names (via `sorted(dc_dir.iterdir())`).
---
## Related Documentation
- [Resource System API](../api/resource.md) — `ResourceHandler` protocol and built-in handlers
- [ADR-043 Devcontainer Integration](../adr/ADR-043-devcontainer-integration.md) — design rationale
- [Resource System](../api/resource.md#devcontainerhandler) — `DevcontainerHandler` CRUD operations
+45
View File
@@ -0,0 +1,45 @@
# Quick Start Guide
This quick start guide will walk you through creating a new project, registering a resource, running a plan, and applying changes with CleverAgents.
## Prerequisites
- Python 3.11+ and virtualenv
- Git
- A working CleverAgents installation (see development/testing.md for details)
## Install (local development)
```bash
python -m venv .venv
source .venv/bin/activate
pip install -e .[dev]
```
## Create a new project
```bash
# Create a new directory for your project
mkdir my-project && cd my-project
# Initialize a CleverAgents project (example command)
cleveragents init --name my-project
```
## Register a resource
Create a resource file under `resources/` (example YAML) and register it with the CLI or API.
## Plan and apply
```bash
# Create a plan using an action on your project
cleveragents plan use <action-name> --project my-project
# List plans to find the plan ID
cleveragents plan list
# Review the plan, then apply it by plan ID
cleveragents plan apply <plan-id>
```
## Troubleshooting
If you encounter issues running the examples above, consult `docs/development/testing.md` and the project README for local development tips.
+11
View File
@@ -578,6 +578,16 @@ def before_scenario(context, scenario):
context.acms_error = None
context.assemble_error = None
# --- Env-var save/restore for provider registry ---
# Proactively save env vars so they are restored even if a scenario fails
# before reaching the step that normally records them.
for attr, env_var in [
("original_provider_env", "CLEVERAGENTS_DEFAULT_PROVIDER"),
("original_model_env", "CLEVERAGENTS_DEFAULT_MODEL"),
("original_allow_mock_env", "CLEVERAGENTS_ALLOW_MOCK_PROVIDER"),
]:
setattr(context, attr, os.environ.get(env_var))
# Ensure mock AI flag is always set so plan service tests can resolve actors
os.environ["CLEVERAGENTS_TESTING_USE_MOCK_AI"] = "true"
@@ -740,6 +750,7 @@ def after_scenario(context, scenario):
for attr, env_var in [
("original_provider_env", "CLEVERAGENTS_DEFAULT_PROVIDER"),
("original_model_env", "CLEVERAGENTS_DEFAULT_MODEL"),
("original_allow_mock_env", "CLEVERAGENTS_ALLOW_MOCK_PROVIDER"),
]:
if hasattr(context, attr):
original = getattr(context, attr)
+16
View File
@@ -267,3 +267,19 @@ Feature: Built-in Git Tools
Given a tool registry
When I register all git tools
Then all registered git tools should be read-only
# ---- Thread Safety (_get_base_env TOCTOU fix) ----
Scenario: _get_base_env returns the same dict object on repeated calls
When I call _get_base_env twice
Then both calls should return the same dict object
Scenario: _get_base_env returns a dict containing git override keys
When I call _get_base_env
Then the result should contain key "GIT_PAGER"
And the result should contain key "NO_COLOR"
And the result should contain key "GIT_TERMINAL_PROMPT"
Scenario: _get_base_env is safe under concurrent initialisation
When I call _get_base_env concurrently from 20 threads
Then all threads should receive the same dict object
+97 -112
View File
@@ -54,29 +54,26 @@ Feature: Provider Registry Coverage
And the is_configured should be False by default
@unit @providers @registry
Scenario: Optional string helper coerces non-string values
Given I import the optional string coercion helper
When I coerce optional string with integer 42
Then the coerced optional string result should be "42"
@unit @providers @registry
Scenario: Get all providers from registry
Given I have a ProviderRegistry instance
When I call get_all_providers
Then the provider registry result should be a list of ProviderInfo
And the provider registry result should include core provider types
@unit @providers @registry
Scenario: Get configured providers when none configured
Given I have a ProviderRegistry with no API keys
Scenario: Get configured providers when none configured and mock not allowed
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is not set
And I have a ProviderRegistry with no API keys
When I call get_configured_providers
Then the provider registry result should be an empty list
@unit @providers @registry
Scenario: Get configured providers when none configured
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set
And I have a ProviderRegistry with no API keys
When I call get_configured_providers
Then the provider registry result should contain only the MOCK provider
@unit @providers @registry
Scenario: Get configured providers with OpenAI key
Given I have a ProviderRegistry with OpenAI API key set
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set
And I have a ProviderRegistry with OpenAI API key set
When I call get_configured_providers
Then the provider registry result should contain OpenAI provider
And the provider registry result should contain the MOCK provider
And the provider registry OpenAI entry should be configured
@unit @providers @registry
@@ -295,7 +292,8 @@ Feature: Provider Registry Coverage
@unit @providers @registry
Scenario: Create LLM for mock provider skips API key requirement
Given I have a ProviderRegistry with no API keys
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set
And I have a ProviderRegistry with no API keys
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
And CLEVERAGENTS_DEFAULT_MODEL is not set
And the provider registry LLM factory is stubbed
@@ -303,17 +301,34 @@ Feature: Provider Registry Coverage
Then the stubbed LLM factory should be called for ProviderType.MOCK with model "mock-gpt"
@unit @providers @registry
Scenario: _create_provider_llm builds stubbed OpenAI client
Scenario: Create LLM for mock provider is rejected without env var
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is not set
And I have a ProviderRegistry with no API keys
When I try to create an LLM for provider "mock"
Then a provider registry ValueError should be raised
And the provider registry error should mention "not allowed"
@unit @providers @registry
Scenario: Create LLM for mock provider returns FakeListLLM end-to-end
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set
And I have a ProviderRegistry with no API keys
When I call create_llm for provider "mock" without specifying a model
Then the created LLM should be a FakeListLLM instance
And the FakeListLLM produces response "mock response" on invoke
And the FakeListLLM produces response "mock response" on invoke
@unit @providers @registry
Scenario: _create_provider_instance builds stubbed OpenAI client
Given I have a ProviderRegistry with OpenAI API key set
And langchain OpenAI client is stubbed
When I call _create_provider_llm for provider "openai" with model "custom-openai"
When I call _create_provider_instance for provider "openai" with model "custom-openai"
Then the stubbed ChatOpenAI should be created with model "custom-openai"
@unit @providers @registry
Scenario Outline: _create_provider_llm builds <ProviderName> client
Scenario Outline: _create_provider_instance builds <ProviderName> client
Given I have a ProviderRegistry with "<provider_key>" API key set
And the "<ProviderName>" LangChain client is stubbed
When I call _create_provider_llm for provider "<provider_key>" with model "<model>"
When I call _create_provider_instance for provider "<provider_key>" with model "<model>"
Then the stubbed <ProviderName> client should be created with argument "model" value "<model>"
Examples:
@@ -326,92 +341,12 @@ Feature: Provider Registry Coverage
| Cohere | cohere | command-cover |
@unit @providers @registry
Scenario: _create_provider_llm builds Azure client with deployment override
Given I have a ProviderRegistry with "azure" API key set
And the "Azure" LangChain client is stubbed
When I call _create_provider_llm for provider "azure" with model "azure-cover"
Then the stubbed Azure client should be created with argument "deployment_name" value "azure-cover"
@unit @providers @registry
Scenario: _create_provider_llm raises when Azure endpoint is missing
Given I have a ProviderRegistry with "azure" API key set
And the Azure endpoint setting is cleared
When I try to call _create_provider_llm for provider "azure" with model "azure-cover"
Then a provider registry ValueError should be raised
And the provider registry error should mention missing Azure endpoint
@unit @providers @registry
Scenario: _create_provider_llm builds OpenRouter client with ChatOpenAI
Given I have a ProviderRegistry with "openrouter" API key set
And the "openrouter" LangChain client is stubbed
When I call _create_provider_llm for provider "openrouter" with model "claude-openrouter-cover"
Then the stubbed openrouter client should be created with argument "model" value "claude-openrouter-cover"
And the stubbed openrouter client should be created with argument "openai_api_base" value "https://openrouter.ai/api/v1"
And the stubbed openrouter client should be created with argument "openai_api_key" value "sk-openrouter-test"
@unit @providers @registry
Scenario: _create_provider_llm builds OpenRouter client with sanitized default headers
Given I have a ProviderRegistry with "openrouter" API key set
And the "openrouter" LangChain client is stubbed
And the OpenRouter default headers include numeric entries
When I call _create_provider_llm for provider "openrouter" with model "claude-openrouter-cover" and custom default headers
Then the stubbed OpenRouter client headers should include "123" value "456"
And the stubbed OpenRouter client headers should include "X-Debug" value "789"
And the stubbed openrouter client should be created with argument "openai_api_key" value "sk-openrouter-test"
And the stubbed OpenRouter client headers should not include key 123
@unit @providers @registry
Scenario: create_llm succeeds for OpenRouter and delegates to _create_provider_llm
Given I have a ProviderRegistry with "openrouter" API key set
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
And CLEVERAGENTS_DEFAULT_MODEL is not set
And the provider registry LLM factory is stubbed
When I call create_llm for provider "openrouter" and model "claude-openrouter"
Then the stubbed LLM factory should be called for ProviderType.OPENROUTER with model "claude-openrouter"
@unit @providers @registry
Scenario: _create_provider_llm adds organization headers for OpenRouter
Given I have a ProviderRegistry with "openrouter" API key set
And the OpenRouter organization setting is "my-org"
And the "openrouter" LangChain client is stubbed
When I call _create_provider_llm for provider "openrouter" with model "claude-openrouter-cover"
Then the stubbed OpenRouter client headers should include "HTTP-Referer" value "my-org"
And the stubbed OpenRouter client headers should include "X-Title" value "my-org"
@unit @providers @registry
Scenario: _create_provider_llm uses default model for OpenRouter when model_id is None
Given I have a ProviderRegistry with "openrouter" API key set
And the "openrouter" LangChain client is stubbed
When I call _create_provider_llm for provider "openrouter" without specifying a model
Then the stubbed openrouter client should be created with argument "model" value "anthropic/claude-sonnet-4-20250514"
@unit @providers @registry
Scenario: _create_provider_llm treats empty default_headers as None for OpenRouter
Given I have a ProviderRegistry with "openrouter" API key set
And the "openrouter" LangChain client is stubbed
When I call _create_provider_llm for provider "openrouter" with model "claude-openrouter-cover" and empty default headers
Then the stubbed openrouter client should not have default_headers
@unit @providers @registry
Scenario: _create_provider_llm forwards extra kwargs to ChatOpenAI for OpenRouter
Given I have a ProviderRegistry with "openrouter" API key set
And the "openrouter" LangChain client is stubbed
When I call _create_provider_llm for provider "openrouter" with model "claude-openrouter-cover" and temperature 0.7
Then the stubbed openrouter client should be created with argument "temperature" value "0.7"
@unit @providers @registry
Scenario: _create_provider_llm raises error when OpenRouter API key is empty
Given I have a ProviderRegistry with no API keys
When I try to call _create_provider_llm for provider "openrouter" with model "claude-openrouter-cover"
Then a provider registry ValueError should be raised
And the provider registry error should mention missing "OPENROUTER_API_KEY" environment variable
@unit @providers @registry
Scenario: _create_provider_llm raises error for unsupported provider
Given I have a ProviderRegistry instance
When I try to call _create_provider_llm for provider "mock" with model "mock-gpt"
Then a provider registry ValueError should be raised
Scenario: _create_provider_instance handles MOCK provider
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set
And I have a ProviderRegistry with no API keys
And the "mock" LangChain client is stubbed
When I call _create_provider_instance for provider "MOCK" with model "mock-gpt"
Then the stubbed mock client should be instantiated
@unit @providers @registry
Scenario: Create AI provider raises error when no provider configured
@@ -434,12 +369,30 @@ Feature: Provider Registry Coverage
Then a provider registry ValueError should be raised
And the provider registry error should mention missing "GOOGLE_API_KEY" environment variable
@unit @providers @registry
Scenario: Create AI provider succeeds for MOCK provider without API key
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set
And I have a ProviderRegistry with no API keys
And CLEVERAGENTS_DEFAULT_MODEL is not set
When I create an AI provider for provider "mock"
Then the provider registry AI provider should be a LangChainChatProvider
And the provider registry AI provider name should be "mock"
And the provider registry AI provider model_id should be "mock-gpt"
@unit @providers @registry
Scenario: Create AI provider for MOCK is rejected without env var
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is not set
And I have a ProviderRegistry with no API keys
When I try to create an AI provider for provider "mock"
Then a provider registry ValueError should be raised
And the provider registry error should mention "not allowed"
@unit @providers @registry
Scenario: Create AI provider succeeds for configured provider string
Given I have a ProviderRegistry with OpenAI API key set
And CLEVERAGENTS_DEFAULT_MODEL is not set
When I create an AI provider for provider "openai"
Then the provider registry AI provider should be an OpenAIChatProvider
Then the provider registry AI provider should be a LangChainChatProvider
And the provider registry AI provider name should be "openai"
And the provider registry AI provider model_id should be "gpt-4o"
@@ -448,7 +401,7 @@ Feature: Provider Registry Coverage
Given I have a ProviderRegistry with Anthropic API key set
And CLEVERAGENTS_DEFAULT_MODEL is not set
When I create an AI provider for provider "anthropic"
Then the provider registry AI provider should be an AnthropicChatProvider
Then the provider registry AI provider should be a LangChainChatProvider
And the provider registry AI provider name should be "anthropic"
And the provider registry AI provider model_id should be "claude-sonnet-4-20250514"
@@ -471,7 +424,7 @@ Feature: Provider Registry Coverage
Given I have a ProviderRegistry with "google" API key set
And CLEVERAGENTS_DEFAULT_MODEL is not set
When I create an AI provider for provider "google"
Then the provider registry AI provider should be a GoogleChatProvider
Then the provider registry AI provider should be a LangChainChatProvider
And the provider registry AI provider name should be "google"
And the provider registry AI provider model_id should be "gemini-2.0-flash"
@@ -480,7 +433,7 @@ Feature: Provider Registry Coverage
Given I have a ProviderRegistry with "openrouter" API key set
And CLEVERAGENTS_DEFAULT_MODEL is not set
When I create an AI provider for provider "openrouter"
Then the provider registry AI provider should be an OpenRouterChatProvider
Then the provider registry AI provider should be a LangChainChatProvider
And the provider registry AI provider name should be "openrouter"
And the provider registry AI provider model_id should be "anthropic/claude-sonnet-4-20250514"
@@ -490,7 +443,7 @@ Feature: Provider Registry Coverage
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
And CLEVERAGENTS_DEFAULT_MODEL is not set
When I create an AI provider without specifying a provider
Then the provider registry AI provider should be an OpenAIChatProvider
Then the provider registry AI provider should be a LangChainChatProvider
And the provider registry AI provider name should be "openai"
@@ -509,6 +462,38 @@ Feature: Provider Registry Coverage
And I call the AI provider llm factory with model "llama-mini-cover"
Then the stubbed LLM factory should be called for ProviderType.GROQ with model "llama-mini-cover"
@unit @providers @registry
Scenario: _create_provider_instance handles MOCK provider and response list
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set
And I have a ProviderRegistry with no API keys
And the "mock" LangChain client is stubbed
When I call _create_provider_instance for provider "MOCK" with model "mock-gpt"
Then the stubbed mock client should be instantiated
And the stubbed mock client responses should be ["mock response"] * 10
@unit @providers @registry
Scenario: API key is forwarded through create_ai_provider closure
Given I have a ProviderRegistry with OpenAI API key set
And the provider registry LLM factory is stubbed
When I create an AI provider for provider "openai"
And I call the AI provider llm factory with model "gpt-mini"
Then the stubbed LLM factory should be called for ProviderType.OPENAI with model "gpt-mini"
And the stubbed LLM factory should be called with api_key present
@unit @providers @registry
Scenario: resolve_provider_by_name rejects mock without env var
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is not set
When I call resolve_provider_by_name with "mock"
Then a provider registry ValueError should be raised
And the provider registry error should mention "not configured"
@unit @providers @registry
Scenario: resolve_provider_by_name resolves mock with env var
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set
And I have a ProviderRegistry with no API keys
When I call resolve_provider_by_name with "mock"
Then the resolved provider should be a LangChainChatProvider
@unit @providers @registry
Scenario: ProviderCapabilities is immutable
Given I import the ProviderCapabilities class
@@ -9,30 +9,26 @@ Feature: Provider Registry Coverage Boost
And CLEVERAGENTS_DEFAULT_PROVIDER env var is cleared
When I request the default provider type from the boost registry
Then the boost result should be ProviderType ANTHROPIC
# Covers lines 335-338: settings_default valid ProviderType but not_configured
@unit @providers @registry
Scenario: Azure deployment resolves from settings when model_id is None
Given a registry with azure API key and deployment setting "my-azure-deploy"
And the Azure LangChain client is stubbed for boost
When I call _create_provider_llm for AZURE with model_id None via boost
When I call _create_provider_instance for AZURE with model_id None via boost
Then the stubbed Azure client deployment_name should be "my-azure-deploy"
# Covers line 502: self._settings.azure_openai_deployment branch
@unit @providers @registry
Scenario: Azure deployment falls back to gpt-4o when settings deployment is also empty
Given a registry with azure API key and no deployment setting
And the Azure LangChain client is stubbed for boost
When I call _create_provider_llm for AZURE with model_id None via boost
When I call _create_provider_instance for AZURE with model_id None via boost
Then the stubbed Azure client deployment_name should be "gpt-4o"
# Covers line 503: final "gpt-4o" fallback
@unit @providers @registry
Scenario: Google create_ai_provider uses DEFAULT_MODELS when model_id is empty string
Given a registry with google API key configured for boost
When I create an AI provider for google with empty string model_id via boost
Then the boost AI provider model_id should be "gemini-2.0-flash"
# Covers line 620: DEFAULT_MODELS.get fallback in Google branch
@unit @providers @registry
Scenario: OpenRouter create_ai_provider raises error when API key is missing
@@ -41,11 +37,37 @@ Feature: Provider Registry Coverage Boost
Then a boost ValueError should be raised
And the boost error message should contain "openrouter"
And the boost error message should contain "not configured"
# Covers lines 632-637: OpenRouter not-configured error path
@unit @providers @registry
Scenario: OpenRouter create_ai_provider uses DEFAULT_MODELS when model_id is empty string
Given a registry with openrouter API key configured for boost
When I create an AI provider for openrouter with empty string model_id via boost
Then the boost AI provider model_id should be "anthropic/claude-sonnet-4-20250514"
# Covers lines 643-644: DEFAULT_MODELS.get fallback in OpenRouter branch
@unit @providers @registry
Scenario: OpenRouter _create_provider_instance with sanitized numeric headers and organization
Given a registry with openrouter API key configured for boost
And the openrouter organization is set to "my-org"
When I call _create_provider_instance for openrouter with model "claude-or" and numeric headers
Then the returned LLM should be recorded with sanitized headers including "X-Debug"
And the openrouter kwargs should include HTTP-Referer "my-org"
@unit @providers @registry
Scenario: _create_provider_instance accepts pre-validated api_key sentinel None
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set
And a registry with no API keys for boost
When I call _create_provider_instance for mock with api_key sentinel None
Then the boost result should be a FakeListLLM instance
And calling it twice yields the same response
@unit @providers @registry
Scenario: Azure endpoint missing raises ValueError
Given a registry with azure API key but no endpoint setting
When I try to call _create_provider_instance for AZURE with model_id None via boost
Then a boost ValueError should be raised
And the boost error message should contain "endpoint"
@unit @providers @registry
Scenario: coerce_optional_str coerces non-string values
When I coerce optional string with integer 42
Then the coerced optional string result should be "42"
@@ -0,0 +1,106 @@
"""Step definitions for _get_base_env thread-safety (TOCTOU fix) scenarios.
These steps exercise the double-checked locking pattern introduced in
``git_tools._get_base_env()`` to eliminate the TOCTOU race condition
where two concurrent threads could both observe ``_BASE_ENV is None``
and write potentially different snapshots.
"""
from __future__ import annotations
import threading
from typing import Any
import cleveragents.tool.builtins.git_tools as _git_tools_mod
from behave import then, when
from cleveragents.tool.builtins.git_tools import _get_base_env
__all__: list[str] = []
# ---------------------------------------------------------------------------
# Whens
# ---------------------------------------------------------------------------
@when("I call _get_base_env twice")
def step_when_call_get_base_env_twice(context: Any) -> None:
# Reset the module-level cache so the test is independent of call order.
original = _git_tools_mod._BASE_ENV
_git_tools_mod._BASE_ENV = None
try:
context.base_env_first = _get_base_env()
context.base_env_second = _get_base_env()
finally:
_git_tools_mod._BASE_ENV = original
@when("I call _get_base_env")
def step_when_call_get_base_env(context: Any) -> None:
original = _git_tools_mod._BASE_ENV
_git_tools_mod._BASE_ENV = None
try:
context.base_env_result = _get_base_env()
finally:
_git_tools_mod._BASE_ENV = original
@when("I call _get_base_env concurrently from 20 threads")
def step_when_call_get_base_env_concurrent(context: Any) -> None:
n_threads = 20
results: list[dict[str, str] | None] = [None] * n_threads
barrier = threading.Barrier(n_threads)
# Reset the module-level cache so all threads race on first initialisation.
original = _git_tools_mod._BASE_ENV
_git_tools_mod._BASE_ENV = None
def worker(idx: int) -> None:
barrier.wait() # synchronise all threads to maximise race probability
results[idx] = _get_base_env()
threads = [threading.Thread(target=worker, args=(i,)) for i in range(n_threads)]
try:
for t in threads:
t.start()
for t in threads:
t.join(timeout=10)
finally:
_git_tools_mod._BASE_ENV = original
context.concurrent_results = results
# ---------------------------------------------------------------------------
# Thens
# ---------------------------------------------------------------------------
@then("both calls should return the same dict object")
def step_then_same_dict_object(context: Any) -> None:
assert context.base_env_first is context.base_env_second, (
"_get_base_env() returned different dict objects on repeated calls; "
"caching is broken"
)
@then('the result should contain key "{key}"')
def step_then_result_contains_key(context: Any, key: str) -> None:
assert key in context.base_env_result, (
f"Expected key '{key}' in _get_base_env() result, "
f"got keys: {list(context.base_env_result.keys())}"
)
@then("all threads should receive the same dict object")
def step_then_all_threads_same_object(context: Any) -> None:
results = context.concurrent_results
assert all(r is not None for r in results), (
"Some threads did not receive a result from _get_base_env()"
)
first = results[0]
for i, r in enumerate(results[1:], start=1):
assert r is first, (
f"Thread {i} received a different dict object than thread 0; "
"double-checked locking is broken"
)
@@ -24,6 +24,9 @@ from cleveragents.providers.registry import (
# ---------------------------------------------------------------------------
_AZURE_ENDPOINT_DEFAULT = object()
def _make_boost_settings(
openai: str | None = None,
anthropic: str | None = None,
@@ -36,7 +39,7 @@ def _make_boost_settings(
together: str | None = None,
default_provider: str | None = None,
default_model: str | None = None,
azure_endpoint: str | None = None,
azure_endpoint: str | None = _AZURE_ENDPOINT_DEFAULT,
azure_api_version: str | None = None,
azure_deployment: str | None = None,
openrouter_org: str | None = None,
@@ -54,8 +57,8 @@ def _make_boost_settings(
settings.together_api_key = together
settings.default_provider = default_provider
settings.default_model = default_model
if azure and not azure_endpoint:
azure_endpoint = "https://example.openai.azure.com"
if azure_endpoint is _AZURE_ENDPOINT_DEFAULT:
azure_endpoint = "https://example.openai.azure.com" if azure else None
settings.azure_openai_endpoint = azure_endpoint
settings.azure_openai_api_version = azure_api_version
settings.azure_openai_deployment = azure_deployment
@@ -168,6 +171,18 @@ def step_boost_openrouter_registry(context: Any) -> None:
)
@given('the openrouter organization is set to "{org}"')
def step_boost_openrouter_org(context: Any, org: str) -> None:
context.boost_registry._settings.openrouter_organization = org
@given("a registry with azure API key but no endpoint setting")
def step_boost_azure_no_endpoint(context: Any) -> None:
context.boost_registry = ProviderRegistry(
settings=_make_boost_settings(azure="sk-azure-test", azure_endpoint=None)
)
# ---------------------------------------------------------------------------
# When
# ---------------------------------------------------------------------------
@@ -178,13 +193,63 @@ def step_boost_get_default_provider_type(context: Any) -> None:
context.boost_result = context.boost_registry.get_default_provider_type()
@when("I call _create_provider_llm for AZURE with model_id None via boost")
def step_boost_create_azure_llm_no_model(context: Any) -> None:
context.boost_result = context.boost_registry._create_provider_llm(
@when("I call _create_provider_instance for AZURE with model_id None via boost")
def step_boost_create_azure_instance_no_model(context: Any) -> None:
context.boost_result = context.boost_registry._create_provider_instance(
ProviderType.AZURE, None
)
@when(
'I call _create_provider_instance for openrouter with model "{model}" and numeric headers'
)
def step_boost_create_openrouter_with_headers(context: Any, model: str) -> None:
original = sys.modules.get("langchain_openai")
calls: list[dict[str, Any]] = []
class FakeChatOpenAI:
def __init__(self, **kwargs: Any) -> None:
calls.append(kwargs)
stub = types.ModuleType("langchain_openai")
stub.ChatOpenAI = FakeChatOpenAI
stub.AzureChatOpenAI = lambda **_: None
sys.modules["langchain_openai"] = stub
context._boost_openrouter_calls = calls
def _restore() -> None:
if original is None:
sys.modules.pop("langchain_openai", None)
else:
sys.modules["langchain_openai"] = original
context._boost_cleanup = getattr(context, "_boost_cleanup", [])
context._boost_cleanup.append(_restore)
context.boost_result = context.boost_registry._create_provider_instance(
ProviderType.OPENROUTER,
model,
default_headers={123: 456, "X-Debug": 789},
)
@when("I call _create_provider_instance for mock with api_key sentinel None")
def step_boost_create_mock_with_sentinel(context: Any) -> None:
context.boost_result = context.boost_registry._create_provider_instance(
ProviderType.MOCK, None, __api_key_sentinel=None
)
@when("I try to call _create_provider_instance for AZURE with model_id None via boost")
def step_boost_try_create_azure_no_model(context: Any) -> None:
try:
context.boost_registry._create_provider_instance(ProviderType.AZURE, None)
context.boost_error = None
except ValueError as exc:
context.boost_error = exc
@when("I create an AI provider for google with empty string model_id via boost")
def step_boost_google_ai_provider_empty_model(context: Any) -> None:
# Ensure env vars don't interfere
@@ -215,6 +280,13 @@ def step_boost_openrouter_ai_provider_empty_model(context: Any) -> None:
)
@when("I coerce optional string with integer {value:d}")
def step_boost_coerce_optional_str(context: Any, value: int) -> None:
from cleveragents.providers.registry import _coerce_optional_str
context.coerced_value = _coerce_optional_str(value)
# ---------------------------------------------------------------------------
# Then
# ---------------------------------------------------------------------------
@@ -256,3 +328,51 @@ def step_boost_error_contains(context: Any, fragment: str) -> None:
assert fragment.lower() in message, (
f"Expected error to contain {fragment!r}, got: {message!r}"
)
@then('the returned LLM should be recorded with sanitized headers including "{header}"')
def step_boost_openrouter_sanitized_headers(context: Any, header: str) -> None:
calls = getattr(context, "_boost_openrouter_calls", [])
assert calls, "Expected ChatOpenAI to be recorded"
kwargs = calls[-1]
headers = kwargs.get("default_headers", {})
assert header in headers, f"Expected {header!r} in headers, got {headers!r}"
# Verify numeric keys were coerced to strings
assert "123" in headers, "Expected key '123' (coerced from int) in headers"
@then('the openrouter kwargs should include HTTP-Referer "{expected}"')
def step_boost_openrouter_referer(context: Any, expected: str) -> None:
calls = getattr(context, "_boost_openrouter_calls", [])
assert calls, "Expected ChatOpenAI to be recorded"
kwargs = calls[-1]
headers = kwargs.get("default_headers", {})
assert headers.get("HTTP-Referer") == expected, (
f"Expected HTTP-Referer={expected!r}, got {headers.get('HTTP-Referer')!r}"
)
@then("the boost result should be a FakeListLLM instance")
def step_boost_result_is_fake_list_llm(context: Any) -> None:
from langchain_community.llms import FakeListLLM
assert isinstance(context.boost_result, FakeListLLM), (
f"Expected FakeListLLM, got {type(context.boost_result)}"
)
@then("calling it twice yields the same response")
def step_boost_llm_invokes_twice(context: Any) -> None:
llm = context.boost_result
response1 = llm.invoke("test prompt")
response2 = llm.invoke("test prompt 2")
assert response1 == response2 == "mock response", (
f"Expected 'mock response' for both, got {response1!r} and {response2!r}"
)
@then('the coerced optional string result should be "{expected}"')
def step_boost_coerced_result(context: Any, expected: str) -> None:
assert getattr(context, "coerced_value", None) == expected, (
f"Expected {expected!r}, got {getattr(context, 'coerced_value', None)!r}"
)
+151 -57
View File
@@ -9,11 +9,7 @@ from unittest.mock import MagicMock
from behave import given, then, when # type: ignore[import-untyped]
from cleveragents.providers.llm.anthropic_provider import AnthropicChatProvider
from cleveragents.providers.llm.google_provider import GoogleChatProvider
from cleveragents.providers.llm.langchain_chat_provider import LangChainChatProvider
from cleveragents.providers.llm.openai_provider import OpenAIChatProvider
from cleveragents.providers.llm.openrouter_provider import OpenRouterChatProvider
from cleveragents.providers.registry import (
ProviderCapabilities,
ProviderInfo,
@@ -139,6 +135,7 @@ LANGCHAIN_CLIENT_SPECS: dict[str, tuple[str, str]] = {
"groq": ("langchain_groq", "ChatGroq"),
"together": ("langchain_together", "ChatTogether"),
"cohere": ("langchain_cohere", "ChatCohere"),
"mock": ("langchain_community.llms", "FakeListLLM"),
}
@@ -301,6 +298,18 @@ def step_impl_unset_default_model_env(context: Any) -> None:
os.environ.pop("CLEVERAGENTS_DEFAULT_MODEL", None)
@given("CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set")
def step_impl_allow_mock_provider_set(context: Any) -> None:
context.original_allow_mock_env = os.environ.get("CLEVERAGENTS_ALLOW_MOCK_PROVIDER")
os.environ["CLEVERAGENTS_ALLOW_MOCK_PROVIDER"] = "true"
@given("CLEVERAGENTS_ALLOW_MOCK_PROVIDER is not set")
def step_impl_allow_mock_provider_unset(context: Any) -> None:
context.original_allow_mock_env = os.environ.get("CLEVERAGENTS_ALLOW_MOCK_PROVIDER")
os.environ.pop("CLEVERAGENTS_ALLOW_MOCK_PROVIDER", None)
@given("langchain OpenAI client is stubbed")
def step_impl_stub_langchain_openai(context: Any) -> None:
module_name, class_name = LANGCHAIN_CLIENT_SPECS["openai"]
@@ -326,7 +335,9 @@ def step_impl_stub_named_langchain_client(context: Any, provider_name: str) -> N
@given("the provider registry LLM factory is stubbed")
def step_impl_stub_provider_factory(context: Any) -> None:
context.stubbed_llm = object()
context.registry._create_provider_llm = MagicMock(return_value=context.stubbed_llm)
context.registry._create_provider_instance = MagicMock(
return_value=context.stubbed_llm
)
# -- When -------------------------------------------------------------------
@@ -337,12 +348,6 @@ def step_impl_create_registry_default(context: Any) -> None:
context.registry = context.ProviderRegistry()
@when("I coerce optional string with integer {value:d}")
def step_impl_coerce_optional_string_with_integer(context: Any, value: int) -> None:
coerce_helper = getattr(context, "coerce_optional_str", _coerce_optional_str)
context.coerced_value = coerce_helper(value)
@when("I create a ProviderCapabilities with all fields")
def step_impl_capabilities_all_fields(context: Any) -> None:
context.capabilities = context.ProviderCapabilities(
@@ -498,67 +503,75 @@ def step_impl_call_create_llm_no_model(context: Any, value: str) -> None:
context.created_llm = context.registry.create_llm(provider_type=value)
@when('I call _create_provider_llm for provider "{value}" with model "{model}"')
def step_impl_call_private_llm(context: Any, value: str, model: str) -> None:
@when('I call _create_provider_instance for provider "{value}" with model "{model}"')
def step_impl_call_private_instance(context: Any, value: str, model: str) -> None:
provider_enum = _ensure_provider_type(context)
provider_type = getattr(provider_enum, value.upper())
context.created_llm = context.registry._create_provider_llm(provider_type, model)
context.created_llm = context.registry._create_provider_instance(
provider_type, model
)
@when(
'I call _create_provider_llm for provider "{value}" with model "{model}" and custom default headers'
'I call _create_provider_instance for provider "{value}" with model "{model}" and custom default headers'
)
def step_impl_call_private_llm_with_headers(
def step_impl_call_private_instance_with_headers(
context: Any, value: str, model: str
) -> None:
provider_enum = _ensure_provider_type(context)
provider_type = getattr(provider_enum, value.upper())
default_headers = getattr(context, "openrouter_default_headers", None)
assert default_headers is not None, "OpenRouter default headers were not configured"
context.created_llm = context.registry._create_provider_llm(
context.created_llm = context.registry._create_provider_instance(
provider_type, model, default_headers=default_headers
)
@when('I call _create_provider_llm for provider "{value}" without specifying a model')
def step_impl_call_private_llm_no_model(context: Any, value: str) -> None:
@when(
'I call _create_provider_instance for provider "{value}" without specifying a model'
)
def step_impl_call_private_instance_no_model(context: Any, value: str) -> None:
provider_enum = _ensure_provider_type(context)
provider_type = getattr(provider_enum, value.upper())
context.created_llm = context.registry._create_provider_llm(provider_type, None)
context.created_llm = context.registry._create_provider_instance(
provider_type, None
)
@when(
'I call _create_provider_llm for provider "{value}" with model "{model}" and empty default headers'
'I call _create_provider_instance for provider "{value}" with model "{model}" and empty default headers'
)
def step_impl_call_private_llm_empty_headers(
def step_impl_call_private_instance_empty_headers(
context: Any, value: str, model: str
) -> None:
provider_enum = _ensure_provider_type(context)
provider_type = getattr(provider_enum, value.upper())
context.created_llm = context.registry._create_provider_llm(
context.created_llm = context.registry._create_provider_instance(
provider_type, model, default_headers={}
)
@when(
'I call _create_provider_llm for provider "{value}" with model "{model}" and temperature {temperature}'
'I call _create_provider_instance for provider "{value}" with model "{model}" and temperature {temperature}'
)
def step_impl_call_private_llm_with_temperature(
def step_impl_call_private_instance_with_temperature(
context: Any, value: str, model: str, temperature: str
) -> None:
provider_enum = _ensure_provider_type(context)
provider_type = getattr(provider_enum, value.upper())
context.created_llm = context.registry._create_provider_llm(
context.created_llm = context.registry._create_provider_instance(
provider_type, model, temperature=float(temperature)
)
@when('I try to call _create_provider_llm for provider "{value}" with model "{model}"')
def step_impl_try_private_llm(context: Any, value: str, model: str) -> None:
@when(
'I try to call _create_provider_instance for provider "{value}" with model "{model}"'
)
def step_impl_try_private_instance(context: Any, value: str, model: str) -> None:
provider_enum = _ensure_provider_type(context)
provider_type = getattr(provider_enum, value.upper())
try:
context.registry._create_provider_llm(provider_type, model)
context.registry._create_provider_instance(provider_type, model)
context.error = None
except Exception as exc: # pragma: no cover - defensive path
context.error = exc
@@ -630,13 +643,6 @@ def step_impl_registry_has_providers(context: Any) -> None:
assert len(context.registry.get_all_providers()) > 0
@then('the coerced optional string result should be "{expected}"')
def step_impl_coerced_optional_string_result(context: Any, expected: str) -> None:
assert getattr(context, "coerced_value", None) == expected, (
f"Expected coerced value {expected!r}, got {getattr(context, 'coerced_value', None)!r}"
)
@then('ProviderType should have {enum_name} value "{value}"')
def step_impl_provider_type_values(context: Any, enum_name: str, value: str) -> None:
enum_member = getattr(_ensure_provider_type(context), enum_name.upper())
@@ -816,24 +822,14 @@ def step_impl_ai_provider_is_langchain(context: Any) -> None:
assert isinstance(context.ai_provider, LangChainChatProvider)
@then("the provider registry AI provider should be an OpenAIChatProvider")
def step_impl_ai_provider_is_openai(context: Any) -> None:
assert isinstance(context.ai_provider, OpenAIChatProvider)
@then("the provider registry AI provider should be an AnthropicChatProvider")
def step_impl_ai_provider_is_anthropic(context: Any) -> None:
assert isinstance(context.ai_provider, AnthropicChatProvider)
@then("the provider registry AI provider should be a GoogleChatProvider")
def step_impl_ai_provider_is_google(context: Any) -> None:
assert isinstance(context.ai_provider, GoogleChatProvider)
@then("the provider registry AI provider should be an OpenRouterChatProvider")
def step_impl_ai_provider_is_openrouter(context: Any) -> None:
assert isinstance(context.ai_provider, OpenRouterChatProvider)
@then("the stubbed mock client should be instantiated")
def step_impl_stubbed_mock_client_instantiated(context: Any) -> None:
stubbed_clients = getattr(context, "stubbed_clients", {})
client_data = stubbed_clients.get("mock")
assert client_data, "No stubbed client recorded for mock"
calls = client_data["calls"]
assert calls, "Expected FakeListLLM to be instantiated"
assert isinstance(context.created_llm, client_data["class"])
@then('the provider registry AI provider name should be "{value}"')
@@ -854,8 +850,8 @@ def step_impl_stubbed_llm_factory_called(
) -> None:
provider_enum = _ensure_provider_type(context)
expected_type = getattr(provider_enum, enum_name.upper())
context.registry._create_provider_llm.assert_called_once()
args, kwargs = context.registry._create_provider_llm.call_args
context.registry._create_provider_instance.assert_called_once()
args, kwargs = context.registry._create_provider_instance.call_args
assert args[:2] == (expected_type, model)
if kwargs:
# max_retries is propagated by create_ai_provider to keep retry semantics aligned
@@ -863,6 +859,10 @@ def step_impl_stubbed_llm_factory_called(
assert max_retries is None or isinstance(max_retries, int), (
f"Expected max_retries int or None, got {max_retries!r}"
)
# api_key should be forwarded (even None) so _get_api_key is not called twice
assert "__api_key_sentinel" in kwargs or "api_key" in kwargs, (
f"Expected api_key or __api_key_sentinel in factory kwargs, got {sorted(kwargs.keys())!r}"
)
assert context.created_llm is context.stubbed_llm
@@ -999,6 +999,100 @@ def step_impl_default_models(context: Any, provider: str, model: str) -> None:
assert context.default_models[enum_member] == model
# -- Hooks ------------------------------------------------------------------
@then('the provider registry error should mention "{message}"')
def step_impl_error_mentions_message(context: Any, message: str) -> None:
assert context.error is not None
assert message.lower() in str(context.error).lower(), (
f"Expected error to mention {message!r}, got {str(context.error)!r}"
)
@then("the created LLM should be a FakeListLLM instance")
def step_impl_created_llm_is_fake_list_llm(context: Any) -> None:
from langchain_community.llms import FakeListLLM
assert isinstance(context.created_llm, FakeListLLM), (
f"Expected FakeListLLM, got {type(context.created_llm)}"
)
@then("the provider registry result should contain only the MOCK provider")
def step_impl_result_contains_only_mock(context: Any) -> None:
provider_types = [p.provider_type for p in context.result]
provider_enum = _ensure_provider_type(context)
assert provider_types == [provider_enum.MOCK], (
f"Expected only MOCK, got {provider_types}"
)
@then("the provider registry result should contain the MOCK provider")
def step_impl_result_contains_mock(context: Any) -> None:
provider_types = [p.provider_type for p in context.result]
provider_enum = _ensure_provider_type(context)
assert provider_enum.MOCK in provider_types, (
f"Expected MOCK in list, got {provider_types}"
)
@then("the stubbed {provider_name} client responses should be {expected_responses}")
def step_impl_stubbed_client_responses(
context: Any, provider_name: str, expected_responses: str
) -> None:
provider_key = provider_name.lower()
stubbed_clients = getattr(context, "stubbed_clients", {})
assert provider_key in stubbed_clients, (
f"No stubbed client recorded for {provider_name}"
)
client_data = stubbed_clients[provider_key]
calls = client_data["calls"]
assert calls, f"Expected {provider_name} client to be instantiated"
last_call = calls[-1]
actual = last_call["kwargs"].get("responses")
if actual is None:
actual = last_call.get("args")
expected = eval(expected_responses) # e.g. ["mock response"] * 10
assert actual == expected, f"Expected responses={expected!r}, got {actual!r}"
assert isinstance(context.created_llm, client_data["class"])
@then("the stubbed LLM factory should be called with api_key present")
def step_impl_stubbed_llm_factory_api_key(context: Any) -> None:
context.registry._create_provider_instance.assert_called()
_args, kwargs = context.registry._create_provider_instance.call_args
assert "__api_key_sentinel" in kwargs or "api_key" in kwargs, (
f"Expected api_key or __api_key_sentinel in factory kwargs, got {sorted(kwargs.keys())!r}"
)
@then('the FakeListLLM produces response "{response}" on invoke')
def step_impl_fake_list_llm_invokes(context: Any, response: str) -> None:
from langchain_community.llms import FakeListLLM
llm = context.created_llm
assert isinstance(llm, FakeListLLM), f"Expected FakeListLLM, got {type(llm)}"
assert llm.invoke("test prompt") == response, (
f"Expected FakeListLLM to produce {response!r}"
)
@when('I call resolve_provider_by_name with "{name}"')
def step_impl_resolve_provider_by_name(context: Any, name: str) -> None:
from cleveragents.providers.registry import (
resolve_provider_by_name,
reset_provider_registry,
)
reset_provider_registry()
try:
context.resolved_provider = resolve_provider_by_name(name)
context.error = None
except ValueError as exc:
context.error = exc
@then("the resolved provider should be a LangChainChatProvider")
def step_impl_resolved_provider_is_langchain(context: Any) -> None:
assert isinstance(context.resolved_provider, LangChainChatProvider)
# Hook logic lives in features/environment.py to ensure scenarios reset state.
@@ -0,0 +1,389 @@
"""Step definitions for features/tdd_mcp_adapter_rlock_concurrency.feature.
Tests that MCPToolAdapter releases its RLock before making transport calls,
allowing concurrent operations to proceed without being blocked.
Bug #10512: MCPToolAdapter holds RLock during entire transport call,
blocking concurrent operations.
"""
from __future__ import annotations
import threading
import time
from typing import Any
from behave import given, then, when
from behave.runner import Context
from cleveragents.mcp.adapter import (
MCPServerConfig,
MCPToolAdapter,
MCPToolResult,
)
from features.mocks.mock_mcp_transport import MockMCPTransport
def _mock_tool(
name: str, desc: str = "", schema: dict[str, Any] | None = None
) -> dict[str, Any]:
return {
"name": name,
"description": desc or f"Mock tool {name}",
"inputSchema": schema or {},
}
class SlowMCPTransport(MockMCPTransport):
"""Transport that introduces a configurable delay during call()."""
def __init__(
self,
delay: float,
tools: list[dict[str, Any]] | None = None,
) -> None:
super().__init__(tools=tools)
self._delay = delay
def call(self, method: str, params: dict[str, Any]) -> dict[str, Any]:
# Use the real time.sleep (not the capped test version) to ensure
# the delay is meaningful for concurrency testing.
_real_sleep = getattr(time, "_original_sleep", time.sleep)
_real_sleep(self._delay)
return super().call(method, params)
class ConcurrencyTrackingTransport(MockMCPTransport):
"""Transport that tracks the maximum number of concurrent in-flight calls.
Uses a threading.Lock-protected counter to record how many calls are
executing simultaneously. The ``max_concurrent`` attribute reflects the
peak concurrency observed across all calls.
If the adapter holds its RLock during transport calls, only one call can
be in-flight at a time (max_concurrent == 1). If the lock is released
before the transport call, multiple calls can overlap (max_concurrent > 1).
"""
def __init__(
self,
delay: float,
tools: list[dict[str, Any]] | None = None,
) -> None:
super().__init__(tools=tools)
self._delay = delay
self._counter_lock = threading.Lock()
self._current_concurrent: int = 0
self.max_concurrent: int = 0
def call(self, method: str, params: dict[str, Any]) -> dict[str, Any]:
with self._counter_lock:
self._current_concurrent += 1
if self._current_concurrent > self.max_concurrent:
self.max_concurrent = self._current_concurrent
# Use the real time.sleep (not the capped test version) to ensure
# the delay is meaningful for concurrency testing.
_real_sleep = getattr(time, "_original_sleep", time.sleep)
_real_sleep(self._delay)
with self._counter_lock:
self._current_concurrent -= 1
return super().call(method, params)
class LockCheckingInvokeTransport(MockMCPTransport):
"""Transport that checks whether the adapter's RLock is held during call()."""
def __init__(
self,
adapter_lock: threading.RLock,
tools: list[dict[str, Any]] | None = None,
) -> None:
super().__init__(tools=tools)
self._adapter_lock = adapter_lock
self.lock_was_held_during_call = False
def call(self, method: str, params: dict[str, Any]) -> dict[str, Any]:
if method == "tools/call":
# Try to acquire the lock from a different thread.
# If the lock is held by the calling thread, a different thread
# will fail to acquire it (non-blocking).
acquired_event = threading.Event()
lock_held = [True]
def _try_acquire() -> None:
got_it = self._adapter_lock.acquire(blocking=False)
if got_it:
lock_held[0] = False
self._adapter_lock.release()
acquired_event.set()
t = threading.Thread(target=_try_acquire, daemon=True)
t.start()
t.join(timeout=1.0)
acquired_event.wait(timeout=1.0)
self.lock_was_held_during_call = lock_held[0]
return super().call(method, params)
class LockCheckingDiscoveryTransport(MockMCPTransport):
"""Transport that checks whether the adapter's RLock is held during tools/list."""
def __init__(
self,
adapter_lock: threading.RLock,
tools: list[dict[str, Any]] | None = None,
) -> None:
super().__init__(tools=tools)
self._adapter_lock = adapter_lock
self.lock_was_held_during_call = False
def call(self, method: str, params: dict[str, Any]) -> dict[str, Any]:
if method == "tools/list":
acquired_event = threading.Event()
lock_held = [True]
def _try_acquire() -> None:
got_it = self._adapter_lock.acquire(blocking=False)
if got_it:
lock_held[0] = False
self._adapter_lock.release()
acquired_event.set()
t = threading.Thread(target=_try_acquire, daemon=True)
t.start()
t.join(timeout=1.0)
acquired_event.wait(timeout=1.0)
self.lock_was_held_during_call = lock_held[0]
return super().call(method, params)
# ---------------------------------------------------------------
# Given Steps
# ---------------------------------------------------------------
@given(
'a connected MCP adapter with a slow transport tool "{tool_name}" taking {delay:f} seconds'
)
def step_slow_transport_tool(context: Context, tool_name: str, delay: float) -> None:
tools = [_mock_tool(tool_name)]
config = MCPServerConfig(name="test-server", transport="stdio", command="echo")
context.mcp_transport = SlowMCPTransport(delay=delay, tools=tools)
context.mcp_adapter = MCPToolAdapter(config=config, transport=context.mcp_transport)
context.mcp_adapter.connect()
context.mcp_adapter.discover_tools()
@given(
'a connected MCP adapter with a concurrency-tracking transport tool "{tool_name}" taking {delay:f} seconds'
)
def step_concurrency_tracking_transport_tool(
context: Context, tool_name: str, delay: float
) -> None:
tools = [_mock_tool(tool_name)]
config = MCPServerConfig(name="test-server", transport="stdio", command="echo")
context.mcp_transport = ConcurrencyTrackingTransport(delay=delay, tools=tools)
context.mcp_adapter = MCPToolAdapter(config=config, transport=context.mcp_transport)
context.mcp_adapter.connect()
context.mcp_adapter.discover_tools()
@given(
"a connected MCP adapter with a slow discovery transport taking {delay:f} seconds"
)
def step_slow_discovery_transport(context: Context, delay: float) -> None:
tools = [_mock_tool("tool_0"), _mock_tool("tool_1")]
config = MCPServerConfig(name="test-server", transport="stdio", command="echo")
context.mcp_transport = SlowMCPTransport(delay=delay, tools=tools)
context.mcp_adapter = MCPToolAdapter(config=config, transport=context.mcp_transport)
context.mcp_adapter.connect()
@given(
"a connected MCP adapter with a concurrency-tracking discovery transport taking {delay:f} seconds"
)
def step_concurrency_tracking_discovery_transport(
context: Context, delay: float
) -> None:
tools = [_mock_tool("tool_0"), _mock_tool("tool_1")]
config = MCPServerConfig(name="test-server", transport="stdio", command="echo")
context.mcp_transport = ConcurrencyTrackingTransport(delay=delay, tools=tools)
context.mcp_adapter = MCPToolAdapter(config=config, transport=context.mcp_transport)
context.mcp_adapter.connect()
@given('a connected MCP adapter with a lock-checking transport tool "{tool_name}"')
def step_lock_checking_invoke_transport(context: Context, tool_name: str) -> None:
tools = [_mock_tool(tool_name)]
config = MCPServerConfig(name="test-server", transport="stdio", command="echo")
# Create adapter first to get its lock, then set up transport
transport_placeholder = MockMCPTransport(tools=tools)
context.mcp_adapter = MCPToolAdapter(config=config, transport=transport_placeholder)
# Now create the lock-checking transport with the adapter's actual lock
lock_transport = LockCheckingInvokeTransport(
adapter_lock=context.mcp_adapter._lock,
tools=tools,
)
# Replace the transport
context.mcp_adapter._transport = lock_transport
context.mcp_transport = lock_transport
context.mcp_adapter._connected = True
context.mcp_adapter._capabilities = {"tools": True}
context.mcp_adapter.discover_tools()
@given("a connected MCP adapter with a lock-checking discovery transport")
def step_lock_checking_discovery_transport(context: Context) -> None:
tools = [_mock_tool("tool_0"), _mock_tool("tool_1")]
config = MCPServerConfig(name="test-server", transport="stdio", command="echo")
transport_placeholder = MockMCPTransport(tools=tools)
context.mcp_adapter = MCPToolAdapter(config=config, transport=transport_placeholder)
lock_transport = LockCheckingDiscoveryTransport(
adapter_lock=context.mcp_adapter._lock,
tools=tools,
)
context.mcp_adapter._transport = lock_transport
context.mcp_transport = lock_transport
context.mcp_adapter._connected = True
context.mcp_adapter._capabilities = {"tools": True}
# ---------------------------------------------------------------
# When Steps
# ---------------------------------------------------------------
@when('I invoke "{tool_name}" concurrently from {count:d} threads')
def step_concurrent_invoke(context: Context, tool_name: str, count: int) -> None:
# Use threading.Thread directly instead of ThreadPoolExecutor to avoid
# deadlocks in forked worker processes (behave-parallel uses fork).
# ThreadPoolExecutor holds internal locks that can be in a locked state
# after fork(), causing child processes to deadlock on acquisition.
results: list[MCPToolResult | None] = [None] * count
errors: list[BaseException | None] = [None] * count
errors_lock = threading.Lock()
def _invoke(idx: int) -> None:
try:
results[idx] = context.mcp_adapter.invoke(tool_name, {})
except BaseException as exc:
with errors_lock:
errors[idx] = exc
threads = [threading.Thread(target=_invoke, args=(i,)) for i in range(count)]
for t in threads:
t.start()
for t in threads:
t.join()
# Re-raise the first error encountered, if any
for err in errors:
if err is not None:
raise err
context.concurrent_invoke_results = [r for r in results if r is not None]
@when("I call discover_tools concurrently from {count:d} threads")
def step_concurrent_discover(context: Context, count: int) -> None:
# Use threading.Thread directly instead of ThreadPoolExecutor to avoid
# deadlocks in forked worker processes (behave-parallel uses fork).
# ThreadPoolExecutor holds internal locks that can be in a locked state
# after fork(), causing child processes to deadlock on acquisition.
results: list[Any] = []
errors: list[Exception] = []
collect_lock = threading.Lock()
def _discover() -> None:
try:
result = context.mcp_adapter.discover_tools()
with collect_lock:
results.append(result)
except Exception as exc:
with collect_lock:
errors.append(exc)
threads = [threading.Thread(target=_discover) for _ in range(count)]
for t in threads:
t.start()
for t in threads:
t.join()
context.concurrent_discover_results = results
context.concurrent_discover_errors = errors
# ---------------------------------------------------------------
# Then Steps
# ---------------------------------------------------------------
@then("all {count:d} concurrent invocations should succeed")
def step_all_invocations_succeed(context: Context, count: int) -> None:
results = context.concurrent_invoke_results
assert len(results) == count, f"Expected {count} results, got {len(results)}"
for i, result in enumerate(results):
assert result.success, f"Invocation {i} failed: {result.error}"
@then(
"at least {min_concurrent:d} invocations should have been in-flight simultaneously"
)
def step_at_least_n_invocations_concurrent(
context: Context, min_concurrent: int
) -> None:
transport = context.mcp_transport
actual = transport.max_concurrent
assert actual >= min_concurrent, (
f"Expected at least {min_concurrent} concurrent in-flight invocations, "
f"but observed max_concurrent={actual}. "
f"The adapter's RLock appears to be held during transport calls, "
f"serializing concurrent operations."
)
@then("all {count:d} concurrent discoveries should succeed")
def step_all_discoveries_succeed(context: Context, count: int) -> None:
results = context.concurrent_discover_results
errors = context.concurrent_discover_errors
assert not errors, f"Expected no errors, got {len(errors)}: {errors}"
assert len(results) == count, f"Expected {count} results, got {len(results)}"
@then(
"at least {min_concurrent:d} discoveries should have been in-flight simultaneously"
)
def step_at_least_n_discoveries_concurrent(
context: Context, min_concurrent: int
) -> None:
transport = context.mcp_transport
actual = transport.max_concurrent
assert actual >= min_concurrent, (
f"Expected at least {min_concurrent} concurrent in-flight discoveries, "
f"but observed max_concurrent={actual}. "
f"The adapter's RLock appears to be held during transport calls, "
f"serializing concurrent operations."
)
@then("the lock should not have been held during the transport call")
def step_lock_not_held_during_invoke(context: Context) -> None:
transport = context.mcp_transport
assert not transport.lock_was_held_during_call, (
"The adapter's RLock was held during the transport.call() invocation. "
"The lock should be released before making the transport call."
)
@then("the lock should not have been held during the discovery transport call")
def step_lock_not_held_during_discovery(context: Context) -> None:
transport = context.mcp_transport
assert not transport.lock_was_held_during_call, (
"The adapter's RLock was held during the transport.call('tools/list') invocation. "
"The lock should be released before making the transport call."
)
@@ -0,0 +1,52 @@
@tdd_issue @tdd_issue_10512
Feature: MCPToolAdapter releases RLock during transport calls
As a developer using MCPToolAdapter concurrently
I want the adapter to release its RLock before making transport calls
So that concurrent operations are not blocked by slow transport calls
# -------------------------------------------------------------------
# Bug #10512: MCPToolAdapter holds RLock during entire transport call,
# blocking concurrent operations.
#
# The fix releases the lock before the transport.call() and
# re-acquires it only when shared state must be mutated.
# -------------------------------------------------------------------
Scenario: Concurrent invoke calls are not blocked by a slow transport
Given a connected MCP adapter with a concurrency-tracking transport tool "slow_op" taking 0.05 seconds
When I invoke "slow_op" concurrently from 3 threads
Then all 3 concurrent invocations should succeed
And at least 2 invocations should have been in-flight simultaneously
Scenario: Concurrent discover_tools calls are not blocked by a slow transport
Given a connected MCP adapter with a concurrency-tracking discovery transport taking 0.05 seconds
When I call discover_tools concurrently from 3 threads
Then all 3 concurrent discoveries should succeed
And at least 2 discoveries should have been in-flight simultaneously
Scenario: Lock is not held during invoke transport call
Given a connected MCP adapter with a lock-checking transport tool "check_lock"
When I invoke "check_lock" with arguments {}
Then the invocation should succeed
And the lock should not have been held during the transport call
Scenario: Lock is not held during discover_tools transport call
Given a connected MCP adapter with a lock-checking discovery transport
When I discover tools from the adapter
Then the lock should not have been held during the discovery transport call
Scenario: Invoke still validates under lock before transport call
Given a connected MCP adapter with a slow transport tool "slow_op" taking 0.1 seconds
When I invoke "nonexistent_tool" with arguments {}
Then the invocation should fail
And the invocation error should mention "not found"
Scenario: Invoke on disconnected adapter still raises under lock
Given an MCP adapter with a mock transport
When I invoke MCP tool "any_tool" while disconnected
Then the adapter error should mention "not connected"
Scenario: Discover on disconnected adapter still raises under lock
Given an MCP adapter with a mock transport
When I discover tools expecting an error
Then the adapter error should mention "not connected"
+3 -1
View File
@@ -30,6 +30,7 @@ nav:
- Invariant Reconciliation: modules/invariant-reconciliation.md
- ACMS Context Hydration: modules/context-hydration.md
- Git Worktree Sandbox: modules/git-worktree-sandbox.md
- Devcontainer Auto-Discovery: modules/devcontainer-discovery.md
- Showcase:
- Overview: showcase/index.md
- CLI Tools: showcase/cli-tools/
@@ -60,6 +61,7 @@ nav:
- Reference/Command Input & Sessions: tui/input-and-sessions.md
- Configuration, Key Bindings & Integration: tui/configuration-and-integration.md
- FAQ: faq.md
- Quick Start: quickstart.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- Reference: reference/
@@ -92,7 +94,7 @@ nav:
- ADR-025 Observability & Logging: adr/ADR-025-observability-and-logging.md
- ADR-026 Agent-to-Agent Protocol (A2A): adr/ADR-026-agent-client-protocol.md
- ADR-027 Language Server Protocol (LSP) Integration: adr/ADR-027-language-server-protocol.md
- ADR-028 Agent Skills Standard (AgentSkills.io): adr/ADR-028-agent-skills-standard.md
- ADR-028 Skill/Tool Abstraction Definition: adr/ADR-028-agent-skills-standard.md
- ADR-029 Model Context Protocol (MCP) Adoption: adr/ADR-029-model-context-protocol.md
- ADR-030 Skill Abstraction Definition: adr/ADR-030-skill-abstraction-definition.md
- ADR-031 Actor Abstraction Definition: adr/ADR-031-actor-abstraction-definition.md
+1 -1
View File
@@ -9,7 +9,7 @@ Suite Teardown Cleanup Test Environment
*** Test Cases ***
Noxfile Contains Coverage Threshold Constant
[Documentation] Verify COVERAGE_THRESHOLD = 96.5 is defined in noxfile.py
[Tags] coverage config
[Tags] coverage config tdd_issue tdd_issue_4305
${content}= Get File ${WORKSPACE}/noxfile.py
Should Contain ${content} COVERAGE_THRESHOLD = 96.5
+85 -62
View File
@@ -174,6 +174,12 @@ class MCPTransport:
class MCPToolAdapter:
"""Adapter bridging an MCP server into the CleverAgents ToolRegistry.
Thread-safety strategy: the adapter's ``RLock`` protects shared
mutable state (``_connected``, ``_capabilities``, ``_tools``,
``_notification_listeners``). The lock is **released** before
making transport calls (``_transport.call()``) so that concurrent
operations are not blocked by slow network I/O. See bug #10512.
Parameters
----------
config:
@@ -411,6 +417,10 @@ class MCPToolAdapter:
) -> list[MCPToolDescriptor]:
"""Enumerate tools from the connected MCP server.
The adapter's RLock is held only for state validation and
mutation it is released before the transport call so that
concurrent operations are not blocked by slow network I/O.
Parameters
----------
tool_filter:
@@ -421,6 +431,7 @@ class MCPToolAdapter:
RuntimeError
If the adapter is not connected.
"""
# Phase 1: Validate connection state under lock.
with self._lock:
if not self._connected:
msg = (
@@ -428,25 +439,30 @@ class MCPToolAdapter:
f"Call connect() first."
)
raise RuntimeError(msg)
transport = self._transport
result = self._transport.call("tools/list", {})
raw_tools = result.get("tools", [])
# Phase 2: Transport call WITHOUT lock — allows concurrent access.
result = transport.call("tools/list", {})
raw_tools = result.get("tools", [])
descriptors: list[MCPToolDescriptor] = []
for raw in raw_tools:
desc = MCPToolDescriptor(
name=raw.get("name", ""),
description=raw.get("description", ""),
input_schema=raw.get("inputSchema", {}),
annotations=raw.get("annotations", {}),
)
descriptors.append(desc)
descriptors: list[MCPToolDescriptor] = []
for raw in raw_tools:
desc = MCPToolDescriptor(
name=raw.get("name", ""),
description=raw.get("description", ""),
input_schema=raw.get("inputSchema", {}),
annotations=raw.get("annotations", {}),
)
descriptors.append(desc)
if tool_filter:
descriptors = self._apply_filter(descriptors, tool_filter)
if tool_filter:
descriptors = self._apply_filter(descriptors, tool_filter)
# Phase 3: Update shared state under lock.
with self._lock:
self._tools = {d.name: d for d in descriptors}
return descriptors
return descriptors
def invoke(
self,
@@ -456,7 +472,9 @@ class MCPToolAdapter:
"""Invoke a discovered MCP tool by name.
Validates inputs against the tool's JSON Schema before calling
the server.
the server. The adapter's RLock is held only for state
validation it is released before the transport call so that
concurrent invocations are not blocked by slow network I/O.
Parameters
----------
@@ -470,6 +488,7 @@ class MCPToolAdapter:
MCPToolResult
Invocation outcome with success flag, data, and timing.
"""
# Phase 1: Validate state and inputs under lock.
with self._lock:
if not self._connected:
msg = (
@@ -494,57 +513,61 @@ class MCPToolAdapter:
f"{validation_error}",
)
start = time.monotonic()
try:
result = self._transport.call(
"tools/call",
{"name": tool_name, "arguments": arguments},
)
except TimeoutError as exc:
elapsed = (time.monotonic() - start) * 1000
return MCPToolResult(
success=False,
error=f"Tool '{tool_name}' timeout: {exc}",
duration_ms=elapsed,
)
except Exception as exc:
elapsed = (time.monotonic() - start) * 1000
return MCPToolResult(
success=False,
error=f"MCP server error invoking '{tool_name}': {exc}",
duration_ms=elapsed,
)
transport = self._transport
# Phase 2: Transport call WITHOUT lock — allows concurrent access.
start = time.monotonic()
try:
result = transport.call(
"tools/call",
{"name": tool_name, "arguments": arguments},
)
except TimeoutError as exc:
elapsed = (time.monotonic() - start) * 1000
if result.get("isError"):
content = result.get("content", [])
if content and isinstance(content, list) and len(content) > 0:
error_text = content[0].get("text", "unknown error")
else:
error_text = "unknown error"
return MCPToolResult(
success=False,
error=f"MCP server error: {error_text}",
duration_ms=elapsed,
)
# MCP 1.4.0 returns ``content`` as a list of ContentItem dicts
# (e.g. ``[{"type": "text", "text": "..."}]``). Normalise to a
# plain dict so that ``MCPToolResult.data`` is always
# ``dict[str, Any]`` and downstream code can rely on dict access.
raw_content = result.get("content", [])
if isinstance(raw_content, list):
normalised: dict[str, Any] = {"content": raw_content}
elif isinstance(raw_content, dict):
normalised = raw_content
else:
# Unexpected type — wrap it so the dict contract holds.
normalised = {"content": raw_content}
return MCPToolResult(
success=True,
data=normalised,
success=False,
error=f"Tool '{tool_name}' timeout: {exc}",
duration_ms=elapsed,
)
except Exception as exc:
elapsed = (time.monotonic() - start) * 1000
return MCPToolResult(
success=False,
error=f"MCP server error invoking '{tool_name}': {exc}",
duration_ms=elapsed,
)
elapsed = (time.monotonic() - start) * 1000
# Phase 3: Process result (no lock needed — local variables only).
if result.get("isError"):
content = result.get("content", [])
if content and isinstance(content, list) and len(content) > 0:
error_text = content[0].get("text", "unknown error")
else:
error_text = "unknown error"
return MCPToolResult(
success=False,
error=f"MCP server error: {error_text}",
duration_ms=elapsed,
)
# MCP 1.4.0 returns ``content`` as a list of ContentItem dicts
# (e.g. ``[{"type": "text", "text": "..."}]``). Normalise to a
# plain dict so that ``MCPToolResult.data`` is always
# ``dict[str, Any]`` and downstream code can rely on dict access.
raw_content = result.get("content", [])
if isinstance(raw_content, list):
normalised: dict[str, Any] = {"content": raw_content}
elif isinstance(raw_content, dict):
normalised = raw_content
else:
# Unexpected type — wrap it so the dict contract holds.
normalised = {"content": raw_content}
return MCPToolResult(
success=True,
data=normalised,
duration_ms=elapsed,
)
def register_tools(
self,
+166 -162
View File
@@ -39,6 +39,21 @@ def _coerce_optional_str(value: object | None) -> str | None:
return str(value)
def _coerce_env_bool(value: str) -> bool:
"""Consistent boolean coercion for environment variables.
Accepts ``true``, ``1``, ``yes``, and ``on`` (case-insensitive) as truthy.
"""
return value.strip().lower() in ("true", "1", "yes", "on")
class _ApiKeyMissing:
"""Sentinel distinguishing "api_key was not provided" from ``None``."""
_API_KEY_MISSING = _ApiKeyMissing()
class ProviderType(StrEnum):
"""Supported AI provider types."""
@@ -165,7 +180,7 @@ class ProviderRegistry:
supports_json_mode=False,
),
ProviderType.MOCK: ProviderCapabilities(
supports_streaming=True,
supports_streaming=False,
supports_tool_calls=False,
supports_vision=False,
max_context_length=4096,
@@ -240,6 +255,25 @@ class ProviderRegistry:
)
self._providers[provider_type] = provider_info
# Register MOCK with conditional ``is_configured``. The guard field
# ``api_key_env_var`` points to ``CLEVERAGENTS_ALLOW_MOCK_PROVIDER``
# semantically (it is the activation flag) even though it is not a
# traditional API key.
mock_allowed = _coerce_env_bool(
os.environ.get("CLEVERAGENTS_ALLOW_MOCK_PROVIDER", "")
)
mock_info = ProviderInfo(
provider_type=ProviderType.MOCK,
name=ProviderType.MOCK.value.title(),
api_key_env_var="CLEVERAGENTS_ALLOW_MOCK_PROVIDER",
default_model=self.DEFAULT_MODELS.get(ProviderType.MOCK, "unknown"),
capabilities=self.DEFAULT_CAPABILITIES.get(
ProviderType.MOCK, ProviderCapabilities()
),
is_configured=mock_allowed,
)
self._providers[ProviderType.MOCK] = mock_info
def get_configured_providers(self) -> list[ProviderInfo]:
"""Get list of all configured providers.
@@ -401,45 +435,30 @@ class ProviderRegistry:
return self.DEFAULT_MODELS.get(provider_type)
def create_llm(
self,
provider_type: ProviderType | str | None = None,
model_id: str | None = None,
**kwargs: object,
) -> BaseLanguageModel:
"""Create a LangChain LLM instance for a provider.
def _get_api_key(self, provider_type: ProviderType) -> str | None:
"""Get and validate the API key for a provider.
Args:
provider_type: The provider type. Uses default if not specified.
model_id: The model ID. Uses default for provider if not specified.
**kwargs: Additional arguments passed to the LLM constructor.
provider_type: The provider type to look up.
Returns:
A LangChain BaseLanguageModel instance.
The API key string, or ``None`` if the provider has no key
requirement (e.g. ``MOCK`` when explicitly allowed).
Raises:
ValueError: If no provider is configured or provider is invalid.
ValueError: If the provider requires an API key but it is not
configured, or if ``MOCK`` is requested without the
``CLEVERAGENTS_ALLOW_MOCK_PROVIDER`` environment variable,
or if the provider type has no API key mapping.
"""
# Resolve provider type
if provider_type is None:
provider_type = self.get_default_provider_type()
if provider_type is None:
raise ValueError(
"No AI provider configured. Please set one of the following "
"environment variables: OPENAI_API_KEY, ANTHROPIC_API_KEY, "
"GOOGLE_API_KEY, or another supported provider key."
)
elif not isinstance(provider_type, ProviderType):
try:
provider_type = ProviderType(provider_type.lower())
except ValueError as e:
raise ValueError(f"Unknown provider type: {provider_type}") from e
if provider_type == ProviderType.MOCK:
if _coerce_env_bool(os.environ.get("CLEVERAGENTS_ALLOW_MOCK_PROVIDER", "")):
return None
raise ValueError(
"Mock provider is not allowed in production. "
"Set CLEVERAGENTS_ALLOW_MOCK_PROVIDER=true to enable it."
)
# Resolve model ID
if model_id is None:
model_id = self.get_default_model(provider_type)
# Get API key
key_attr = self.PROVIDER_KEY_ATTRS.get(provider_type)
if key_attr:
api_key = getattr(self._settings, key_attr, None)
@@ -448,37 +467,62 @@ class ProviderRegistry:
f"Provider {provider_type.value} is not configured. "
f"Please set the {key_attr.upper()} environment variable."
)
return api_key
# Create the appropriate LLM
return self._create_provider_llm(provider_type, model_id, **kwargs)
# ProviderType is in the enum but not in PROVIDER_KEY_ATTRS
raise ValueError(
f"Provider {provider_type.value} has no API key mapping in "
"PROVIDER_KEY_ATTRS."
)
def _create_provider_llm(
def _create_provider_instance(
self,
provider_type: ProviderType,
model_id: str | None,
**kwargs: Any,
) -> BaseLanguageModel:
"""Create a LangChain LLM for the specified provider.
"""Single internal factory for provider instantiation.
This is the single source of truth for raw LLM instantiation. Both
``create_llm()`` and ``create_ai_provider()`` delegate here so that
adding a new provider requires touching exactly one place.
Args:
provider_type: The provider type.
model_id: The model ID.
**kwargs: Additional arguments for the LLM.
model_id: The model ID (provider default used when ``None``).
**kwargs: Extra constructor arguments forwarded to the LLM.
If ``api_key`` is present, it is consumed here instead of
being fetched from settings. ``max_retries`` is consumed
if present and never forwarded to LangChain constructors.
Returns:
A LangChain BaseLanguageModel instance.
A ``BaseLanguageModel`` instance for the requested provider.
Raises:
ValueError: If the provider is not configured or not supported.
"""
# Import the appropriate LangChain module based on provider
# Consume retry metadata so it never leaks into provider constructors.
kwargs.pop("max_retries", None)
# If the caller explicitly passed api_key (even ``None``), use it
# directly; otherwise fall back to _get_api_key.
sentinel = kwargs.pop("__api_key_sentinel", _API_KEY_MISSING)
if sentinel is _API_KEY_MISSING:
api_key: str | None = self._get_api_key(provider_type)
else:
api_key = sentinel
if provider_type == ProviderType.OPENAI:
from langchain_openai import ChatOpenAI
return ChatOpenAI(model=model_id or "gpt-4o", **kwargs)
return ChatOpenAI(model=model_id or "gpt-4o", api_key=api_key, **kwargs)
if provider_type == ProviderType.ANTHROPIC:
from langchain_anthropic import ChatAnthropic
return ChatAnthropic(
model=model_id or "claude-sonnet-4-20250514",
api_key=api_key,
**kwargs,
)
@@ -487,6 +531,7 @@ class ProviderRegistry:
return ChatGoogleGenerativeAI(
model=model_id or "gemini-2.0-flash",
google_api_key=api_key,
**kwargs,
)
@@ -522,6 +567,7 @@ class ProviderRegistry:
deployment_name=deployment_name,
azure_endpoint=azure_endpoint,
api_version=api_version,
api_key=api_key,
**kwargs,
)
@@ -530,6 +576,7 @@ class ProviderRegistry:
return ChatGroq(
model=model_id or "llama-3.1-70b-versatile",
api_key=api_key,
**kwargs,
)
@@ -538,6 +585,7 @@ class ProviderRegistry:
return ChatTogether(
model=model_id or "meta-llama/Llama-3.1-70B-Instruct-Turbo",
api_key=api_key,
**kwargs,
)
@@ -546,17 +594,13 @@ class ProviderRegistry:
return ChatCohere(
model=model_id or "command-r-plus",
api_key=api_key,
**kwargs,
)
if provider_type == ProviderType.OPENROUTER:
from langchain_openai import ChatOpenAI
api_key = self._settings.openrouter_api_key
if not api_key:
raise ValueError(
"Provider openrouter requires OPENROUTER_API_KEY to be set."
)
raw_headers = kwargs.pop("default_headers", None)
sanitized_headers: dict[str, str] | None = (
{str(k): str(v) for k, v in raw_headers.items()}
@@ -581,8 +625,68 @@ class ProviderRegistry:
openrouter_kwargs.update(kwargs)
return ChatOpenAI(**openrouter_kwargs)
if provider_type == ProviderType.MOCK:
from langchain_community.llms import FakeListLLM
return FakeListLLM(responses=["mock response"] * 10)
raise ValueError(f"Unsupported provider type: {provider_type}")
def _resolve_provider_type(
self,
provider_type: ProviderType | str | None = None,
) -> ProviderType:
"""Resolve a possibly ``None`` or string provider type to ``ProviderType``.
Uses the configured fallback chain when ``None`` and performs
case-insensitive string matching otherwise.
Raises:
ValueError: If no provider is configured or the name is unknown.
"""
if provider_type is None:
resolved = self.get_default_provider_type()
if resolved is None:
raise ValueError(
"No AI provider configured. Please set one of the following "
"environment variables: OPENAI_API_KEY, ANTHROPIC_API_KEY, "
"GOOGLE_API_KEY, or another supported provider key."
)
return resolved
if not isinstance(provider_type, ProviderType):
try:
return ProviderType(provider_type.lower())
except ValueError as e:
raise ValueError(f"Unknown provider type: {provider_type}") from e
return provider_type
def create_llm(
self,
provider_type: ProviderType | str | None = None,
model_id: str | None = None,
**kwargs: Any,
) -> BaseLanguageModel:
"""Create a LangChain LLM instance for a provider.
Args:
provider_type: The provider type. Uses default if not specified.
model_id: The model ID. Uses default for provider if not specified.
**kwargs: Additional arguments passed to the LLM constructor.
Returns:
A LangChain BaseLanguageModel instance.
Raises:
ValueError: If no provider is configured or provider is invalid.
"""
resolved = self._resolve_provider_type(provider_type)
if model_id is None:
model_id = self.get_default_model(resolved)
# Delegate to the unified factory (API key validation happens inside)
return self._create_provider_instance(resolved, model_id, **kwargs)
def create_ai_provider(
self,
provider_type: ProviderType | str | None = None,
@@ -591,8 +695,9 @@ class ProviderRegistry:
) -> AIProviderInterface:
"""Create an AIProviderInterface instance for a provider.
This creates a LangChainChatProvider that wraps the LangChain LLM
with the AIProviderInterface protocol.
All providers are wrapped uniformly in ``LangChainChatProvider``.
Raw LLM instantiation is delegated to ``_create_provider_instance()``
so that adding a new provider requires touching exactly one place.
Args:
provider_type: The provider type. Uses default if not specified.
@@ -609,135 +714,34 @@ class ProviderRegistry:
LangChainChatProvider,
)
# Resolve types
if provider_type is None:
provider_type = self.get_default_provider_type()
if provider_type is None:
raise ValueError(
"No AI provider configured. Please set one of the following "
"environment variables: OPENAI_API_KEY, ANTHROPIC_API_KEY, "
"GOOGLE_API_KEY, or another supported provider key."
)
elif not isinstance(provider_type, ProviderType):
try:
provider_type = ProviderType(provider_type.lower())
except ValueError as e:
raise ValueError(f"Unknown provider type: {provider_type}") from e
resolved = self._resolve_provider_type(provider_type)
if model_id is None:
model_id = self.get_default_model(provider_type)
model_id = self.get_default_model(resolved)
provider_info = self.get_provider_info(provider_type)
provider_info = self.get_provider_info(resolved)
capabilities = (
provider_info.capabilities if provider_info else ProviderCapabilities()
)
if provider_type == ProviderType.OPENAI:
from cleveragents.providers.llm.openai_provider import OpenAIChatProvider
key_attr = self.PROVIDER_KEY_ATTRS.get(provider_type)
api_key = getattr(self._settings, key_attr, None) if key_attr else None
if not api_key:
missing_env = (
key_attr.upper() if key_attr else provider_type.value.upper()
)
raise ValueError(
f"Provider {provider_type.value} is not configured. "
f"Please set the {missing_env} environment variable."
)
return OpenAIChatProvider(
api_key=api_key,
model=model_id or self.DEFAULT_MODELS.get(provider_type, "gpt-4o"),
max_retries=max_retries,
)
if provider_type == ProviderType.ANTHROPIC:
from cleveragents.providers.llm.anthropic_provider import (
AnthropicChatProvider,
)
key_attr = self.PROVIDER_KEY_ATTRS.get(provider_type)
api_key = getattr(self._settings, key_attr, None) if key_attr else None
if not api_key:
missing_env = (
key_attr.upper() if key_attr else provider_type.value.upper()
)
raise ValueError(
f"Provider {provider_type.value} is not configured. "
f"Please set the {missing_env} environment variable."
)
return AnthropicChatProvider(
api_key=api_key,
model=model_id
or self.DEFAULT_MODELS.get(provider_type, "claude-sonnet-4-20250514"),
max_retries=max_retries,
)
if provider_type == ProviderType.GOOGLE:
from cleveragents.providers.llm.google_provider import GoogleChatProvider
key_attr = self.PROVIDER_KEY_ATTRS.get(provider_type)
api_key = getattr(self._settings, key_attr, None) if key_attr else None
if not api_key:
missing_env = (
key_attr.upper() if key_attr else provider_type.value.upper()
)
raise ValueError(
f"Provider {provider_type.value} is not configured. "
f"Please set the {missing_env} environment variable."
)
return GoogleChatProvider(
api_key=api_key,
model=model_id
or self.DEFAULT_MODELS.get(provider_type, "gemini-2.0-flash"),
max_retries=max_retries,
)
if provider_type == ProviderType.OPENROUTER:
from cleveragents.providers.llm.openrouter_provider import (
OpenRouterChatProvider,
)
key_attr = self.PROVIDER_KEY_ATTRS.get(provider_type)
api_key = getattr(self._settings, key_attr, None) if key_attr else None
if not api_key:
missing_env = (
key_attr.upper() if key_attr else provider_type.value.upper()
)
raise ValueError(
f"Provider {provider_type.value} is not configured. "
f"Please set the {missing_env} environment variable."
)
return OpenRouterChatProvider(
api_key=api_key,
model=model_id
or self.DEFAULT_MODELS.get(
provider_type, "anthropic/claude-sonnet-4-20250514"
),
organization=self._settings.openrouter_organization,
max_retries=max_retries,
)
# Create factory function for the LLM.
# Capture the narrowed ProviderType in a local variable so the closure
# holds a ProviderType rather than the wider ProviderType | str | None.
resolved_provider_type: ProviderType = provider_type
# Eager validation: fail fast before building the factory closure,
# then reuse the validated key to avoid a second lookup.
api_key: str | None = self._get_api_key(resolved)
def llm_factory(mid: str) -> BaseLanguageModel:
factory_kwargs: dict[str, Any] = {"max_retries": max_retries}
return self._create_provider_llm(
resolved_provider_type,
# Use sentinel so _create_provider_instance knows the key was
# pre-validated and should not trigger a second _get_api_key.
factory_kwargs["__api_key_sentinel"] = api_key
return self._create_provider_instance(
resolved,
mid,
**factory_kwargs,
)
return LangChainChatProvider(
name=provider_type.value,
model_id=model_id or self.DEFAULT_MODELS.get(provider_type, "unknown"),
name=resolved.value,
model_id=model_id or self.DEFAULT_MODELS.get(resolved, "unknown"),
llm_factory=llm_factory,
max_retries=max_retries,
supports_streaming=capabilities.supports_streaming,
+16 -1
View File
@@ -37,6 +37,7 @@ from __future__ import annotations
import os
import subprocess
import threading
from pathlib import Path
from typing import Any
@@ -65,16 +66,30 @@ _GIT_ENV: dict[str, str] = {
# Cached merge of ``os.environ`` with ``_GIT_ENV``, populated on first use.
_BASE_ENV: dict[str, str] | None = None
# Lock that serialises the one-time initialisation of ``_BASE_ENV``.
# The outer ``if _BASE_ENV is None`` check keeps the warm-cache path
# lock-free; the inner check inside the ``with`` block prevents duplicate
# initialisation when two threads race on the very first call.
_BASE_ENV_LOCK: threading.Lock = threading.Lock()
def _get_base_env() -> dict[str, str]:
"""Return a cached merge of the process environment with git overrides.
The snapshot is taken on first call and reused thereafter, avoiding a
full ``os.environ`` copy on every git invocation.
Thread safety: double-checked locking eliminates the TOCTOU race where
two concurrent threads could both observe ``_BASE_ENV is None``, both
snapshot ``os.environ``, and write potentially different snapshots.
The outer check keeps the warm-cache path lock-free; the inner check
inside the lock prevents duplicate initialisation on the first call.
"""
global _BASE_ENV
if _BASE_ENV is None:
_BASE_ENV = {**os.environ, **_GIT_ENV}
with _BASE_ENV_LOCK:
if _BASE_ENV is None:
_BASE_ENV = {**os.environ, **_GIT_ENV}
return _BASE_ENV