Compare commits

..

7 Commits

Author SHA1 Message Date
HAL9000 7f47ec85da docs(milestones): split oversized milestone pages into sub-pages (PR #9957)
CI / build (pull_request) Successful in 1m7s
CI / lint (pull_request) Successful in 1m13s
CI / quality (pull_request) Successful in 1m13s
CI / helm (pull_request) Successful in 43s
CI / typecheck (pull_request) Successful in 1m22s
CI / security (pull_request) Successful in 1m25s
CI / push-validation (pull_request) Successful in 31s
CI / unit_tests (pull_request) Successful in 5m11s
CI / integration_tests (pull_request) Successful in 8m22s
CI / docker (pull_request) Successful in 2m22s
CI / coverage (pull_request) Successful in 10m39s
CI / status-check (pull_request) Successful in 4s
Refactor Milestones documentation to address PR #9957 reviewer feedback:

- Split v3.0.0.md (506 lines) into 3 sub-pages under milestones/v3.0.0/:
  - index.md — Overview, goals, feature table, architecture notes (87 lines)
  - cli-reference.md — Full CLI command reference (142 lines)
  - deep-dive.md — Sandbox, persistence, domain model internals (95 lines)

- Split v3.1.0.md (682 lines) into 5 sub-pages under milestones/v3.1.0/:
  - index.md — Overview, goals, feature table, architecture notes (69 lines)
  - actor-yaml.md — Actor YAML format + Compiler/LangGraph integration (131 lines)
  - integration.md — MCP Adapter + Tool Router provider normalization (91 lines)
  - skills.md — Skill Registry + Validation Runner gates (127 lines)
  - quickstart.md — Common workflows and reference guide (66 lines)

- Update mkdocs.yml navigation to reflect the new sub-page structure
- Total: from 2 files at 506/682 lines to 9 files all under 150 lines

ISSUES CLOSED: #9957
2026-06-04 01:49:54 -04:00
HAL9000 21728618ca Merge pull request 'test: restore complete M2 acceptance e2e test' (#11191) from fix/m2-acceptance-test into master
CI / build (push) Successful in 42s
CI / lint (push) Successful in 51s
CI / quality (push) Successful in 57s
CI / typecheck (push) Successful in 1m26s
CI / security (push) Successful in 1m25s
CI / push-validation (push) Successful in 27s
CI / helm (push) Successful in 34s
CI / benchmark-regression (push) Has started running
CI / e2e_tests (push) Successful in 1m11s
CI / unit_tests (push) Successful in 5m38s
CI / integration_tests (push) Successful in 9m47s
CI / docker (push) Successful in 1m38s
CI / coverage (push) Successful in 11m45s
CI / benchmark-publish (push) Has started running
CI / status-check (push) Successful in 4s
2026-06-04 05:46:34 +00:00
HAL9000 09bc5222a5 test(e2e): fix M2 acceptance test LLM provider and actor validation
CI / lint (pull_request) Successful in 39s
CI / helm (pull_request) Successful in 33s
CI / build (pull_request) Successful in 50s
CI / quality (pull_request) Successful in 1m6s
CI / typecheck (pull_request) Successful in 1m10s
CI / push-validation (pull_request) Successful in 26s
CI / security (pull_request) Successful in 1m18s
CI / unit_tests (pull_request) Successful in 6m2s
CI / docker (pull_request) Successful in 1m28s
CI / integration_tests (pull_request) Successful in 10m12s
CI / coverage (pull_request) Successful in 10m56s
CI / status-check (pull_request) Successful in 3s
- Replace hardcoded gpt-4/openai/gpt-4 references with Resolve LLM Actor
  keyword so the test falls back to Anthropic when OpenAI is unavailable
- Add explicit return-code check (rc==0) for actor registration in Step 2
- Update CHANGELOG.md with entry for PR #11191
2026-06-04 00:33:43 -04:00
freemo c3ed77a95c test: restore complete M2 acceptance e2e test
The M2 acceptance test was incomplete (truncated at line 48).
Restore the full test case that exercises the complete M2 milestone:
- Actor YAML compilation and registration
- Resource registration and project creation
- Action creation with custom actor configuration
- Full plan lifecycle: use, strategize, execute, diff, apply
- Verification of plan status and integrity

Exercises real LLM API integration using OpenAI GPT-4 model.

ISSUES CLOSED: #10812
2026-06-04 00:24:09 -04:00
HAL9000 00be1004f1 Merge pull request '[AUTO-DOCS-8] docs: document ACP to A2A module rename and symbol standardization' (#10230) from docs/auto-docs-8-a2a-rename-documentation into master
CI / build (push) Successful in 32s
CI / lint (push) Successful in 41s
CI / helm (push) Successful in 38s
CI / push-validation (push) Successful in 25s
CI / typecheck (push) Successful in 56s
CI / security (push) Successful in 1m18s
CI / benchmark-publish (push) Has started running
CI / e2e_tests (push) Successful in 59s
CI / unit_tests (push) Successful in 5m12s
CI / integration_tests (push) Successful in 9m56s
CI / quality (push) Failing after 14m25s
CI / benchmark-regression (push) Successful in 1h10m46s
CI / coverage (push) Has been cancelled
CI / docker (push) Has been cancelled
CI / status-check (push) Has been cancelled
2026-06-04 04:16:22 +00:00
HAL9000 82502404cd docs: add CHANGELOG entry for A2A migration guide (#10230)
CI / lint (pull_request) Successful in 47s
CI / quality (pull_request) Successful in 51s
CI / typecheck (pull_request) Successful in 1m11s
CI / security (pull_request) Successful in 1m26s
CI / build (pull_request) Successful in 37s
CI / helm (pull_request) Successful in 32s
CI / push-validation (pull_request) Successful in 26s
CI / unit_tests (pull_request) Successful in 5m19s
CI / integration_tests (pull_request) Successful in 9m45s
CI / docker (pull_request) Successful in 1m51s
CI / coverage (pull_request) Successful in 10m46s
CI / status-check (pull_request) Successful in 7s
2026-06-03 23:46:13 -04:00
HAL9000 980fcabc48 docs: document ACP to A2A module rename and symbol standardization [AUTO-DOCS-8]
CI / push-validation (pull_request) Successful in 28s
CI / lint (pull_request) Successful in 32s
CI / helm (pull_request) Successful in 36s
CI / build (pull_request) Successful in 43s
CI / quality (pull_request) Successful in 55s
CI / typecheck (pull_request) Successful in 1m18s
CI / security (pull_request) Successful in 1m19s
CI / unit_tests (pull_request) Successful in 6m51s
CI / integration_tests (pull_request) Successful in 10m33s
CI / docker (pull_request) Failing after 12m6s
CI / coverage (pull_request) Failing after 12m22s
CI / status-check (pull_request) Failing after 3s
2026-06-03 22:51:33 -04:00
20 changed files with 1460 additions and 1050 deletions
+2 -9
View File
@@ -6,6 +6,8 @@ Changed `wf10_batch.robot` to be less likely to create files, and
`plan_generation_graph.robot` to give more test answers.
## [Unreleased]
- **test(e2e): restore complete M2 acceptance test** (#11191): Restored the truncated M2 full actor compiler and LLM integration e2e acceptance test to its complete 10-step form. Added dynamic LLM provider selection via `Resolve LLM Actor` (falls back to Anthropic when OpenAI is unavailable or quota-exhausted), replacing hardcoded `gpt-4` / `openai/gpt-4` references in the actor config and action YAML. Added explicit return-code validation (`Should Be Equal As Integers ${r_actor.rc} 0`) for the actor registration step.
- **docs(a2a): ACP to A2A migration guide** (#10230): Added migration guide documenting how to upgrade from the ACP module to the A2A module introduced in v3.6.0, including symbol renames, field renames, operation-name mappings, and YAML configuration updates.
- **Plan Prompt JSON Timing Field** (#9353): `agents plan prompt --format json` now
includes `timing.started` as an ISO 8601 UTC timestamp in the JSON envelope,
matching the spec (§CLI Commands — `agents plan prompt`). Extended
@@ -407,15 +409,6 @@ ensuring data is stored with proper parameter values.
### Changed
- Fixed stale `AUTO-BUG-POOL` tracking prefix references in automation-tracking.md documentation and agent-system-specification.md spec document, replaced with correct `AUTO-BUG-SUP` prefix used by the bug-hunt-pool-supervisor agent (#7875).
=======
- **ACMS index performance optimized via parallel concurrent processing** (#9330): Introduced
`ThreadPoolExecutor`-based parallel file hashing in the ACMS indexing pipeline, enabling
projects with 10,000+ files to be indexed without timeout. Includes early-stage binary
detection (null-byte heuristic), configurable size thresholds, pattern-based exclusion
via `.acmsignore`/`.gitignore`, on-disk JSON cache persistence with atomic replacements,
and thread-safe progress tracking. The parallel indexer replaces the previous sequential
file walk-and-index approach, reducing indexing time from minutes to seconds for large
projects while maintaining safety through atomic cache operations and bounded memory usage.
- **`agents session list` now displays full 26-character session ULIDs** (#10970): The Rich table
and Summary panel ("Most Recent" / "Oldest") previously showed only the first 8 characters of
-1
View File
@@ -81,7 +81,6 @@ Below are some of the specific details of various contributions.
* HAL 9000 has contributed the ACMS Context Tier Hydration documentation (PR #9208 / issue #6175): documented the `context_tier_hydrator` module in the ACMS Architecture section of the specification, covering its public interface, file listing strategy, budget limits, and fragment structure.
* HAL 9000 has contributed the agent task memory leak fix (#9044): replaced `list.remove` with `set.discard` as the done_callback for asyncio tasks in `Agent._tasks`, preventing unbounded memory growth in long-lived agents and ensuring safe concurrent task removal.
* HAL 9000 has contributed the ACMS context show/clear CLI commands (PR #9675 / issue #9586): implemented `context show <view>` displaying assembled context with per-tier budget utilization summary (hot/warm/cold), and `context clear` with --path, --tag, --tier filtering plus confirmation prompt with --yes bypass. Includes 12 Behave BDD scenarios, 9 Robot Framework integration tests, ASV benchmarks, full type annotations, and _TierServiceProtocol for type safety.
* HAL 9000 has contributed the ACMS parallel processing optimization (PR #9981 / issue #9330): introduced `ThreadPoolExecutor`-based concurrent file hashing to the ACMS indexing pipeline, enabling projects with 10,000+ files to be indexed without timeout. Includes binary detection via null-byte heuristic, configurable size thresholds, `.acmsignore`/`.gitignore` pattern exclusion, on-disk JSON cache persistence with atomic replacements, and thread-safe progress tracking.
# Details (PR Contributions)
+35
View File
@@ -134,3 +134,38 @@ config = ServerConnectionConfig(url="https://my-server.example.com", token="..."
```
Validates URL format and token presence.
---
## ACP to A2A Rename (v3.6.0)
> **Breaking change in v3.6.0.** The `cleveragents.acp` module was renamed to
> `cleveragents.a2a`. All `Acp*` class names were renamed to `A2a*`. The
> `AcpRequest` and `AcpResponse` field names were updated to comply with the
> JSON-RPC 2.0 specification.
For the complete migration guide, see
[`docs/development/acp-to-a2a-migration.md`](../development/acp-to-a2a-migration.md).
**Symbol rename summary:**
| Old (ACP) | New (A2A) |
|---|---|
| `AcpRequest` | `A2aRequest` |
| `AcpResponse` | `A2aResponse` |
| `AcpLocalFacade` | `A2aLocalFacade` |
| `AcpError` | `A2aError` |
| `AcpEventQueue` | `A2aEventQueue` |
**Field rename summary (`A2aRequest` / `A2aResponse`):**
| Old Field | New Field |
|---|---|
| `a2a_version` | `jsonrpc` |
| `request_id` | `id` |
| `operation` | `method` |
| `payload` | `params` |
| `data` | `result` |
| `error_detail` | `error` |
See [ADR-047](../adr/ADR-047-acp-standard-adoption.md) for architectural rationale.
+264
View File
@@ -0,0 +1,264 @@
# ACP to A2A Module Rename and Symbol Standardization
> **Introduced in:** v3.6.0
> **Architectural decision:** [ADR-047 - A2A Standard Adoption](../adr/ADR-047-acp-standard-adoption.md)
> **Module path:** `cleveragents.a2a` (commit `449c33b7`)
---
## Overview
In v3.6.0 the internal **ACP (Agent Client Protocol)** module was renamed to
**A2A (Agent-to-Agent Protocol)** and all public symbols were standardized to
align with the external [A2A open standard](https://a2a-protocol.org) (Linux
Foundation, Apache 2.0).
This was not merely a rename - it was an architectural shift from a bespoke
JSON envelope protocol to the industry-standard A2A wire format built on
**JSON-RPC 2.0**. The change eliminates the proprietary ACP envelope, replaces
custom REST routing with JSON-RPC 2.0 method dispatch, and introduces
**Agent Card** discovery (`/.well-known/agent.json`).
---
## What Changed
### 1. Module Path
| Before (ACP) | After (A2A) |
|---|---|
| `cleveragents.acp` | `cleveragents.a2a` |
| `from cleveragents.acp import ...` | `from cleveragents.a2a import ...` |
### 2. Class and Symbol Names
All public symbols were renamed from `Acp*` to `A2a*`:
| Old Symbol (ACP) | New Symbol (A2A) | Notes |
|---|---|---|
| `AcpRequest` | `A2aRequest` | Wire format changed - see section 3 |
| `AcpResponse` | `A2aResponse` | Wire format changed - see section 3 |
| `AcpError` | `A2aError` | Base exception |
| `AcpNotAvailableError` | `A2aNotAvailableError` | Server-mode guard |
| `AcpOperationNotFoundError` | `A2aOperationNotFoundError` | Unknown method |
| `AcpVersionMismatchError` | `A2aVersionMismatchError` | Protocol version guard |
| `AcpLocalFacade` | `A2aLocalFacade` | Local-mode dispatcher |
| `AcpHttpTransport` | `A2aHttpTransport` | Server-mode transport stub |
| `AcpEventQueue` | `A2aEventQueue` | In-process event delivery |
| `AcpVersionNegotiator` | `A2aVersionNegotiator` | Version negotiation |
| `AcpVersion` | `A2aVersion` | Version constants |
| `AcpEvent` | `A2aEvent` | SSE event envelope |
| `AcpErrorDetail` | `A2aErrorDetail` | JSON-RPC error object |
### 3. Request and Response Field Names
`A2aRequest` and `A2aResponse` now use **JSON-RPC 2.0** field names instead of
the proprietary ACP envelope fields. These models are defined in
`cleveragents.a2a.models` (commit `449c33b7`):
| Old Field (ACP) | New Field (A2A / JSON-RPC 2.0) | Type |
|---|---|---|
| `a2a_version` | `jsonrpc` (always `"2.0"`) | `str` |
| `request_id` | `id` | `str` |
| `operation` | `method` | `str` |
| `payload` | `params` | `dict[str, Any]` |
| `data` (success path) | `result` | `dict[str, Any] or None` |
| `error_detail` | `error` | `A2aErrorDetail or None` |
**Before (ACP):**
```python
from cleveragents.acp import AcpRequest, AcpLocalFacade
request = AcpRequest(
a2a_version="1.0",
request_id="req-001",
operation="plan.status",
payload={"plan_id": "01HXRCF1..."},
)
response = facade.dispatch(request)
if response.data:
print(response.data["phase"])
```
**After (A2A):**
```python
from cleveragents.a2a import A2aRequest, A2aLocalFacade
request = A2aRequest(
jsonrpc="2.0",
id="req-001",
method="_cleveragents/plan/status",
params={"plan_id": "01HXRCF1..."},
)
response = facade.dispatch(request)
if response.result:
print(response.result["phase"])
```
### 4. Operation Names
The ACP proprietary operation names were replaced with the A2A standard
`_cleveragents/` extension method namespace. The legacy names are still
accepted by `A2aLocalFacade.dispatch()` in `cleveragents.a2a.facade`
(commit `449c33b7`) for backward compatibility but are **deprecated** and
will be removed in a future major version.
| Old Operation (ACP / deprecated) | New Operation (A2A) |
|---|---|
| `session.create` | `_cleveragents/session/create` (via `message/send`) |
| `session.close` | `_cleveragents/session/close` |
| `plan.create` | `_cleveragents/plan/use` |
| `plan.execute` | `_cleveragents/plan/execute` |
| `plan.status` | `_cleveragents/plan/status` |
| `plan.diff` | `_cleveragents/plan/diff` |
| `plan.apply` | `_cleveragents/plan/apply` |
| `registry.list_tools` | `_cleveragents/registry/tool/list` |
| `registry.list_resources` | `_cleveragents/registry/resource/list` |
| `context.get` | `_cleveragents/context/show` |
| `event.subscribe` | (event delivery via SSE `message/stream`) |
The full list of supported A2A extension methods is documented in
[`docs/reference/a2a.md`](../reference/a2a.md#extension-methods).
### 5. YAML and Configuration Files
If you have YAML configuration files, actor definitions, or skill files that
reference the old ACP module path or operation names, update them as follows:
**Actor YAML - protocol binding:**
```yaml
# Before (ACP)
protocol:
type: acp
version: "1.0"
# After (A2A)
protocol:
type: a2a
version: "2.0"
```
**Skill YAML - operation references:**
```yaml
# Before (ACP)
operations:
- operation: plan.create
- operation: plan.execute
# After (A2A)
operations:
- method: _cleveragents/plan/use
- method: _cleveragents/plan/execute
```
**Server connection config:**
```yaml
# Before (ACP)
server:
protocol: acp
endpoint: https://my-server.example.com/acp
# After (A2A)
server:
protocol: a2a
endpoint: https://my-server.example.com/a2a
```
---
## Migration Steps
Follow these steps when upgrading from a codebase that used the ACP module:
### Step 1 - Update imports
Replace all `cleveragents.acp` imports with `cleveragents.a2a`:
```bash
# Find all files that import from the old ACP module
grep -r "from cleveragents.acp" src/ tests/
grep -r "import cleveragents.acp" src/ tests/
```
Then update each import:
```python
# Before
from cleveragents.acp import AcpLocalFacade, AcpRequest, AcpResponse
# After
from cleveragents.a2a import A2aLocalFacade, A2aRequest, A2aResponse
```
### Step 2 - Rename symbol references
Use your editor's find-and-replace to rename all `Acp*` symbols to `A2a*`
(see the complete symbol table in section 2 above).
### Step 3 - Update request field names
Update all `AcpRequest` and `AcpResponse` constructor calls and field accesses
to use the new JSON-RPC 2.0 field names (see section 3 above).
### Step 4 - Update operation names
Replace deprecated ACP operation strings with the new `_cleveragents/`
extension method names (see section 4 above). The legacy names still work but
emit deprecation warnings.
### Step 5 - Update YAML files
Search for any YAML files that reference `acp` in protocol type fields or
operation names and update them (see section 5 above).
### Step 6 - Verify
Run the full test suite to confirm no regressions:
```bash
nox -s unit_tests
nox -s integration_tests
```
---
## Backward Compatibility
The `A2aLocalFacade.dispatch()` method in `cleveragents.a2a.facade.A2aLocalFacade`
(commit `449c33b7`) continues to accept the legacy ACP operation names
(`session.create`, `plan.create`, etc.) via the `_LEGACY_OPERATIONS` list.
This backward compatibility shim will be removed in the next major version.
The `A2aVersion` class in `cleveragents.a2a.models.A2aVersion` (commit
`449c33b7`) retains the `CURRENT` and `SUPPORTED` constants for backward
compatibility, though the wire format now uses the `jsonrpc` field rather than
a proprietary `a2a_version` field.
---
## Architectural Rationale
The rename was driven by the adoption of the external A2A open standard
([a2a-protocol.org](https://a2a-protocol.org)), which supersedes the bespoke
ACP protocol. Key reasons for the change:
- **Ecosystem alignment**: A2A is governed by the Linux Foundation with 22k+
GitHub stars and SDKs in Python, Go, JavaScript, Java, and .NET.
- **Agent Card discovery**: The standard introduces `/.well-known/agent.json`
for dynamic capability advertisement.
- **JSON-RPC 2.0**: Replaces the proprietary envelope with a mature, widely-
tooled standard.
- **Extensibility**: The `_cleveragents/` extension namespace cleanly separates
platform-specific operations from standard agent interactions.
See [ADR-047](../adr/ADR-047-acp-standard-adoption.md) for the full
architectural rationale and decision record.
---
## Related Documentation
- [A2A Reference](../reference/a2a.md) - complete API reference for the `cleveragents.a2a` module
- [A2A API](../api/a2a.md) - class-level API documentation
- [ADR-026 - Agent-to-Agent Protocol](../adr/ADR-026-agent-client-protocol.md) - protocol boundary design
- [ADR-047 - A2A Standard Adoption](../adr/ADR-047-acp-standard-adoption.md) - rationale for adopting the external standard
- [ADR-048 - Server Application Architecture](../adr/ADR-048-server-application-architecture.md) - server implementation using A2A
+34
View File
@@ -0,0 +1,34 @@
# Milestones
This section documents the major milestones in the CleverAgents v3 release series.
Each milestone page covers the features delivered, CLI commands introduced, and
architectural decisions made during that release cycle.
## Release Overview
| Milestone | Version | Title | Status |
|-----------|---------|-------|--------|
| M1 | [v3.0.0](v3.0.0.md) | Minimal Local Source-Code Workflow | Released |
| M2 | [v3.1.0](v3.1.0.md) | Actor Compiler + Full LLM Integration | Released |
## What Is a Milestone?
CleverAgents uses **Clever Semantic Versioning** (MAJOR.MINOR.PATCH) where each
MINOR version bump corresponds to a planned milestone delivering a coherent set of
features. Milestones are tracked as Forgejo milestones and linked to epics
in the issue tracker.
Each milestone page documents:
- **Goals** — What the milestone set out to achieve
- **Delivered Features** — Concrete capabilities added
- **CLI Reference** — Commands introduced or changed
- **Architecture Notes** — Key design decisions
- **Migration Guide** — Breaking changes and upgrade steps (if any)
## Related Resources
- [Implementation Timeline](../timeline.md) — Full chronological timeline
- [Architecture](../architecture.md) — System architecture overview
- [API Reference](../api/index.md) — Module-level API documentation
- [CHANGELOG](https://git.cleverthis.com/cleveragents/cleveragents-core/src/branch/master/CHANGELOG.md) — Detailed change log
+142
View File
@@ -0,0 +1,142 @@
# v3.0.0 — Action, Resource, Project & Plan CLI Reference
Complete command documentation for all CleverAgents v3.0.0 CLI commands.
---
## Action Management (`agents action`)
Actions are reusable plan templates defined in YAML. They specify which actors
handle each phase of the plan lifecycle and what arguments the plan accepts.
### `agents action create`
```bash
agents action create --config ./my-action.yaml
```
**Minimal action YAML:**
```yaml
name: local/code-coverage
description: Increase code coverage to the target percentage
strategy_actor: openai/gpt-4
execution_actor: openai/gpt-4
definition_of_done: |
Line coverage reaches the target percentage for all modules under src/.
arguments:
- name: target_coverage
type: integer
required: true
```
**Full action YAML:**
```yaml
name: local/refactor
description: Refactor a module to follow SOLID principles
strategy_actor: openai/gpt-4
execution_actor: anthropic/claude-3
estimation_actor: openai/gpt-4
invariant_actor: anthropic/claude-3
automation_profile: trusted
definition_of_done: |
All classes follow Single Responsibility Principle.
No method exceeds 20 lines.
invariants:
- "No reduction in test coverage"
- "All public APIs remain backward compatible"
arguments:
- name: module_path
type: string
required: true
```
### `agents action list` / `show` / `archive`
```bash
agents action list # all actions
agents action list --namespace local # filter by namespace
agents action list --state available # filter by state
agents action show local/code-coverage # show action details
agents action archive local/old-action # soft-delete action
```
---
## Resource Management (`agents resource`)
Resources are external entities (git repos, directories, mounts) that plans
can read or modify.
### `agents resource add`
```bash
agents resource add git-checkout local/my-repo --path /home/user/projects/my-repo
agents resource add fs-directory local/data --path /data --read-only
```
**Built-in resource types (v3.0.0):** `git-checkout`, `fs-directory`, `fs-mount`
### `agents resource list` / `show` / `remove`
```bash
agents resource list # all resources
agents resource show local/my-repo # resource details
agents resource remove --yes local/my-repo # remove resource
```
---
## Project Management (`agents project`)
Projects group resources together for plan execution context.
### `agents project create` / `link-resource` / `list` / `show`
```bash
agents project create local/my-project --description "My application"
agents project link-resource local/my-project local/my-repo
agents project list
agents project show local/my-project
```
---
## Plan Lifecycle (`agents plan`)
Plans are the central execution unit. Lifecycle: **Strategize -> Execute -> Apply**.
### `agents plan use` — Create a plan
```bash
agents plan use local/code-coverage my-project --arg target_coverage=80
agents plan use local/lint proj-1 proj-2 # multiple projects
agents plan use local/refactor my-project \ # with automation profile & invariants
--automation-profile trusted --invariant "No new warnings"
agents plan use local/code-coverage my-project \ # with actor overrides
--strategy-actor openai/gpt-4 --execution-actor anthropic/claude-3
```
### `agents plan execute` / `diff` / `apply` — Run and merge changes
```bash
agents plan execute # Strategize -> Execute phases
agents plan diff 01HXYZ... # preview proposed changes
agents plan apply --yes # merge changes (skip confirmation)
```
### `agents plan list` / `status` / `cancel` — Manage plans
```bash
agents plan list --state complete # filter plans
agents plan status 01HXYZ... # plan details
agents plan cancel 01HXYZ... --reason "Requirements changed"
```
---
## See Also
- [CLI overview (v3.0.0)](index.md) — Goals, architecture notes, feature table
- [Deep Dive](deep-dive.md) — Sandbox, persistence, domain model internals
+95
View File
@@ -0,0 +1,95 @@
# v3.0.0 — Deep Dive: Sandbox, Persistence & Domain Model
Internal architecture details for CleverAgents v3.0.0.
---
## Git Worktree Sandbox
The git worktree sandbox isolates LLM-generated changes in a dedicated branch
and worktree until the user approves them via `agents plan apply`.
See the full [Git Worktree Sandbox](../../modules/git-worktree-sandbox.md) module docs.
### How It Works
1. Plan execute creates branch `cleveragents/plan-<id>` from current HEAD.
2. A temporary git worktree is created in a system temp directory.
3. The LLM writes changes only inside the worktree (not the real repo).
4. `plan diff` shows proposed changes as unified diff.
5. `plan apply` merges sandbox into original via `git merge`, then cleans up.
6. `plan cancel` discards the worktree — no real repo changes.
Non-git resources (e.g. `fs-directory`) fall back to `shutil.copy2`.
### Apply Summary Output
```
╭─ Apply Summary ────────────────────────────────────────╮
│ Plan ID: plan-01HZ... │
│ Artifacts: 3 files changed │
│ Insertions: +142 │
│ Deletions: -17 │
╰───────────────────────────────────────────────────────╯
╭─ Sandbox Cleanup ──────────────────────────────────────╮
│ [x] Worktree removed │
│ [x] Branch deleted │
╰───────────────────────────────────────────────────────╯
```
---
## SQLite Persistence with Alembic
All entities persist in `~/.cleveragents/cleveragents.db`.
### Managed Tables
| Table | Description |
|-------|-------------|
| `actions` | Action templates |
| `action_arguments`, `action_invariants` | Per-action metadata |
| `resources` | Resource instances |
| `projects`, `project_resource_links` | Projects and links |
| `plans`, `plan_project_links` | Plans and project bindings |
| `decisions` | Decision tree nodes per plan |
### Migration Commands
```bash
alembic upgrade head # apply pending migrations
alembic current # show current revision
alembic history # migration history
```
---
## Domain Model (Pydantic v2)
All domain entities use Pydantic v2 with `frozen=True` for immutability.
```python
from cleveragents.domain.models import Action, Resource, Project, Plan
action = Action(
name="local/code-coverage",
description="Increase code coverage",
strategy_actor="openai/gpt-4",
execution_actor="openai/gpt-4",
definition_of_done="Coverage reaches target.",
)
# action.name = "other" # raises ValidationError (frozen)
```
ULIDs (Universally Unique Lexicographically Sortable Identifiers) are the primary keys.
See [ADR-019](../../adr/ADR-019-storage-and-persistence.md) for storage details.
---
## See Also
- [CLI Reference](cli-reference.md) — All command documentation
- [Git Worktree Sandbox](../../modules/git-worktree-sandbox.md)
- [ADR-006](../../adr/ADR-006-plan-lifecycle.md): Plan Lifecycle
- [ADR-019](../../adr/ADR-019-storage-and-persistence.md): Storage & Persistence
+87
View File
@@ -0,0 +1,87 @@
# v3.0.0 — Minimal Local Source-Code Workflow (M1) — Overview
**Released:** v3.0.0
**Milestone:** M1 — Minimal Local Source-Code Workflow
**Theme:** Establish the foundational CLI, domain model, and persistence layer for local agent-driven code workflows.
---
## Overview
v3.0.0 delivers the first complete end-to-end workflow for using CleverAgents to
manage source-code changes on a local machine. It introduces the core CLI command
groups (`agents action`, `agents resource`, `agents project`, `agents plan`),
a SQLite-backed persistence layer with Alembic migrations, and a git worktree
sandbox that isolates LLM-generated changes until the user approves them.
This milestone establishes the **specification-first** development model: every
feature is driven by a YAML-defined action template, executed through a structured
plan lifecycle (Strategize -> Execute -> Apply), and persisted in a local SQLite
database.
---
## Goals
1. Provide a minimal but complete CLI for local source-code automation.
2. Introduce a typed, versioned domain model using Pydantic v2.
3. Persist all entities (actions, resources, projects, plans) in SQLite via Alembic.
4. Isolate LLM-generated changes in a git worktree sandbox before applying them.
5. Enable plan lifecycle management: create, execute, diff, and apply plans.
---
## Delivered Features
v3.0.0 delivers the following feature groups:
| Feature Group | CLI Commands | Description | Page |
|---------------|-------------|-------------|------|
| **Action Mgmt** | `agents action` | YAML plan templates | [CLI Reference](cli-reference.md) |
| **Resource Mgmt** | `agents resource` | Git repos, filesystems | [CLI Reference](cli-reference.md) |
| **Project Mgmt** | `agents project` | Group resources for plans | [CLI Reference](cli-reference.md) |
| **Plan Lifecycle** | `agents plan` | Strategy-Execute-Apply cycle | [CLI Reference](cli-reference.md) |
| **Git Worktree Sandbox** | (internal) | Isolated LLM change sandbox | [Deep Dive](deep-dive.md) |
| **SQLite Persistence** | (internal) | SQLite + Alembic with 9 tables | [Deep Dive](deep-dive.md) |
| **Pydantic v2 Domain Model** | (internal) | Frozen=True, ULID keys | [Deep Dive](deep-dive.md) |
---
## Architecture Notes
### Layered Architecture
v3.0.0 follows a strict layered architecture (see [ADR-001](../../adr/ADR-001-layered-architecture.md)):
```
CLI Layer agents action/resource/project/plan
|
Application Layer PlanLifecycleService, ActionService, ResourceService
|
Domain Layer Action, Resource, Project, Plan (Pydantic v2, frozen=True)
|
Infrastructure SQLite + Alembic, GitWorktreeSandbox, ToolRegistry
```
### Dependency Injection
All services are wired via a DI container (see [ADR-003](../../adr/ADR-003-dependency-injection.md)).
### Plan Lifecycle State Machine
Plans follow a strict state machine (see [ADR-006](../../adr/ADR-006-plan-lifecycle.md)):
```
CREATED -> STRATEGIZE/queued -> STRATEGIZE/complete
-> EXECUTE/queued -> EXECUTE/complete
-> APPLY/queued -> APPLY/complete -> CANCELLED / ERRORED (terminal)
```
---
## See Also
- [CLI Reference](cli-reference.md) — Full command-by-command documentation
- [Deep Dive](deep-dive.md) — Sandbox, persistence, domain model internals
- [ADR-001](../../adr/ADR-001-layered-architecture.md): Layered Architecture
- [ADR-006](../../adr/ADR-006-plan-lifecycle.md): Plan Lifecycle
+131
View File
@@ -0,0 +1,131 @@
# v3.1.0 — Actor YAML Format & Compiler
Declarative actor definitions compiled into LangGraph graphs.
---
## Actor YAML Format
Every actor file must specify `version: "3"` and one of three types: `llm`, `tool`, or `graph`.
See [Actor YAML Schema Reference](../../reference/actors_schema.md) for full fields.
### LLM Actor
```yaml
version: "3"
name: assistants/code-reviewer
type: llm
model: gpt-4
system_prompt: |
You are an expert Python code reviewer.
context_view: reviewer
memory:
enabled: true
max_messages: 20
```
### Tool Actor
```yaml
version: "3"
name: utilities/file-ops
type: tool
tools:
- files/read_file
- files/write_file
- files/list_directory
```
### Graph Actor
```yaml
version: "3"
name: workflows/tdd-cycle
type: graph
model: gpt-4
route:
nodes:
- id: planner; type: agent; name: Task Planner
config: { model: gpt-4, prompt: "Plan implementation tasks." }
- id: implementer; type: agent; name: Code Implementer
config: { model: gpt-4, prompt: "Implement planned tasks." }
- id: verifier; type: tool; name: Test Runner
config: { tool_name: testing/run_pytest }
- id: check_results; type: conditional
config: { conditions: [{check: "passed==True", route_to: done}, {check: "passed==False", route_to: implementer}] }
- id: done; type: agent; name: Summary Writer
edges:
- {from_node: planner, to_node: implementer}
- {from_node: implementer, to_node: verifier}
- {from_node: verifier, to_node: check_results}
entry_node: planner
exit_nodes: [done]
```
### Actor Name Format
Must use `namespace/name`:
- `assistants/code-reviewer` — valid
- `code-reviewer` — invalid (no namespace)
- `a/b/c` — invalid (too many slashes)
---
## Actor Compiler (LangGraph Integration)
Translates GRAPH-type YAML into LangGraph `StateGraph` structures.
See [Actor Compiler Reference](../../reference/actor_compiler.md).
### Compilation Pipeline
```
ActorConfigSchema (type=GRAPH)
-> validate type, route
-> reference validation (all node IDs exist)
-> intra-graph cycle detection
-> cross-actor subgraph cycle detection
-> node mapping (NodeDef -> LangGraph NodeConfig)
-> edge mapping
-> metadata assembly
=> CompiledActor (nodes, edges, entry_point, metadata)
```
### Node Type Mapping
| Actor Node | LangGraph Type |
|------------|----------------|
| `agent` | AGENT |
| `tool` | TOOL |
| `conditional` | CONDITIONAL |
| `subgraph` | SUBGRAPH |
### Compilation Errors
| Error | Class | When |
|-------|-------|------|
| Non-GRAPH type | `ActorCompilationError` | config.type != GRAPH |
| Missing route | `ActorCompilationError` | config.route is None |
| Missing node | `MissingNodeError` | edge refs unknown node |
| Invalid entry/exit | `InvalidEntryExitError` | node not in graph |
| Cycle detected | `SubgraphCycleError` | intra or cross-actor cycle |
### Graph Validation Rules
1. Unique node IDs within graph
2. Entry node exists and is valid
3. All exit_nodes reference valid nodes
4. All edge from_node/to_node exist
5. No cycles (graphs are acyclic)
6. All nodes reachable from entry_node
---
## See Also
- [Actor YAML Schema Reference](../../reference/actors_schema.md)
- [Actor Compiler Reference](../../reference/actor_compiler.md)
- [ADR-010](../../adr/ADR-010-actor-and-agent-architecture.md): Actor Architecture
- [ADR-022](../../adr/ADR-022-langchain-langgraph-integration.md): LangGraph Integration
- [ADR-031](../../adr/ADR-031-actor-abstraction-definition.md): Actor Abstraction Definition
+69
View File
@@ -0,0 +1,69 @@
# v3.1.0 — Actor Compiler + Full LLM Integration (M2) — Overview
**Released:** v3.1.0
**Milestone:** M2 — Actor Compiler + Full LLM Integration
**Theme:** Introduce the Actor system, compile YAML-defined actors into executable LangGraph graphs, and wire full LLM integration through MCP adapter, tool router, skill registry, and validation runner.
---
## Overview
v3.1.0 builds on the v3.0.0 foundation by introducing the **Actor system**
declarative YAML definitions for AI agents compiled into LangGraph `StateGraph`
structures at runtime. It also delivers MCP adapter (external tool server
connectivity), tool router (provider normalization), skill registry (reusable
tool collections), and validation runner (pre-apply quality gates).
---
## Goals
1. Define a declarative YAML format for actors (`llm`, `tool`, `graph`).
2. Compile GRAPH-type actors into executable LangGraph `StateGraph` structures.
3. Connect to external MCP tool servers for tool discovery and invocation.
4. Route LLM tool calls across OpenAI, Anthropic, and LangChain formats.
5. Manage reusable tool collections via the skill registry.
6. Enforce validation gates before plan apply via the validation runner.
---
## Feature Index
| Feature | Description | Documentation |
|---------|-------------|---------------|
| **Actor YAML** | Declarative defs: llm, tool, graph types | [Actor Format](actor-yaml.md) |
| **Actor Compiler** | Compile to LangGraph StateGraph | [Actor Format](actor-yaml.md) |
| **MCP Adapter** | External MCP tool server connectivity | [Integration](integration.md) |
| **Tool Router** | Normalize across OpenAI/Anthropic/LC formats | [Integration](integration.md) |
| **Skill Registry** | Persistent, composable tool collections | [Skills](skills.md) |
| **Validation Runner** | Pre-apply required/informational gates | [Skills](skills.md) |
---
## Architecture Notes
### Actor System Design
Follows the **Actor Abstraction Definition** (see [ADR-031](../../adr/ADR-031-actor-abstraction-definition.md)):
- **Declarative** — defined in YAML, not code.
- **Composable** — GRAPH actors embed other actors as subgraphs.
- **Compiled** — validated and compiled to LangGraph at load time.
- **Namespaced**`namespace/name` format required.
### LangGraph Integration
See [ADR-022](../../adr/ADR-022-langchain-langgraph-integration.md). The compiler
takes GRAPH actors through an 8-step pipeline, producing LangGraph `StateGraph` structures.
### MCP & Skills
MCP adapter implements Model Context Protocol (see [ADR-029](../../adr/ADR-029-model-context-protocol.md)). Skill registry implements the Skill Abstraction Definition (see [ADR-030](../../adr/ADR-030-skill-abstraction-definition.md) and [ADR-012](../../adr/ADR-012-skill-system.md)).
---
## See Also
- [Actor Format & Compiler](actor-yaml.md)
- [MCP + Tool Router](integration.md)
- [Skills + Validation Runner](skills.md)
- [Quick Start Guide](quickstart.md) — common workflows
+91
View File
@@ -0,0 +1,91 @@
# v3.1.0 — MCP Adapter & Tool Router
External tool server integration and LLM provider normalization.
---
## MCP Adapter
The Model Context Protocol adapter connects CleverAgents to external tool servers.
### Supported Transports
| Transport | Description | Config Required |
|-----------|-------------|-----------------|
| `stdio` | Spawns subprocess | `command` |
| `sse` | Server-Sent Events HTTP | `url` |
| `streamable-http` | Streamable HTTP | `url` |
### Configuration & Usage
```python
from cleveragents.mcp.adapter import MCPToolAdapter, MCPServerConfig
config = MCPServerConfig(
name="my-tools", transport="stdio", command="python",
args=["-m", "my_mcp_server"], env={"API_KEY": "secret"},
)
adapter = MCPToolAdapter(config)
adapter.connect(timeout=10.0)
tools = adapter.discover_tools()
result = adapter.invoke("create_issue", {"title": "Bug"})
# result: {success: true, data: {"issue_id": 42}, duration_ms: 145.3}
adapter.disconnect()
```
### ToolRegistry Integration
```python
from cleveragents.tool.registry import ToolRegistry
registry = ToolRegistry()
adapter.register_tools(registry, namespace="my-tools")
# Tools available as "my-tools/<tool_name>"
```
### Capability Inference (name-based heuristics)
- `read`, `get`, `list`, `search`, `find` — inferred as read-only
- `write`, `create`, `update`, `delete`, `set` — inferred as writable
### Reconnect on Failure
```python
try: adapter.invoke("tool", {"arg": "x"})
except RuntimeError:
adapter.reconnect(timeout=10.0)
adapter.discover_tools()
```
---
## Tool Router
Normalizes LLM provider tool calls into a unified internal representation.
### Supported Provider Formats
| Provider | Shape | Arguments Key | Type |
|----------|-------|---------------|------|
| OpenAI | `{"name": "...", "arguments": "..."}` | `arguments` | JSON string |
| Anthropic | `{"name": "...", "input": {...}}` | `input` | dict |
| LangChain | `{"name": "...", "type": "tool_call", "args": {...}}` | `args` | dict |
### Usage
```python
from cleveragents.tool.router import ToolCallRouter
router = ToolCallRouter(registry, runner, plan_id="plan-001")
result = router.route({"name": "files/read_file", 'arguments': '{"path": "a.py"}'})
# result.provider_format: "openai" result.tool_call_id: "tc_<24-hex>"
# Stable IDs: generate_tool_call_id("plan-001", 0) always returns same value
```
---
## See Also
- [MCP Tool Adapter Reference](../../reference/mcp_adapter.md)
- [Tool Call Router Reference](../../reference/tool_router.md)
- [ADR-011](../../adr/ADR-011-tool-system.md): Tool System
- [ADR-029](../../adr/ADR-029-model-context-protocol.md): MCP Adoption
+66
View File
@@ -0,0 +1,66 @@
# v3.1.0 — Quick Start Guide
Common workflows: actor definitions, MCP setup, skills, & validation gates.
---
## Step-by-Step Workflow
```bash
# 1. Define a graph actor YAML
cat > actors/tdd.yaml << 'EOF'
version: "3"
name: assistants/tdd
type: graph
model: gpt-4
route:
nodes:
- id: planner; type: agent; name: Planner
config: { model: gpt-4, prompt: "Plan tasks from requirements." }
- id: implementer; type: agent; name: Implementer
config: { model: gpt-4, prompt: "Write clean tested code." }
edges: [{from_node: planner, to_node: implementer}]
entry_node: planner
exit_nodes: [implementer]
EOF
# 2. Create an action using the actor
cat > review-action.yaml << 'EOF'
name: local/code-review
description: Review code changes
execution_actor: assistants/tdd
definition_of_done: |
All changes reviewed for correctness and style.
EOF
agents action create --config review-action.yaml
# 3. Configure MCP tool server in ~/.cleveragents/config.toml
# 4. Register a skill
agents skill add --config my-skill.yaml
# 5. Execute plan with validation gate
agents plan use local/code-review my-project
agents plan execute # runs validation before apply
agents plan diff
agents plan apply --yes # blocked if required validations fail
```
---
## Actor Creation Checklist
- [ ] `version: "3"` specified
- [ ] Valid type: `llm`, `tool`, or `graph`
- [ ] Name follows `namespace/name` format
- [ ] All referenced actors and tools are resolvable
- [ ] Graph actors have valid entry/exit nodes and acyclic edges
---
## Skill & Validation Quick Tips
- Skills can include other skills for composition
- Agent Skills Standard discovers skills from filesystem paths in `agent_skills_paths`
- Validations run deterministically: alphabetical (resource, mode, name) order
- Run with `max_workers=1` in CI for predictable sequential validation
+127
View File
@@ -0,0 +1,127 @@
# v3.1.0 — Skill Registry & Validation Runner
Reusable tool collections and pre-apply validation quality gates.
---
## Skill Registry
Persistent storage for named, reusable tool collections in SQLite.
### Item Types
| Type | Description |
|------|-------------|
| `tool_ref` | Reference to a named tool |
| `include` | Recursive inclusion of another skill |
| `inline_tool` | Anonymous tool defined inline |
| `mcp_source` | MCP server tool source |
| `agent_source` | Agent Skills Standard folder source |
### CLI Commands
```bash
agents skill list # all skills
agents skill tools local/devops-toolkit # show skill's tools
agents skill add --config ./my-skill.yaml # register from YAML
agents skill remove local/my-skill # remove a skill
```
### Service API
```python
from cleveragents.application.services import SkillRegistryService
svc = SkillRegistryService(skill_repo=SkillRepository(session_factory))
svc.add_skill(skill) # register
skill = svc.get_skill("local/code-tools") # retrieve
skills = svc.list_skills(namespace="local") # filter
svc.remove_skill("local/code-tools") # delete
```
### Agent Skills Standard Integration
Filesystem-based tool bundles with `SKILL.md` + YAML front-matter.
**Default discovery path:** `~/.cleveragents/agent_skills`
**SKILL.md format:**
```markdown
---
name: my-tool
description: A custom agent skill
input_schema: {type: object, properties: {query: {type: string}}}
---
# My Tool
Additional documentation.
```
### Configuration
```toml
[skills]
agent_skills_paths = "/home/user/.cleveragents/agent_skills,/opt/skills"
```
---
## Validation Runner
Executes validations against plan resources before Apply phase.
### Modes
| Mode | Pass Behavior | Fail Behavior |
|------|---------------|----------------|
| `required` | OK | **Blocks apply** — sets `all_required_passed = False` |
| `informational` | Logged | Logged only — does NOT block apply |
### Attachment Scopes
| Scope | Active When |
|-------|-------------|
| direct | Always for the resource |
| project | Resource accessed via a project |
| plan | Only for specific plan |
### Apply Validation Gate
```python
from cleveragents.validation.gate import ApplyValidationGate
gate = ApplyValidationGate(runner=DefaultValidationRunner())
summary = gate.run(plan_id, attachments, context)
if gate.should_block_apply(summary):
# report failure reasons, block apply
else:
# proceed
```
### Blocking vs Allowed Output
**Blocked:**
```
Validation Gate: BLOCKED
Required: 1 passed, 1 failed
Failures: lint-check on res-001: 3 lint errors found
```
**Allowed:**
```
Validation Gate: PASSED
Required: 2 passed, 0 failed
Informational: 1 passed, 0 failed
```
### Concurrency
Uses `ThreadPoolExecutor` with configurable `max_workers` (default 4).
---
## See Also
- [Skill Registry Reference](../../reference/skill_registry.md)
- [Validation Pipeline Reference](../../reference/validation_pipeline.md)
- [ADR-012](../../adr/ADR-012-skill-system.md): Skill System
- [ADR-013](../../adr/ADR-013-validation-abstraction.md): Validation Abstraction
+67 -54
View File
@@ -88,8 +88,8 @@ via the `getExtendedAgentCard` operation.
## Transport Modes
| Mode | Transport | Class / SDK Component | Behaviour |
|--------|------------------------|------------------------------------|--------------------------------------------------------------------|
| Mode | Transport | Class / SDK Component | Behaviour |
|--------|------------------------|------------------------------------|--------------------------------------------------------|
| Local | A2A JSON-RPC over stdio | `A2aLocalFacade` + JSON-RPC stdio | Agent as subprocess; extensions resolved in-process |
| Server | A2A JSON-RPC over HTTP | A2A SDK HTTP transport | JSON-RPC 2.0 to CleverAgents server |
@@ -165,7 +165,7 @@ Pattern applies to all entity types (`actor`, `skill`, `tool`, `validation`,
### Context Operations
| Method | Service Method | Required Params |
|--------|----------------|-----------------|
|--------|---------------|-----------------|
| `_cleveragents/context/show` | `ContextService.show()` | `project_name` |
| `_cleveragents/context/inspect` | `ContextService.inspect()` | `project_name` |
| `_cleveragents/context/simulate` | `ContextService.simulate()` | `project_name`, simulation params |
@@ -174,7 +174,7 @@ Pattern applies to all entity types (`actor`, `skill`, `tool`, `validation`,
### Sync Operations
| Method | Service Method | Required Params |
|--------|----------------|-----------------|
|--------|---------------|-----------------|
| `_cleveragents/sync/pull` | `SyncService.pull()` | `namespace` |
| `_cleveragents/sync/push` | `SyncService.push()` | `namespace`, `entities` |
| `_cleveragents/sync/status` | `SyncService.status()` | `namespace` (optional) |
@@ -182,7 +182,7 @@ Pattern applies to all entity types (`actor`, `skill`, `tool`, `validation`,
### Namespace Operations
| Method | Service Method | Required Params |
|--------|----------------|-----------------|
|--------|---------------|-----------------|
| `_cleveragents/namespace/list` | `NamespaceService.list()` | -- |
| `_cleveragents/namespace/show` | `NamespaceService.show()` | `namespace` |
| `_cleveragents/namespace/members` | `NamespaceService.members()` | `namespace` |
@@ -198,20 +198,13 @@ Pattern applies to all entity types (`actor`, `skill`, `tool`, `validation`,
## Streaming Events
A2A streaming uses SSE via the `message/stream` operation (client → server
request), delivering typed events as the task progresses. The SSE data
payloads are JSON-RPC 2.0 **notifications** (no `id` field):
A2A streaming uses SSE via the `message/stream` operation, delivering typed
events as the task progresses:
| Event Type | JSON-RPC Method | Emitted When |
|------------|-----------------|--------------|
| `TaskStatusUpdateEvent` | `task/statusUpdate` | Task state changes (submitted -> working -> completed, etc.) |
| `TaskArtifactUpdateEvent` | `task/artifactUpdate` | Agent produces output (response chunks, plan entries, tool results) |
> **Note:** `message/stream` is the **request** method (client → server) that
> initiates streaming. The SSE data payloads use `task/statusUpdate` or
> `task/artifactUpdate` as the notification method name — not `message/stream`.
> Non-spec fields (`event_id`, `event_type`, `timestamp`) are carried in SSE
> envelope headers (`event:` and `id:` lines), not in the data payload.
| Event Type | Emitted When |
|-----------|-------------|
| `TaskStatusUpdateEvent` | Task state changes (submitted -> working -> completed, etc.) |
| `TaskArtifactUpdateEvent` | Agent produces output (response chunks, plan entries, tool results) |
### Task Lifecycle States
@@ -227,30 +220,16 @@ payloads are JSON-RPC 2.0 **notifications** (no `id` field):
### Example Streaming Event
SSE data payload for a task status update (JSON-RPC 2.0 notification):
```json
{
"jsonrpc": "2.0",
"method": "task/statusUpdate",
"method": "message/stream",
"params": {
"taskId": "task_01HXR...",
"status": { "state": "working" }
}
}
```
SSE data payload for an artifact update:
```json
{
"jsonrpc": "2.0",
"method": "task/artifactUpdate",
"params": {
"taskId": "task_01HXR...",
"artifact": {
"id": "task_01HXR...",
"status": { "state": "working" },
"artifacts": [{
"parts": [{ "text": "I'll start by extracting..." }]
}
}]
}
}
```
@@ -298,9 +277,9 @@ response = facade.dispatch(A2aRequest(
### Constructor
| Parameter | Type | Default | Description |
|------------|----------------------------------|---------|--------------------------------|
| `services` | `dict[str, Any] \| None` | `None` | Named services for routing |
| Parameter | Type | Default | Description |
|------------|----------------------------|---------|-------------------------------|
| `services` | `dict[str, Any] \| None` | `None` | Named services for routing |
### Methods
@@ -320,15 +299,15 @@ registered later with `register_service()`.
### Service Keys
| Key | Type | Wired Methods |
|----------------------------------|----------------------------------|--------------------------------------------|
| `session_service` | `SessionWorkflow` | Standard message operations |
| `plan_lifecycle_service` | `PlanLifecycleService` | `_cleveragents/plan/*` |
| `tool_registry` | `ToolRegistry` | `_cleveragents/registry/tool/*` |
| `resource_registry_service` | `ResourceRegistryService` | `_cleveragents/registry/resource/*` |
| `sync_service` | `SyncService` | `_cleveragents/sync/*` |
| `namespace_service` | `NamespaceService` | `_cleveragents/namespace/*` |
| `context_service` | `ContextService` | `_cleveragents/context/*` |
| Key | Type | Wired Methods |
|------------------------------|-----------------------------|-------------------------------------|
| `session_service` | `SessionWorkflow` | Standard message operations |
| `plan_lifecycle_service` | `PlanLifecycleService` | `_cleveragents/plan/*` |
| `tool_registry` | `ToolRegistry` | `_cleveragents/registry/tool/*` |
| `resource_registry_service` | `ResourceRegistryService` | `_cleveragents/registry/resource/*` |
| `sync_service` | `SyncService` | `_cleveragents/sync/*` |
| `namespace_service` | `NamespaceService` | `_cleveragents/namespace/*` |
| `context_service` | `ContextService` | `_cleveragents/context/*` |
---
@@ -423,8 +402,42 @@ CleverAgentsError
└── A2aOperationNotFoundError
```
| Exception | When Raised |
|------------------------------|--------------------------------------------------------------------|
| `A2aNotAvailableError` | Server-mode operation attempted without connection |
| `A2aVersionMismatchError` | Unsupported A2A version |
| `A2aOperationNotFoundError` | Unknown method dispatched |
| Exception | When Raised |
|-----------------------------|----------------------------------------------------|
| `A2aNotAvailableError` | Server-mode operation attempted without connection |
| `A2aVersionMismatchError` | Unsupported A2A version |
| `A2aOperationNotFoundError` | Unknown method dispatched |
---
## ACP to A2A Migration
> **v3.6.0 breaking change.** The `cleveragents.acp` module was renamed to
> `cleveragents.a2a` and all `Acp*` symbols were renamed to `A2a*`. The
> request/response wire format was updated to comply with JSON-RPC 2.0.
For a complete migration guide including symbol rename tables, field name
changes, operation name mapping, and YAML configuration updates, see
[`docs/development/acp-to-a2a-migration.md`](../development/acp-to-a2a-migration.md).
### Quick Reference: Deprecated Legacy Operations
The following ACP operation names are accepted by `A2aLocalFacade.dispatch()`
in `cleveragents.a2a.facade.A2aLocalFacade` (commit `449c33b7`) for backward
compatibility but are **deprecated**:
| Deprecated (ACP) | Replacement (A2A) |
|---|---|
| `session.create` | `_cleveragents/session/create` |
| `session.close` | `_cleveragents/session/close` |
| `plan.create` | `_cleveragents/plan/use` |
| `plan.execute` | `_cleveragents/plan/execute` |
| `plan.status` | `_cleveragents/plan/status` |
| `plan.diff` | `_cleveragents/plan/diff` |
| `plan.apply` | `_cleveragents/plan/apply` |
| `registry.list_tools` | `_cleveragents/registry/tool/list` |
| `registry.list_resources` | `_cleveragents/registry/resource/list` |
| `context.get` | `_cleveragents/context/show` |
See [ADR-047](../adr/ADR-047-acp-standard-adoption.md) for the architectural
rationale behind this change.
-71
View File
@@ -1,71 +0,0 @@
@acms-parallel-indexing
Feature: ACMS parallel indexing optimization (#9981)
Verifies that the FileTraversalEngine uses ThreadPoolExecutor-based
parallel processing to efficiently index large projects with 10,000+ files.
This satisfies the PR #9981 acceptance criterion: "Parallel processing
reduces indexing wall-clock time compared to sequential chunking."
Background:
The parallel engine uses configurable worker threads (max_workers)
distributed across file stat/reading operations while preserving
thread-safe progress tracking and deterministic chunk boundaries.
Scenario: Parallel FileTraversalEngine processes 100 files without timeout
Given a test directory with 100 Python files
And a parallel file traversal engine with 4 workers and chunk size 25
When I traverse and index the directory in parallel
Then the parallel index should contain 100 entries
And the indexing should complete within 30 seconds
Scenario: Parallel FileTraversalEngine skips binary files via null-byte check
Given a test directory with 50 text files and 50 binary PNG files
And a parallel file traversal engine with 2 workers
When I traverse and index the directory in parallel
Then the parallel index should contain approximately 50 entries
And all indexed entries should have text (not binary) content
Scenario: Parallel FileTraversalEngine respects exclusion patterns from .gitignore
Given a test directory with Python files and a node_modules subdirectory
And a parallel file traversal engine with 2 workers
When I traverse and index the directory in parallel with exclusions
Then the parallel index should not contain any node_modules paths
Scenario: Parallel FileTraversalEngine respects exclusion patterns from .acmsignore
Given a test directory with Python files and a __pycache__ subdirectory
And a parallel file traversal engine with 2 workers
When I traverse and index the directory in parallel with acms exclusions
Then the parallel index should not contain any __pycache__ paths
Scenario: Parallel FileTraversalEngine with max_workers=1 runs sequentially
Given a test directory with 50 Python files
And a sequential file traversal engine (max_workers=1) with chunk size 25
When I traverse and index the directory in parallel
Then the sequential index should contain 50 entries
Scenario: Parallel FileTraversalEngine tracks thread-safe progress atomically
Given a test directory with 200 Python files
And a parallel file traversal engine with 4 workers with progress tracking
When I traverse and index the directory in parallel
Then the progress snapshot should show all 200 files indexed
And no errors should have occurred during indexing
Scenario: Parallel FileTraversalEngine saves JSON cache to disk
Given a test directory with 100 Python files
And a parallel file traversal engine with 4 workers and cache at "/tmp/parallel_cache.json"
When I traverse and index the directory in parallel
Then the cache file should exist at the specified path
And the cache should contain valid JSON with all indexed entries
Scenario: Parallel FileTraversalEngine handles permission errors gracefully
Given a test directory with 50 Python files and one unreadable subdirectory
And a parallel file traversal engine with 4 workers
When I traverse and index the directory in parallel
Then the parallel index should contain at least 48 entries
And the progress snapshot should show zero or few errors
Scenario: Parallel FileTraversalEngine processes chunk order deterministically
Given a test directory with files named "file_00.py" through "file_99.py"
And a parallel file traversal engine with 4 workers and chunk size 50
When I traverse and index the directory in parallel
Then all entries should have valid file paths matching the naming pattern
@@ -1,376 +0,0 @@
"""Step definitions for acms_parallel_indexing.feature.
Validates that the parallel FileTraversalEngine correctly uses ThreadPoolExecutor
to index large projects with multiple worker threads, while preserving thread safety
and deterministic results.
References:
- src/cleveragents/acms/index.py (FileTraversalEngine)
- Issue #9981 / PR #9981
"""
from __future__ import annotations
import json
import os
import shutil
import tempfile
import time
from pathlib import Path
from typing import Any
from behave import given, then, when
from cleveragents.acms.index import (
FileTraversalEngine,
IndexProgress,
)
def _create_test_directory_with_py_files(
directory: str, count: int, prefix: str = "file"
) -> None:
"""Create ``count`` Python source files spread across subdirectories."""
root = Path(directory)
for i in range(count):
subdir = root / f"pkg_{i % 10}"
subdir.mkdir(parents=True, exist_ok=True)
(subdir / f"{prefix}_{i:03d}.py").write_text(
f"# Module {i}\nVALUE = {i}\nx = {i} * 2\n", encoding="utf-8"
)
def _create_test_directory_with_binary_files(
directory: str, text_count: int, binary_count: int
) -> None:
"""Create a mix of text and binary (PNG) files."""
root = Path(directory)
half = text_count // 2
for b in range(half):
(root / f"text_{b:02d}").mkdir(parents=True, exist_ok=True)
for i in range(text_count):
bucket = i % half
(root / f"text_{bucket:02d}" / f"file_{i:03d}.py").write_text(
f"# text file {i}\n", encoding="utf-8"
)
half_bin = binary_count // 2
for b in range(half_bin):
(root / f"bin_{b:02d}").mkdir(parents=True, exist_ok=True)
for i in range(binary_count):
bucket = i % half_bin
(root / f"bin_{bucket:02d}" / f"image_{i:03d}.png").write_bytes(
b"\x89PNG\r\n\x1a\n" + bytes(range(256)) * 4
)
def _create_test_directory_with_gitignore(
directory: str, py_count: int, node_modules_entries: int = 30
) -> None:
"""Create Python files plus a node_modules dir with fake JS files + .gitignore."""
root = Path(directory)
for i in range(py_count):
subdir = root / f"src_{i % 5}"
subdir.mkdir(parents=True, exist_ok=True)
(subdir / f"module_{i:03d}.py").write_text(
f"# src module {i}\n", encoding="utf-8"
)
nm = root / "node_modules"
nm.mkdir(parents=True, exist_ok=True)
for i in range(node_modules_entries):
(nm / f"pkg_{i}.js").write_text(f"console.log({i});\n", encoding="utf-8")
gitignore = root / ".gitignore"
gitignore.write_text("node_modules\n", encoding="utf-8")
def _create_test_directory_with_acmsignore(
directory: str, py_count: int, pycache_entries: int = 20
) -> None:
"""Create Python files plus __pycache__ dirs with fake bytecode + .acmsignore."""
root = Path(directory)
for i in range(py_count):
subdir = root / f"src_{i % 5}"
subdir.mkdir(parents=True, exist_ok=True)
(subdir / f"module_{i:03d}.py").write_text(
f"# src module {i}\n", encoding="utf-8"
)
for i in range(pycache_entries):
(root / "__pycache__" / f"module_{i:02d}.cpython-312.pyc").write_bytes(
bytes([i] * 64)
)
acmsignore = root / ".acmsignore"
acmsignore.write_text("__pycache__\n", encoding="utf-8")
def _create_test_directory_with_unreadable_dir(directory: str, py_count: int):
"""Create Python files plus one subdirectory we cannot read."""
import platform
if platform.system() == "Windows":
# On Windows we create a regular dir but the engine handles PermissionError.
root = Path(directory)
forbidden = root / "forbidden"
forbidden.mkdir(parents=True, exist_ok=True)
return
root = Path(directory)
for i in range(py_count):
subdir = root / f"src_{i % 5}"
subdir.mkdir(parents=True, exist_ok=True)
(subdir / f"module_{i:03d}.py").write_text(
f"# src module {i}\n", encoding="utf-8"
)
# Create a subdirectory that we can't read.
forbidden = root / "forbidden_dir"
forbidden.mkdir(parents=True, exist_ok=True)
(forbidden / "secret.py").write_text("# secret\n", encoding="utf-8")
os.chmod(forbidden, 0o000)
# ---------------------------------------------------------------------------
# Given - test data fixtures
# ---------------------------------------------------------------------------
@given("a test directory with {count:d} Python files")
def step_create_py_dir(context: Any, count: int) -> None:
_create_test_directory_with_py_files(
directory=tempfile.mkdtemp(prefix="parallel-idx-"),
count=count,
prefix="file",
)
@given(
"a test directory with {text_count:d} text files and {binary_count:d} binary PNG files"
)
def step_create_mixed_file_dir(
context: Any, text_count: int, binary_count: int
) -> None:
d = tempfile.mkdtemp(prefix="parallel-idx-mixed-")
_create_test_directory_with_binary_files(d, text_count, binary_count)
context.temp_dir_path = d
context.add_cleanup(shutil.rmtree, d, True)
@given("a test directory with Python files and a node_modules subdirectory")
def step_create_gitignore_dir(context: Any) -> None:
d = tempfile.mkdtemp(prefix="parallel-idx-giti-")
_create_test_directory_with_gitignore(d, py_count=50)
context.temp_dir_path = d
context.add_cleanup(shutil.rmtree, d, True)
@given("a test directory with Python files and a __pycache__ subdirectory")
def step_create_acmsignore_dir(context: Any) -> None:
d = tempfile.mkdtemp(prefix="parallel-idx-acms-")
_create_test_directory_with_acmsignore(d, py_count=50)
context.temp_dir_path = d
context.add_cleanup(shutil.rmtree, d, True)
@given('a test directory with files named "file_00.py" through "file_99.py"')
def step_create_named_files(context: Any) -> None:
d = tempfile.mkdtemp(prefix="parallel-idx-named-")
_create_test_directory_with_py_files(d, count=100)
context.temp_dir_path = d
context.add_cleanup(shutil.rmtree, d, True)
@given(
"a parallel file traversal engine with {workers:d} workers and chunk size {chunk:d}"
)
def step_create_parallel_engine(context: Any, workers: int, chunk: int) -> None:
engine = FileTraversalEngine(chunk_size=chunk, max_workers=workers)
context.parallel_engine = engine
@given("a parallel file traversal engine with {workers:d} workers")
def step_create_parallel_engine_simple(context: Any, workers: int) -> None:
engine = FileTraversalEngine(chunk_size=25, max_workers=workers)
context.parallel_engine = engine
@given("a sequential file traversal engine (max_workers=1) with chunk size {chunk:d}")
def step_create_sequential_engine(context: Any, chunk: int) -> None:
engine = FileTraversalEngine(chunk_size=chunk, max_workers=1)
context.parallel_engine = engine
@given(
"a parallel file traversal engine with {workers:d} workers with progress tracking"
)
def step_create_parallel_engine_with_progress(context: Any, workers: int) -> None:
progress = IndexProgress()
engine = FileTraversalEngine(chunk_size=25, max_workers=workers, progress=progress)
context.parallel_engine = engine
context.progress_tracker = progress
@given(
'a parallel file traversal engine with {workers:d} workers and cache at "{cache_path}"'
)
def step_create_parallel_engine_with_cache(
context: Any, workers: int, cache_path: str
) -> None:
engine = FileTraversalEngine(
chunk_size=25, max_workers=workers, cache_path=cache_path
)
context.parallel_engine = engine
@given("a test directory with {count:d} Python files and one unreadable subdirectory")
def step_create_partial_forbidden_dir(context: Any, count: int) -> None:
d = tempfile.mkdtemp(prefix="parallel-idx-fb-")
_create_test_directory_with_unreadable_dir(d, py_count=count)
context.temp_dir_path = d
context.add_cleanup(shutil.rmtree, d, True)
# ---------------------------------------------------------------------------
# When - indexing actions
# ---------------------------------------------------------------------------
@when("I traverse and index the directory in parallel")
def step_traverse_parallel(context: Any) -> None:
"""Run traversal with the pre-configured engine."""
if not hasattr(context, "temp_dir_path"):
# Create inline temp dir for scenarios using generic Given
d = tempfile.mkdtemp(prefix="parallel-idx-wf-")
_create_test_directory_with_py_files(d, count=100)
context.temp_dir_path = d
start = time.monotonic()
engine = context.parallel_engine
engine.reset_index()
context.index = engine.traverse_and_index(context.temp_dir_path)
context.index_elapsed = time.monotonic() - start
if hasattr(context, "progress_tracker"):
context.progress_snapshot = context.progress_tracker.snapshot()
@when("I traverse and index the directory in parallel with exclusions")
def step_traverse_parallel_exclusions(context: Any) -> None:
engine = context.parallel_engine
engine.reset_index()
context.index = engine.traverse_and_index(
context.temp_dir_path, exclude_patterns=["node_modules"]
)
@when("I traverse and index the directory in parallel with acms exclusions")
def step_traverse_parallel_acms_exclusions(context: Any) -> None:
engine = context.parallel_engine
engine.reset_index()
context.index = engine.traverse_and_index(
context.temp_dir_path, exclude_patterns=["__pycache__"]
)
# ---------------------------------------------------------------------------
# Then - assertions
# ---------------------------------------------------------------------------
@then("the parallel index should contain {count:d} entries")
def step_check_parallel_count(context: Any, count: int) -> None:
actual = context.index.get_entry_count()
assert actual == count, f"Expected {count} entries, got {actual}"
@then("the parallel index should contain approximately {count:d} entries")
def step_check_parallel_approx_count(context: Any, count: int) -> None:
actual = context.index.get_entry_count()
assert abs(actual - count) <= count * 0.15, (
f"Expected ~{count} entries, got {actual}"
)
@then("the indexing should complete within {seconds:d} seconds")
def step_check_indexing_time(context: Any, seconds: int) -> None:
elapsed = context.index_elapsed
assert elapsed < seconds, (
f"Indexing took {elapsed:.2f}s, exceeding {seconds}s limit."
)
@then("all indexed entries should have text (not binary) content")
def step_check_all_text_entries(context: Any) -> None:
for entry in context.index.get_all_entries():
assert not str(entry.path).endswith(".png"), (
f"Binary PNG file was incorrectly indexed: {entry.path}"
)
@then("the parallel index should not contain any {pattern:s} paths")
def step_check_no_pattern_paths(context: Any, pattern: str) -> None:
for entry in context.index.get_all_entries():
assert pattern not in entry.path, (
f"Excluded path '{pattern}' found in index: {entry.path}"
)
@then("the sequential index should contain {count:d} entries")
def step_check_sequential_count(context: Any, count: int) -> None:
actual = context.index.get_entry_count()
assert actual == count, f"Expected {count} entries, got {actual}"
@then("the progress snapshot should show all 200 files indexed")
def step_check_progress_200(context: Any) -> None:
snap = context.progress_snapshot
assert snap["files_indexed"] == 200, (
f"Expected 200 indexed, got {snap['files_indexed']}"
)
@then("no errors should have occurred during indexing")
def step_check_no_errors(context: Any) -> None:
snap = context.progress_snapshot
assert snap["errors_occurred"] == 0, (
f"Expected 0 errors, got {snap['errors_occurred']}"
)
@then("the cache file should exist at the specified path")
def step_check_cache_exists(context: Any) -> None:
# The cache path is determined from the engine.
assert context.parallel_engine.cache_path is not None
assert Path(context.parallel_engine.cache_path).exists(), (
f"Cache file does not exist at {context.parallel_engine.cache_path}"
)
@then("the cache should contain valid JSON with all indexed entries")
def step_check_cache_valid_json(context: Any) -> None:
cp = context.parallel_engine.cache_path
assert cp is not None
data = json.loads(Path(cp).read_text(encoding="utf-8"))
entries = data.get("entries", [])
count = data.get("count", 0)
actual_indexed = context.index.get_entry_count()
assert len(entries) == actual_indexed, (
f"Cache has {len(entries)} entries but index has {actual_indexed}"
)
assert count == actual_indexed
@then("the parallel index should contain at least {count:d} entries")
def step_check_parallel_min_count(context: Any, count: int) -> None:
actual = context.index.get_entry_count()
assert actual >= count, f"Expected at least {count} entries, got {actual}"
@then("the progress snapshot should show zero or few errors")
def step_check_few_errors(context: Any) -> None:
snap = context.progress_snapshot
assert snap["errors_occurred"] <= 5, (
f"Expected at most 5 errors, got {snap['errors_occurred']}"
)
@then("all entries should have valid file paths matching the naming pattern")
def step_check_named_entries(context: Any) -> None:
for entry in context.index.get_all_entries():
path = entry.path
assert "/file_" in path or "file_" in path, (
f"Entry path does not match expected naming pattern: {path}"
)
+94 -91
View File
@@ -1,13 +1,8 @@
# yaml-language-server: $schema=https://json.schemastore.org/mkdocs-1.6.json
# yaml-language-server: customTags:
# - !ENV scalar
# - !ENV sequence
site_name: CleverAgents Documentation
site_description: Documentation for CleverAgents.
site_author: CleverThis, Inc.
site_url: https://docs.cleverthis.com/cleveragents
site_dir: build/site
nav:
- Specification: specification.md
- Architecture: architecture.md
@@ -52,6 +47,19 @@ nav:
- Automation Tracking: development/automation-tracking.md
- Custom Sandbox Strategy: development/custom_sandbox_strategy.md
- Documentation Writer: development/docs-writer.md
- ACP to A2A Migration: development/acp-to-a2a-migration.md
- Milestones:
- Overview: milestones/index.md
- "v3.0.0 \u2014 Minimal Local Source-Code Workflow (M1)":
- milestones/v3.0.0/index.md
- milestones/v3.0.0/cli-reference.md
- milestones/v3.0.0/deep-dive.md
- "v3.1.0 \u2014 Actor Compiler + Full LLM Integration (M2)":
- milestones/v3.1.0/index.md
- milestones/v3.1.0/actor-yaml.md
- milestones/v3.1.0/integration.md
- milestones/v3.1.0/skills.md
- milestones/v3.1.0/quickstart.md
- Implementation Timeline: timeline.md
- Advanced Concepts (v3.6.0):
- Overview & Context Strategies: advanced-concepts/index.md
@@ -118,105 +126,100 @@ nav:
- ADR-046 TUI Reference and Command System: adr/ADR-046-tui-reference-and-command-system.md
- ADR-047 A2A Standard Adoption: adr/ADR-047-acp-standard-adoption.md
- ADR-048 Server Application Architecture: adr/ADR-048-server-application-architecture.md
theme:
name: material
custom_dir: docs/overrides
features:
- navigation.tabs
- navigation.tabs.sticky
- toc.integrate
- toc.follow
- navigation.tabs
- navigation.tabs.sticky
- toc.integrate
- toc.follow
palette:
- scheme: default
primary: blue
accent: cyan
toggle:
icon: material/brightness-7
name: Switch to dark mode
- scheme: slate
primary: blue
accent: cyan
toggle:
icon: material/brightness-4
name: Switch to light mode
- scheme: default
primary: blue
accent: cyan
toggle:
icon: material/brightness-7
name: Switch to dark mode
- scheme: slate
primary: blue
accent: cyan
toggle:
icon: material/brightness-4
name: Switch to light mode
hooks:
- hooks/adr_hooks.py
- hooks/adr_hooks.py
extra_css:
- stylesheets/extra.css
- stylesheets/extra.css
extra_javascript:
- javascripts/adr-page.js
- javascripts/toc-collapse.js
- javascripts/diagram-lightbox.js
- javascripts/adr-page.js
- javascripts/toc-collapse.js
- javascripts/diagram-lightbox.js
extra:
adr_tiers:
1:
title: "Foundational"
description: "Structural and technological foundation upon which all other decisions rest."
title: Foundational
description: Structural and technological foundation upon which all other decisions
rest.
2:
title: "Core Domain"
description: "Domain model entities, lifecycles, and relationships that constitute core logic."
title: Core Domain
description: "Domain model \u2014 entities, lifecycles, and relationships that\
\ constitute core logic."
3:
title: "Infrastructure and Behavior"
description: "Cross-cutting behavioral systems and infrastructure concerns."
title: Infrastructure and Behavior
description: Cross-cutting behavioral systems and infrastructure concerns.
4:
title: "Integration and Operations"
description: "External integrations, operational interfaces, and deployment concerns."
title: Integration and Operations
description: External integrations, operational interfaces, and deployment concerns.
plugins:
- search
- gen-files:
scripts:
- docs/gen_ref_pages.py
- literate-nav:
nav_file: SUMMARY.md
- mkdocstrings:
handlers:
python:
paths: [src]
options:
docstring_style: google
docstring_section_style: table
show_root_heading: true
show_source: true
merge_init_into_class: true
separate_signature: true
show_signature_annotations: true
signature_crossrefs: true
summary: true
extensions:
- griffe_pydantic:
schema: true
- kroki:
server_url: https://kroki.qoto.org
#server_url: https://kroki.io
enable_mermaid: true
fence_prefix: kroki-
http_method: POST
- search
- gen-files:
scripts:
- docs/gen_ref_pages.py
- literate-nav:
nav_file: SUMMARY.md
- mkdocstrings:
handlers:
python:
paths:
- src
options:
docstring_style: google
docstring_section_style: table
show_root_heading: true
show_source: true
merge_init_into_class: true
separate_signature: true
show_signature_annotations: true
signature_crossrefs: true
summary: true
extensions:
- griffe_pydantic:
schema: true
- kroki:
server_url: https://kroki.qoto.org
enable_mermaid: true
fence_prefix: kroki-
http_method: POST
markdown_extensions:
- admonition
- pymdownx.details
- pymdownx.superfences
- pymdownx.tabbed:
alternate_style: true
- pymdownx.critic
- pymdownx.caret
- pymdownx.keys
- pymdownx.mark
- pymdownx.tilde
- pymdownx.tasklist:
custom_checkbox: true
- attr_list
- def_list
- footnotes
- md_in_html
- tables
- codehilite
- toc:
permalink: true
toc_depth: 1-6
- admonition
- pymdownx.details
- pymdownx.superfences
- pymdownx.tabbed:
alternate_style: true
- pymdownx.critic
- pymdownx.caret
- pymdownx.keys
- pymdownx.mark
- pymdownx.tilde
- pymdownx.tasklist:
custom_checkbox: true
- attr_list
- def_list
- footnotes
- md_in_html
- tables
- codehilite
- toc:
permalink: true
toc_depth: 1-6
-11
View File
@@ -1201,17 +1201,6 @@ def benchmark_regression(session: nox.Session):
"""Run Airspeed Velocity benchmarks regression test."""
session.install("-e", ".[tests]")
config_path = "asv.conf.json"
# Ensure ASV result directories exist (handles cases where S3 sync is skipped)
results_dir = os.path.join("build", "asv", "results")
machine_dir = os.path.join(results_dir, "forgejo-runner")
session.run(
"mkdir",
"-p",
machine_dir,
external=True,
)
asv_base_sha = os.environ.get("ASV_BASE_SHA", "master")
session.run(
"asv",
+122 -31
View File
@@ -1,48 +1,139 @@
*** Settings ***
Documentation E2E acceptance test for M2 (v3.1.0): Actor Compiler + Full LLM Integration.
Documentation E2E acceptance test for M2 (v3.1.0): Actor Compiler + Full LLM Integration.
...
... Exercises actor YAML compilation into functional graphs, skill registry,
... tool lifecycle, and plan execution with a custom actor using real LLM keys.
... Zero mocking — all CLI invocations hit the real CleverAgents binary with
... real provider keys.
Resource common_e2e.resource
Suite Setup E2E Suite Setup
... Exercises actor YAML compilation into functional graphs, skill registry,
... tool lifecycle, and plan execution with a custom actor using real LLM keys.
... Zero mocking — all CLI invocations hit the real CleverAgents binary with
... real provider keys.
Resource common_e2e.resource
Suite Setup E2E Suite Setup
Suite Teardown E2E Suite Teardown
*** Variables ***
${ACTOR_NAME} local/m2-e2e-actor
${ACTION_NAME} local/m2-e2e-action
${RESOURCE_NAME} local/m2-e2e-repo
${PROJECT_NAME} local/m2-e2e-project
${ACTOR_NAME} local/m2-e2e-actor
${ACTION_NAME} local/m2-e2e-action
${RESOURCE_NAME} local/m2-e2e-repo
${PROJECT_NAME} local/m2-e2e-project
*** Test Cases ***
M2 Full Actor Compiler And LLM Integration
[Documentation] End-to-end acceptance test for the M2 milestone.
...
... Exercises actor YAML compilation, registration via CLI,
... resource and project setup, action creation referencing a
... custom actor, and the full plan lifecycle (use, execute
... strategize, execute, diff, apply) with real LLM API keys.
[Tags] E2E tdd_issue tdd_issue_4189 tdd_expected_fail
Skip If No LLM Keys
# ---- Step 1: Create temp git repo with sample project files ----
${repo_dir}= Create Temp Git Repo m2-e2e-repo
Create Directory ${repo_dir}${/}src
Create File ${repo_dir}${/}src${/}main.py print("hello world")\n
${r_add}= Run Process git add . cwd=${repo_dir} timeout=60s on_timeout=kill
Should Be Equal As Integers ${r_add.rc} 0 msg=git add failed (rc=${r_add.rc}). Check DEBUG logs above.
${r_commit}= Run Process git commit -m Add source files cwd=${repo_dir} timeout=60s on_timeout=kill
Should Be Equal As Integers ${r_commit.rc} 0 msg=git commit failed (rc=${r_commit.rc}). Check DEBUG logs above.
# Detect the default branch name created by git init
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir} timeout=60s on_timeout=kill
${branch}= Strip String ${branch_result.stdout}
Log Detected branch: ${branch}
[Documentation] End-to-end acceptance test for the M2 milestone.
...
... Exercises actor YAML compilation, registration via CLI,
... resource and project setup, action creation referencing a
... custom actor, and the full plan lifecycle (use, execute
... strategize, execute, diff, apply) with real LLM API keys.
[Tags] E2E
Skip If No LLM Keys
# ---- Step 1: Create temp git repo with sample project files ----
${repo_dir}= Create Temp Git Repo m2-e2e-repo
Create Directory ${repo_dir}${/}src
Create File ${repo_dir}${/}src${/}main.py print("hello world")\n
${r_add}= Run Process git add . cwd=${repo_dir} timeout=60s on_timeout=kill
Should Be Equal As Integers ${r_add.rc} 0 msg=git add failed (rc=${r_add.rc}). Check DEBUG logs above.
${r_commit}= Run Process git commit -m Add source files cwd=${repo_dir} timeout=60s on_timeout=kill
Should Be Equal As Integers ${r_commit.rc} 0 msg=git commit failed (rc=${r_commit.rc}). Check DEBUG logs above.
# Detect the default branch name created by git init
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir} timeout=60s on_timeout=kill
${branch}= Strip String ${branch_result.stdout}
Log Detected branch: ${branch}
# ---- Resolve LLM actor (OpenAI if available, Anthropic fallback) ----
${llm_actor}= Resolve LLM Actor
${llm_parts}= Evaluate $llm_actor.split('/', 1)
${llm_provider}= Set Variable ${llm_parts}[0]
${llm_model}= Set Variable ${llm_parts}[1]
# ---- Step 2: Create and register custom actor YAML ----
${actor_yaml}= Catenate SEPARATOR=\n
... name: ${ACTOR_NAME}
... type: llm
... description: M2 E2E acceptance actor for compiler and LLM integration
... version: "1.0"
... model: ${llm_model}
${actor_yaml_path}= Set Variable ${SUITE_HOME}${/}m2_actor.yaml
Create File ${actor_yaml_path} ${actor_yaml}
${actor_config}= Catenate SEPARATOR=\n
... {
... "name": "${ACTOR_NAME}",
... "provider": "${llm_provider}",
... "model": "${llm_model}",
... "options": {"temperature": 0.2}
... }
${config_path}= Set Variable ${SUITE_HOME}${/}actor_config.json
Create File ${config_path} ${actor_config}
${r_actor}= Run CleverAgents Command
... actor add --config ${config_path} --format plain
Should Be Equal As Integers ${r_actor.rc} 0 msg=Actor add failed (rc=${r_actor.rc}). Check DEBUG logs above.
Should Not Contain ${r_actor.stdout}${r_actor.stderr} Traceback
Log Actor registration: ${r_actor.stdout}
# ---- Step 3: Register resource and create project ----
${r_resource}= Run CleverAgents Command
... resource add git-checkout ${RESOURCE_NAME}
... --path ${repo_dir} --branch ${branch} --format plain
Output Should Contain ${r_resource} ${RESOURCE_NAME}
${r_project}= Run CleverAgents Command
... project create ${PROJECT_NAME}
... --description M2 E2E acceptance project
... --resource ${RESOURCE_NAME} --format plain
Output Should Contain ${r_project} ${PROJECT_NAME}
# ---- Step 4: Create action referencing the custom actor ----
${action_yaml}= Catenate SEPARATOR=\n
... name: ${ACTION_NAME}
... description: M2 acceptance test action for actor compiler and LLM integration
... definition_of_done: Generate or modify at least one source file
... strategy_actor: ${llm_actor}
... execution_actor: ${llm_actor}
${action_yaml_path}= Set Variable ${SUITE_HOME}${/}action.yaml
Create File ${action_yaml_path} ${action_yaml}
${r_action}= Run CleverAgents Command
... action create --config ${action_yaml_path} --format plain
Output Should Contain ${r_action} ${ACTION_NAME}
# ---- Step 5: Plan use ----
${r_use}= Run CleverAgents Command
... plan use ${ACTION_NAME} ${PROJECT_NAME} --format plain
Should Not Be Empty ${r_use.stdout}
${plan_ids}= Get Regexp Matches ${r_use.stdout} [0-9A-Z]{26}
Should Not Be Empty ${plan_ids} msg=Expected a ULID plan ID in plan use output
${plan_id}= Set Variable ${plan_ids}[0]
Log Extracted plan_id: ${plan_id}
# ---- Step 6: Plan execute — strategize phase ----
${r_strategize}= Run CleverAgents Command
... plan execute ${plan_id} --format plain
... timeout=180s
Should Not Contain ${r_strategize.stdout}${r_strategize.stderr} INTERNAL
Should Not Contain ${r_strategize.stdout}${r_strategize.stderr} Traceback
Log Strategize phase output: ${r_strategize.stdout}
# ---- Step 7: Plan execute — execute phase ----
${r_execute}= Run CleverAgents Command
... plan execute ${plan_id} --format plain
... timeout=180s
Should Not Contain ${r_execute.stdout}${r_execute.stderr} INTERNAL
Should Not Contain ${r_execute.stdout}${r_execute.stderr} Traceback
Log Execute phase output: ${r_execute.stdout}
# ---- Step 8: Plan diff ----
${r_diff}= Run CleverAgents Command
... plan diff ${plan_id} --format plain
Should Not Contain ${r_diff.stdout}${r_diff.stderr} INTERNAL
Should Not Contain ${r_diff.stdout}${r_diff.stderr} Traceback
Log Diff output: ${r_diff.stdout}
# ---- Step 9: Plan apply ----
${r_apply}= Run CleverAgents Command
... plan apply --yes ${plan_id} --format plain
Should Not Contain ${r_apply.stdout}${r_apply.stderr} INTERNAL
Should Not Contain ${r_apply.stdout}${r_apply.stderr} Traceback
Log Apply output: ${r_apply.stdout}
# ---- Step 10: Verify actor compilation and plan integrity ----
${r_status}= Run CleverAgents Command
... plan status ${plan_id} --format plain
Should Not Be Empty ${r_status.stdout}
Output Should Contain ${r_status} ${plan_id}
Log Final plan status: ${r_status.stdout}
+34 -406
View File
@@ -2,30 +2,20 @@
Provides the foundational data model for indexed context entries and a
file traversal engine that can handle 10,000+ files without timeout using
chunked processing with parallel worker threads.
chunked processing.
Based on issue #9579 and ``docs/specification.md`` ~lines 44405-44420.
Optimized for PR #9981: parallel processing reduces indexing time for
large projects by distributing file stat/reading work across multiple
threads while preserving thread safety and deterministic chunk boundaries.
"""
from __future__ import annotations
import json
import threading
from collections.abc import Iterator
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime
from enum import StrEnum
from pathlib import Path
from typing import TYPE_CHECKING
from pydantic import BaseModel, Field
if TYPE_CHECKING:
from typing import Any
class FileType(StrEnum):
"""File type enumeration for index entries."""
@@ -77,7 +67,7 @@ class IndexEntry(BaseModel):
modified_at: datetime
tags: set[str] = Field(default_factory=set)
tier: TierLevel = TierLevel.COLD
metadata: dict[str, Any] = Field(default_factory=dict)
metadata: dict[str, str] = Field(default_factory=dict)
def add_tag(self, tag: str) -> None:
"""Add a tag to this entry.
@@ -114,74 +104,6 @@ class IndexEntry(BaseModel):
self.tier = tier
class IndexProgress(BaseModel):
"""Thread-safe progress tracker for file traversal indexing.
Tracks how many files have been processed and their types.
Thread-safe via a single lock protecting all counters.
"""
files_processed: int = 0
files_skipped: int = 0
files_indexed: int = 0
bytes_read: int = 0
binary_files_found: int = 0
errors_occurred: int = 0
def __init__(self) -> None:
"""Initialize progress tracker with a reentrant lock."""
super().__init__()
self._lock = threading.RLock()
def record_processed(self, bytes_read: int = 0) -> None:
"""Record that a file was processed (stat+read completed)."""
with self._lock:
self.files_processed += 1
self.bytes_read += bytes_read
def record_skipped(self) -> None:
"""Record that a file was skipped (excluded/by pattern)."""
with self._lock:
self.files_skipped += 1
def record_indexed(self) -> None:
"""Record that an entry was successfully indexed."""
with self._lock:
self.files_indexed += 1
def record_binary(self) -> None:
"""Record a binary file detection."""
with self._lock:
self.binary_files_found += 1
def record_error(self) -> None:
"""Record an indexing error."""
with self._lock:
self.errors_occurred += 1
@property
def progress_percent(self) -> float:
"""Return completion percentage (0-100)."""
with self._lock:
total = self.files_processed + self.files_skipped
if total == 0:
return 0.0
return round((self.files_indexed / total) * 100, 2)
def snapshot(self) -> dict[str, Any]:
"""Return a JSON-serializable snapshot of progress."""
with self._lock:
return {
"files_processed": self.files_processed,
"files_skipped": self.files_skipped,
"files_indexed": self.files_indexed,
"bytes_read": self.bytes_read,
"binary_files_found": self.binary_files_found,
"errors_occurred": self.errors_occurred,
"progress_percent": self.progress_percent,
}
class ACMSIndex:
"""ACMS Index for storing and querying indexed context entries.
@@ -325,91 +247,34 @@ class ACMSIndex:
"""Get the total number of entries in the index."""
return len(self.entries)
def to_json_dict(self) -> dict[str, Any]:
"""Serialize the entire index to a JSON-serializable dictionary."""
return {
"entries": [entry.model_dump() for entry in self.entries.values()],
"count": len(self.entries),
}
@classmethod
def from_json_dict(cls, data: dict[str, Any]) -> ACMSIndex:
"""Deserialize an index from a JSON-serializable dictionary."""
index = cls()
for entry_data in data.get("entries", []):
entry = IndexEntry(**entry_data)
index.add_entry(entry)
return index
class FileTraversalEngine:
"""Engine for traversing and indexing files in large projects.
Handles 10,000+ files without timeout using chunked processing WITH
parallel worker threads to distribute file stat/reading work across
multiple CPU cores.
For PR #9981 optimizations:
- ThreadPoolExecutor-based parallel file processing (max_workers configurable)
- Thread-safe progress tracking via IndexProgress
- Binary file detection via null-byte heuristic
- .acmsignore pattern support for exclusion lists
- On-disk JSON cache persistence with atomic replacements
Handles 10,000+ files without timeout using chunked processing to
prevent memory exhaustion.
Attributes:
chunk_size: Number of files to process in each chunk
max_workers: Max concurrent worker threads (default: 4)
index: The ACMS index to populate
progress: Thread-safe progress tracker
cache_path: Optional path for on-disk JSON cache persistence
"""
def __init__(
self,
chunk_size: int = 100,
max_workers: int = 4,
index: ACMSIndex | None = None,
progress: IndexProgress | None = None,
cache_path: str | Path | None = None,
) -> None:
def __init__(self, chunk_size: int = 100, index: ACMSIndex | None = None) -> None:
"""Initialize the traversal engine.
Args:
chunk_size: Number of files to process per chunk (default: 100).
Must be a positive integer.
max_workers: Max concurrent worker threads (default: 4).
Controls parallelism for file stat/reading operations.
Set to 1 for sequential processing (debugging).
index: Optional pre-populated ACMSIndex to use. If None, a new
empty index is created (supports Dependency Inversion).
progress: Optional IndexProgress tracker. If None, a new one
is created.
cache_path: Optional file path for on-disk JSON cache.
The index is saved after each chunk as an atomic
replacement (write-to-temp-then-rename pattern).
Use pathlib.PosixPath for cross-platform compatibility,
avoiding raw str paths with backslashes on Windows.
Raises:
ValueError: If chunk_size or max_workers is not a positive integer.
ValueError: If chunk_size is not a positive integer.
"""
if chunk_size <= 0:
raise ValueError(f"chunk_size must be a positive integer, got {chunk_size}")
if max_workers <= 0:
raise ValueError(
f"max_workers must be a positive integer, got {max_workers}"
)
self.chunk_size = chunk_size
self.max_workers = max_workers
self.index = index if index is not None else ACMSIndex()
self.progress = progress if progress is not None else IndexProgress()
# Normalize cache_path to a Path for atomic ops.
if isinstance(cache_path, str):
self.cache_path: Path | None = Path(cache_path)
elif cache_path is not None:
self.cache_path = cache_path
else:
self.cache_path = None
def _get_file_type(self, file_path: Path) -> FileType:
"""Determine file type from extension."""
@@ -430,27 +295,6 @@ class FileTraversalEngine:
}
return type_map.get(suffix, FileType.OTHER)
@staticmethod
def _is_binary(file_path: Path) -> bool:
"""Detect binary files via null-byte heuristic.
Reads the first 8192 bytes and checks for NUL (\\x00) characters.
This is lightweight and avoids reading large binary blobs entirely.
Args:
file_path: Path to check.
Returns:
True if file appears to be binary, False otherwise.
"""
try:
with open(file_path, "rb") as f:
chunk = f.read(8192)
return b"\x00" in chunk
except OSError:
# If we can't read it, treat it as binary (skip it).
return True
def _create_index_entry(self, file_path: Path) -> IndexEntry | None:
"""Create an index entry from a file path.
@@ -472,73 +316,6 @@ class FileTraversalEngine:
except (OSError, ValueError):
return None
def _load_gitignore_patterns(self, root: Path) -> list[str]:
"""Load exclusion patterns from a .gitignore file.
Only loads basic directory patterns (lines that do not contain '/'
except in the first segment). This keeps loading O(1) regardless
of the size of .gitignore.
Args:
root: Project root directory path.
Returns:
List of exclusion pattern strings.
"""
gitignore = root / ".gitignore"
if not gitignore.exists():
return []
try:
with open(gitignore, encoding="utf-8") as f:
return [
line.strip()
for line in f
if line.strip() and not line.startswith("#")
]
except (OSError, UnicodeDecodeError):
return []
def _load_acmsignore_patterns(self, root: Path) -> list[str]:
"""Load exclusion patterns from a .acmsignore file.
Supports glob-style directory patterns similar to .gitignore.
Args:
root: Project root directory path.
Returns:
List of exclusion pattern strings.
"""
acmsignore = root / ".acmsignore"
if not acmsignore.exists():
return []
try:
with open(acmsignore, encoding="utf-8") as f:
return [
line.strip()
for line in f
if line.strip() and not line.startswith("#")
]
except (OSError, UnicodeDecodeError):
return []
def _is_excluded(self, file_path: Path, exclude_patterns: list[str]) -> bool:
"""Check if a file path matches any exclusion pattern.
Checks whether any segment of the file's relative path matches
an exclusion pattern (e.g. '.git' or '__pycache__'). Also checks
full path string match for more specific patterns.
Args:
file_path: Absolute file path to check.
exclude_patterns: List of glob-style patterns to exclude.
Returns:
True if the file should be excluded.
"""
rel_str = str(file_path)
return any(pattern in rel_str for pattern in exclude_patterns)
def _traverse_directory(
self,
root_path: Path,
@@ -562,149 +339,6 @@ class FileTraversalEngine:
# Skip directories we can't read
pass
def _collect_all_files(
self, root: Path, exclude_patterns: list[str] | None = None
) -> list[Path]:
"""Collect all file paths respecting exclusion patterns.
Single-pass scan: traverses the tree once and filters files in-order.
Args:
root: Root directory to traverse.
exclude_patterns: List of patterns to exclude (from .gitignore,
.acmsignore, and caller-supplied).
Returns:
List of Path objects for all non-excluded files.
"""
exclude_patterns = exclude_patterns or []
files: list[Path] = []
for file_path in self._traverse_directory(root):
if self._is_excluded(file_path, exclude_patterns):
self.progress.record_skipped()
continue
files.append(file_path)
return files
def _process_file(self, file_path: Path) -> IndexEntry | None:
"""Process a single file in isolation (thread-safe worker function).
Performs stat + optional binary detection. Returns an IndexEntry
on success or None if the file should be skipped.
This function is designed to run inside ThreadPoolExecutor without
shared state -- all progress tracking uses atomic counters protected
by a lock inside IndexProgress.
Args:
file_path: Path to the file to process.
Returns:
IndexEntry or None if binary/undetectable.
"""
# Binary detection via null-byte heuristic (cheap, reads 8KB only)
try:
stat = file_path.stat()
except OSError:
self.progress.record_error()
return None
# Fast path: zero-byte files are OK (e.g., .gitkeep)
if stat.st_size == 0:
entry = IndexEntry(
path=str(file_path),
file_type=self._get_file_type(file_path),
size_bytes=0,
created_at=datetime.fromtimestamp(stat.st_ctime),
modified_at=datetime.fromtimestamp(stat.st_mtime),
)
self.progress.record_processed()
return entry
# Binary check for non-trivial files
if self._is_binary(file_path):
self.progress.record_binary()
self.progress.record_processed()
return None
try:
entry = IndexEntry(
path=str(file_path),
file_type=self._get_file_type(file_path),
size_bytes=stat.st_size,
created_at=datetime.fromtimestamp(stat.st_ctime),
modified_at=datetime.fromtimestamp(stat.st_mtime),
)
self.progress.record_processed(stat.st_size)
return entry
except (OSError, ValueError):
self.progress.record_error()
return None
def _process_chunk_parallel(self, chunk: list[Path]) -> list[IndexEntry]:
"""Process a chunk of files using ThreadPoolExecutor.
Uses concurrent.futures.ThreadPoolExecutor to distribute file
stat/reading work across multiple worker threads within the same
process -- no subprocess spawning overhead.
Args:
chunk: List of file paths to process in parallel.
Returns:
List of successfully created IndexEntry objects (in order).
"""
entries_by_index: dict[int, IndexEntry] = {}
errors_in_chunk: dict[int, Exception] = {}
with ThreadPoolExecutor(max_workers=self.max_workers) as executor:
# Submit all files in the chunk to the thread pool.
future_to_idx = {
executor.submit(self._process_file, fp): idx
for idx, fp in enumerate(chunk)
}
# Collect results as they complete (preserving order via dict).
for future in as_completed(future_to_idx):
idx = future_to_idx[future]
try:
entry = future.result()
if entry is not None:
entries_by_index[idx] = entry
except Exception as exc:
errors_in_chunk[idx] = exc
# Return entries in original order, then record progress.
ordered_entries: list[IndexEntry] = []
for idx in sorted(entries_by_index):
ordered_entries.append(entries_by_index[idx])
self.progress.record_indexed()
for _exc in errors_in_chunk.values():
self.progress.record_error()
return ordered_entries
def _save_cache(self) -> None:
"""Save the current index to disk using atomic write pattern.
Writes to a temporary file then renames it so that readers never
see a partially-written cache (atomic replace). Only performed
if cache_path is configured.
"""
if self.cache_path is None:
return
try:
data = self.index.to_json_dict()
json_content = json.dumps(data)
# Atomic write: write to temp file then rename.
tmp_path = self.cache_path.with_suffix(".part")
tmp_path.write_text(json_content, encoding="utf-8")
tmp_path.rename(self.cache_path)
except OSError:
# Silently ignore cache write failures -- indexing must not fail.
pass
def traverse_and_index(
self,
root_path: str | Path,
@@ -712,9 +346,8 @@ class FileTraversalEngine:
) -> ACMSIndex:
"""Traverse a directory and index all files.
Uses **parallel chunked processing** to handle large projects
without timeout or memory exhaustion. Files are collected in a
single pass, then processed in parallel batches via ThreadPoolExecutor.
Uses chunked processing to handle large projects without timeout
or memory exhaustion.
Args:
root_path: Root directory to traverse
@@ -728,43 +361,39 @@ class FileTraversalEngine:
if not root.exists():
raise ValueError(f"Path does not exist: {root_path}")
# Load exclusion patterns from .gitignore / .acmsignore files.
gitignore_patterns = self._load_gitignore_patterns(root)
acmsignore_patterns = self._load_acmsignore_patterns(root)
combined_excl: set[str] = set()
for p in exclude_patterns or []:
if p is not None:
combined_excl.add(p)
combined_excl |= set(gitignore_patterns)
combined_excl |= set(acmsignore_patterns)
exclude_patterns = list(combined_excl)
exclude_patterns = exclude_patterns or []
chunk: list[IndexEntry] = []
# Single-pass collection respecting exclusions.
all_files = self._collect_all_files(root, exclude_patterns)
for file_path in self._traverse_directory(root):
# Check if file matches any exclude pattern
if any(pattern in str(file_path) for pattern in exclude_patterns):
continue
if not all_files:
return self.index
# Create index entry
entry = self._create_index_entry(file_path)
if entry:
chunk.append(entry)
# Split into chunks for parallel processing.
chunk_start = 0
while chunk_start < len(all_files):
chunk_end = min(chunk_start + self.chunk_size, len(all_files))
chunk_files = all_files[chunk_start:chunk_end]
# Process chunk when it reaches the size limit
if len(chunk) >= self.chunk_size:
self._process_chunk(chunk)
chunk = []
# Process chunk in parallel using ThreadPoolExecutor.
entries = self._process_chunk_parallel(chunk_files)
# Add entries to index (dict set is atomic from one thread at a time).
for entry in entries:
self.index.add_entry(entry)
chunk_start = chunk_end
# Save cache after full traversal completes.
self._save_cache()
# Process remaining entries
if chunk:
self._process_chunk(chunk)
return self.index
def _process_chunk(self, chunk: list[IndexEntry]) -> None:
"""Process a chunk of index entries.
Args:
chunk: List of IndexEntry objects to add to the index
"""
for entry in chunk:
self.index.add_entry(entry)
def get_index(self) -> ACMSIndex:
"""Get the current index."""
return self.index
@@ -779,6 +408,5 @@ __all__ = [
"FileTraversalEngine",
"FileType",
"IndexEntry",
"IndexProgress",
"TierLevel",
]