Compare commits
7 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 7f47ec85da | |||
| 21728618ca | |||
| 09bc5222a5 | |||
| c3ed77a95c | |||
| 00be1004f1 | |||
| 82502404cd | |||
| 980fcabc48 |
+2
-9
@@ -6,6 +6,8 @@ Changed `wf10_batch.robot` to be less likely to create files, and
|
||||
`plan_generation_graph.robot` to give more test answers.
|
||||
|
||||
## [Unreleased]
|
||||
- **test(e2e): restore complete M2 acceptance test** (#11191): Restored the truncated M2 full actor compiler and LLM integration e2e acceptance test to its complete 10-step form. Added dynamic LLM provider selection via `Resolve LLM Actor` (falls back to Anthropic when OpenAI is unavailable or quota-exhausted), replacing hardcoded `gpt-4` / `openai/gpt-4` references in the actor config and action YAML. Added explicit return-code validation (`Should Be Equal As Integers ${r_actor.rc} 0`) for the actor registration step.
|
||||
- **docs(a2a): ACP to A2A migration guide** (#10230): Added migration guide documenting how to upgrade from the ACP module to the A2A module introduced in v3.6.0, including symbol renames, field renames, operation-name mappings, and YAML configuration updates.
|
||||
- **Plan Prompt JSON Timing Field** (#9353): `agents plan prompt --format json` now
|
||||
includes `timing.started` as an ISO 8601 UTC timestamp in the JSON envelope,
|
||||
matching the spec (§CLI Commands — `agents plan prompt`). Extended
|
||||
@@ -407,15 +409,6 @@ ensuring data is stored with proper parameter values.
|
||||
### Changed
|
||||
|
||||
- Fixed stale `AUTO-BUG-POOL` tracking prefix references in automation-tracking.md documentation and agent-system-specification.md spec document, replaced with correct `AUTO-BUG-SUP` prefix used by the bug-hunt-pool-supervisor agent (#7875).
|
||||
=======
|
||||
- **ACMS index performance optimized via parallel concurrent processing** (#9330): Introduced
|
||||
`ThreadPoolExecutor`-based parallel file hashing in the ACMS indexing pipeline, enabling
|
||||
projects with 10,000+ files to be indexed without timeout. Includes early-stage binary
|
||||
detection (null-byte heuristic), configurable size thresholds, pattern-based exclusion
|
||||
via `.acmsignore`/`.gitignore`, on-disk JSON cache persistence with atomic replacements,
|
||||
and thread-safe progress tracking. The parallel indexer replaces the previous sequential
|
||||
file walk-and-index approach, reducing indexing time from minutes to seconds for large
|
||||
projects while maintaining safety through atomic cache operations and bounded memory usage.
|
||||
|
||||
- **`agents session list` now displays full 26-character session ULIDs** (#10970): The Rich table
|
||||
and Summary panel ("Most Recent" / "Oldest") previously showed only the first 8 characters of
|
||||
|
||||
@@ -81,7 +81,6 @@ Below are some of the specific details of various contributions.
|
||||
* HAL 9000 has contributed the ACMS Context Tier Hydration documentation (PR #9208 / issue #6175): documented the `context_tier_hydrator` module in the ACMS Architecture section of the specification, covering its public interface, file listing strategy, budget limits, and fragment structure.
|
||||
* HAL 9000 has contributed the agent task memory leak fix (#9044): replaced `list.remove` with `set.discard` as the done_callback for asyncio tasks in `Agent._tasks`, preventing unbounded memory growth in long-lived agents and ensuring safe concurrent task removal.
|
||||
* HAL 9000 has contributed the ACMS context show/clear CLI commands (PR #9675 / issue #9586): implemented `context show <view>` displaying assembled context with per-tier budget utilization summary (hot/warm/cold), and `context clear` with --path, --tag, --tier filtering plus confirmation prompt with --yes bypass. Includes 12 Behave BDD scenarios, 9 Robot Framework integration tests, ASV benchmarks, full type annotations, and _TierServiceProtocol for type safety.
|
||||
* HAL 9000 has contributed the ACMS parallel processing optimization (PR #9981 / issue #9330): introduced `ThreadPoolExecutor`-based concurrent file hashing to the ACMS indexing pipeline, enabling projects with 10,000+ files to be indexed without timeout. Includes binary detection via null-byte heuristic, configurable size thresholds, `.acmsignore`/`.gitignore` pattern exclusion, on-disk JSON cache persistence with atomic replacements, and thread-safe progress tracking.
|
||||
|
||||
# Details (PR Contributions)
|
||||
|
||||
|
||||
@@ -134,3 +134,38 @@ config = ServerConnectionConfig(url="https://my-server.example.com", token="..."
|
||||
```
|
||||
|
||||
Validates URL format and token presence.
|
||||
|
||||
---
|
||||
|
||||
## ACP to A2A Rename (v3.6.0)
|
||||
|
||||
> **Breaking change in v3.6.0.** The `cleveragents.acp` module was renamed to
|
||||
> `cleveragents.a2a`. All `Acp*` class names were renamed to `A2a*`. The
|
||||
> `AcpRequest` and `AcpResponse` field names were updated to comply with the
|
||||
> JSON-RPC 2.0 specification.
|
||||
|
||||
For the complete migration guide, see
|
||||
[`docs/development/acp-to-a2a-migration.md`](../development/acp-to-a2a-migration.md).
|
||||
|
||||
**Symbol rename summary:**
|
||||
|
||||
| Old (ACP) | New (A2A) |
|
||||
|---|---|
|
||||
| `AcpRequest` | `A2aRequest` |
|
||||
| `AcpResponse` | `A2aResponse` |
|
||||
| `AcpLocalFacade` | `A2aLocalFacade` |
|
||||
| `AcpError` | `A2aError` |
|
||||
| `AcpEventQueue` | `A2aEventQueue` |
|
||||
|
||||
**Field rename summary (`A2aRequest` / `A2aResponse`):**
|
||||
|
||||
| Old Field | New Field |
|
||||
|---|---|
|
||||
| `a2a_version` | `jsonrpc` |
|
||||
| `request_id` | `id` |
|
||||
| `operation` | `method` |
|
||||
| `payload` | `params` |
|
||||
| `data` | `result` |
|
||||
| `error_detail` | `error` |
|
||||
|
||||
See [ADR-047](../adr/ADR-047-acp-standard-adoption.md) for architectural rationale.
|
||||
|
||||
@@ -0,0 +1,264 @@
|
||||
# ACP to A2A Module Rename and Symbol Standardization
|
||||
|
||||
> **Introduced in:** v3.6.0
|
||||
> **Architectural decision:** [ADR-047 - A2A Standard Adoption](../adr/ADR-047-acp-standard-adoption.md)
|
||||
> **Module path:** `cleveragents.a2a` (commit `449c33b7`)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
In v3.6.0 the internal **ACP (Agent Client Protocol)** module was renamed to
|
||||
**A2A (Agent-to-Agent Protocol)** and all public symbols were standardized to
|
||||
align with the external [A2A open standard](https://a2a-protocol.org) (Linux
|
||||
Foundation, Apache 2.0).
|
||||
|
||||
This was not merely a rename - it was an architectural shift from a bespoke
|
||||
JSON envelope protocol to the industry-standard A2A wire format built on
|
||||
**JSON-RPC 2.0**. The change eliminates the proprietary ACP envelope, replaces
|
||||
custom REST routing with JSON-RPC 2.0 method dispatch, and introduces
|
||||
**Agent Card** discovery (`/.well-known/agent.json`).
|
||||
|
||||
---
|
||||
|
||||
## What Changed
|
||||
|
||||
### 1. Module Path
|
||||
|
||||
| Before (ACP) | After (A2A) |
|
||||
|---|---|
|
||||
| `cleveragents.acp` | `cleveragents.a2a` |
|
||||
| `from cleveragents.acp import ...` | `from cleveragents.a2a import ...` |
|
||||
|
||||
### 2. Class and Symbol Names
|
||||
|
||||
All public symbols were renamed from `Acp*` to `A2a*`:
|
||||
|
||||
| Old Symbol (ACP) | New Symbol (A2A) | Notes |
|
||||
|---|---|---|
|
||||
| `AcpRequest` | `A2aRequest` | Wire format changed - see section 3 |
|
||||
| `AcpResponse` | `A2aResponse` | Wire format changed - see section 3 |
|
||||
| `AcpError` | `A2aError` | Base exception |
|
||||
| `AcpNotAvailableError` | `A2aNotAvailableError` | Server-mode guard |
|
||||
| `AcpOperationNotFoundError` | `A2aOperationNotFoundError` | Unknown method |
|
||||
| `AcpVersionMismatchError` | `A2aVersionMismatchError` | Protocol version guard |
|
||||
| `AcpLocalFacade` | `A2aLocalFacade` | Local-mode dispatcher |
|
||||
| `AcpHttpTransport` | `A2aHttpTransport` | Server-mode transport stub |
|
||||
| `AcpEventQueue` | `A2aEventQueue` | In-process event delivery |
|
||||
| `AcpVersionNegotiator` | `A2aVersionNegotiator` | Version negotiation |
|
||||
| `AcpVersion` | `A2aVersion` | Version constants |
|
||||
| `AcpEvent` | `A2aEvent` | SSE event envelope |
|
||||
| `AcpErrorDetail` | `A2aErrorDetail` | JSON-RPC error object |
|
||||
|
||||
### 3. Request and Response Field Names
|
||||
|
||||
`A2aRequest` and `A2aResponse` now use **JSON-RPC 2.0** field names instead of
|
||||
the proprietary ACP envelope fields. These models are defined in
|
||||
`cleveragents.a2a.models` (commit `449c33b7`):
|
||||
|
||||
| Old Field (ACP) | New Field (A2A / JSON-RPC 2.0) | Type |
|
||||
|---|---|---|
|
||||
| `a2a_version` | `jsonrpc` (always `"2.0"`) | `str` |
|
||||
| `request_id` | `id` | `str` |
|
||||
| `operation` | `method` | `str` |
|
||||
| `payload` | `params` | `dict[str, Any]` |
|
||||
| `data` (success path) | `result` | `dict[str, Any] or None` |
|
||||
| `error_detail` | `error` | `A2aErrorDetail or None` |
|
||||
|
||||
**Before (ACP):**
|
||||
```python
|
||||
from cleveragents.acp import AcpRequest, AcpLocalFacade
|
||||
|
||||
request = AcpRequest(
|
||||
a2a_version="1.0",
|
||||
request_id="req-001",
|
||||
operation="plan.status",
|
||||
payload={"plan_id": "01HXRCF1..."},
|
||||
)
|
||||
response = facade.dispatch(request)
|
||||
if response.data:
|
||||
print(response.data["phase"])
|
||||
```
|
||||
|
||||
**After (A2A):**
|
||||
```python
|
||||
from cleveragents.a2a import A2aRequest, A2aLocalFacade
|
||||
|
||||
request = A2aRequest(
|
||||
jsonrpc="2.0",
|
||||
id="req-001",
|
||||
method="_cleveragents/plan/status",
|
||||
params={"plan_id": "01HXRCF1..."},
|
||||
)
|
||||
response = facade.dispatch(request)
|
||||
if response.result:
|
||||
print(response.result["phase"])
|
||||
```
|
||||
|
||||
### 4. Operation Names
|
||||
|
||||
The ACP proprietary operation names were replaced with the A2A standard
|
||||
`_cleveragents/` extension method namespace. The legacy names are still
|
||||
accepted by `A2aLocalFacade.dispatch()` in `cleveragents.a2a.facade`
|
||||
(commit `449c33b7`) for backward compatibility but are **deprecated** and
|
||||
will be removed in a future major version.
|
||||
|
||||
| Old Operation (ACP / deprecated) | New Operation (A2A) |
|
||||
|---|---|
|
||||
| `session.create` | `_cleveragents/session/create` (via `message/send`) |
|
||||
| `session.close` | `_cleveragents/session/close` |
|
||||
| `plan.create` | `_cleveragents/plan/use` |
|
||||
| `plan.execute` | `_cleveragents/plan/execute` |
|
||||
| `plan.status` | `_cleveragents/plan/status` |
|
||||
| `plan.diff` | `_cleveragents/plan/diff` |
|
||||
| `plan.apply` | `_cleveragents/plan/apply` |
|
||||
| `registry.list_tools` | `_cleveragents/registry/tool/list` |
|
||||
| `registry.list_resources` | `_cleveragents/registry/resource/list` |
|
||||
| `context.get` | `_cleveragents/context/show` |
|
||||
| `event.subscribe` | (event delivery via SSE `message/stream`) |
|
||||
|
||||
The full list of supported A2A extension methods is documented in
|
||||
[`docs/reference/a2a.md`](../reference/a2a.md#extension-methods).
|
||||
|
||||
### 5. YAML and Configuration Files
|
||||
|
||||
If you have YAML configuration files, actor definitions, or skill files that
|
||||
reference the old ACP module path or operation names, update them as follows:
|
||||
|
||||
**Actor YAML - protocol binding:**
|
||||
```yaml
|
||||
# Before (ACP)
|
||||
protocol:
|
||||
type: acp
|
||||
version: "1.0"
|
||||
|
||||
# After (A2A)
|
||||
protocol:
|
||||
type: a2a
|
||||
version: "2.0"
|
||||
```
|
||||
|
||||
**Skill YAML - operation references:**
|
||||
```yaml
|
||||
# Before (ACP)
|
||||
operations:
|
||||
- operation: plan.create
|
||||
- operation: plan.execute
|
||||
|
||||
# After (A2A)
|
||||
operations:
|
||||
- method: _cleveragents/plan/use
|
||||
- method: _cleveragents/plan/execute
|
||||
```
|
||||
|
||||
**Server connection config:**
|
||||
```yaml
|
||||
# Before (ACP)
|
||||
server:
|
||||
protocol: acp
|
||||
endpoint: https://my-server.example.com/acp
|
||||
|
||||
# After (A2A)
|
||||
server:
|
||||
protocol: a2a
|
||||
endpoint: https://my-server.example.com/a2a
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Migration Steps
|
||||
|
||||
Follow these steps when upgrading from a codebase that used the ACP module:
|
||||
|
||||
### Step 1 - Update imports
|
||||
|
||||
Replace all `cleveragents.acp` imports with `cleveragents.a2a`:
|
||||
|
||||
```bash
|
||||
# Find all files that import from the old ACP module
|
||||
grep -r "from cleveragents.acp" src/ tests/
|
||||
grep -r "import cleveragents.acp" src/ tests/
|
||||
```
|
||||
|
||||
Then update each import:
|
||||
```python
|
||||
# Before
|
||||
from cleveragents.acp import AcpLocalFacade, AcpRequest, AcpResponse
|
||||
|
||||
# After
|
||||
from cleveragents.a2a import A2aLocalFacade, A2aRequest, A2aResponse
|
||||
```
|
||||
|
||||
### Step 2 - Rename symbol references
|
||||
|
||||
Use your editor's find-and-replace to rename all `Acp*` symbols to `A2a*`
|
||||
(see the complete symbol table in section 2 above).
|
||||
|
||||
### Step 3 - Update request field names
|
||||
|
||||
Update all `AcpRequest` and `AcpResponse` constructor calls and field accesses
|
||||
to use the new JSON-RPC 2.0 field names (see section 3 above).
|
||||
|
||||
### Step 4 - Update operation names
|
||||
|
||||
Replace deprecated ACP operation strings with the new `_cleveragents/`
|
||||
extension method names (see section 4 above). The legacy names still work but
|
||||
emit deprecation warnings.
|
||||
|
||||
### Step 5 - Update YAML files
|
||||
|
||||
Search for any YAML files that reference `acp` in protocol type fields or
|
||||
operation names and update them (see section 5 above).
|
||||
|
||||
### Step 6 - Verify
|
||||
|
||||
Run the full test suite to confirm no regressions:
|
||||
|
||||
```bash
|
||||
nox -s unit_tests
|
||||
nox -s integration_tests
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Backward Compatibility
|
||||
|
||||
The `A2aLocalFacade.dispatch()` method in `cleveragents.a2a.facade.A2aLocalFacade`
|
||||
(commit `449c33b7`) continues to accept the legacy ACP operation names
|
||||
(`session.create`, `plan.create`, etc.) via the `_LEGACY_OPERATIONS` list.
|
||||
This backward compatibility shim will be removed in the next major version.
|
||||
|
||||
The `A2aVersion` class in `cleveragents.a2a.models.A2aVersion` (commit
|
||||
`449c33b7`) retains the `CURRENT` and `SUPPORTED` constants for backward
|
||||
compatibility, though the wire format now uses the `jsonrpc` field rather than
|
||||
a proprietary `a2a_version` field.
|
||||
|
||||
---
|
||||
|
||||
## Architectural Rationale
|
||||
|
||||
The rename was driven by the adoption of the external A2A open standard
|
||||
([a2a-protocol.org](https://a2a-protocol.org)), which supersedes the bespoke
|
||||
ACP protocol. Key reasons for the change:
|
||||
|
||||
- **Ecosystem alignment**: A2A is governed by the Linux Foundation with 22k+
|
||||
GitHub stars and SDKs in Python, Go, JavaScript, Java, and .NET.
|
||||
- **Agent Card discovery**: The standard introduces `/.well-known/agent.json`
|
||||
for dynamic capability advertisement.
|
||||
- **JSON-RPC 2.0**: Replaces the proprietary envelope with a mature, widely-
|
||||
tooled standard.
|
||||
- **Extensibility**: The `_cleveragents/` extension namespace cleanly separates
|
||||
platform-specific operations from standard agent interactions.
|
||||
|
||||
See [ADR-047](../adr/ADR-047-acp-standard-adoption.md) for the full
|
||||
architectural rationale and decision record.
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [A2A Reference](../reference/a2a.md) - complete API reference for the `cleveragents.a2a` module
|
||||
- [A2A API](../api/a2a.md) - class-level API documentation
|
||||
- [ADR-026 - Agent-to-Agent Protocol](../adr/ADR-026-agent-client-protocol.md) - protocol boundary design
|
||||
- [ADR-047 - A2A Standard Adoption](../adr/ADR-047-acp-standard-adoption.md) - rationale for adopting the external standard
|
||||
- [ADR-048 - Server Application Architecture](../adr/ADR-048-server-application-architecture.md) - server implementation using A2A
|
||||
@@ -0,0 +1,34 @@
|
||||
# Milestones
|
||||
|
||||
This section documents the major milestones in the CleverAgents v3 release series.
|
||||
Each milestone page covers the features delivered, CLI commands introduced, and
|
||||
architectural decisions made during that release cycle.
|
||||
|
||||
## Release Overview
|
||||
|
||||
| Milestone | Version | Title | Status |
|
||||
|-----------|---------|-------|--------|
|
||||
| M1 | [v3.0.0](v3.0.0.md) | Minimal Local Source-Code Workflow | Released |
|
||||
| M2 | [v3.1.0](v3.1.0.md) | Actor Compiler + Full LLM Integration | Released |
|
||||
|
||||
## What Is a Milestone?
|
||||
|
||||
CleverAgents uses **Clever Semantic Versioning** (MAJOR.MINOR.PATCH) where each
|
||||
MINOR version bump corresponds to a planned milestone delivering a coherent set of
|
||||
features. Milestones are tracked as Forgejo milestones and linked to epics
|
||||
in the issue tracker.
|
||||
|
||||
Each milestone page documents:
|
||||
|
||||
- **Goals** — What the milestone set out to achieve
|
||||
- **Delivered Features** — Concrete capabilities added
|
||||
- **CLI Reference** — Commands introduced or changed
|
||||
- **Architecture Notes** — Key design decisions
|
||||
- **Migration Guide** — Breaking changes and upgrade steps (if any)
|
||||
|
||||
## Related Resources
|
||||
|
||||
- [Implementation Timeline](../timeline.md) — Full chronological timeline
|
||||
- [Architecture](../architecture.md) — System architecture overview
|
||||
- [API Reference](../api/index.md) — Module-level API documentation
|
||||
- [CHANGELOG](https://git.cleverthis.com/cleveragents/cleveragents-core/src/branch/master/CHANGELOG.md) — Detailed change log
|
||||
@@ -0,0 +1,142 @@
|
||||
# v3.0.0 — Action, Resource, Project & Plan CLI Reference
|
||||
|
||||
Complete command documentation for all CleverAgents v3.0.0 CLI commands.
|
||||
|
||||
---
|
||||
|
||||
## Action Management (`agents action`)
|
||||
|
||||
Actions are reusable plan templates defined in YAML. They specify which actors
|
||||
handle each phase of the plan lifecycle and what arguments the plan accepts.
|
||||
|
||||
### `agents action create`
|
||||
|
||||
```bash
|
||||
agents action create --config ./my-action.yaml
|
||||
```
|
||||
|
||||
**Minimal action YAML:**
|
||||
|
||||
```yaml
|
||||
name: local/code-coverage
|
||||
description: Increase code coverage to the target percentage
|
||||
strategy_actor: openai/gpt-4
|
||||
execution_actor: openai/gpt-4
|
||||
definition_of_done: |
|
||||
Line coverage reaches the target percentage for all modules under src/.
|
||||
arguments:
|
||||
- name: target_coverage
|
||||
type: integer
|
||||
required: true
|
||||
```
|
||||
|
||||
**Full action YAML:**
|
||||
|
||||
```yaml
|
||||
name: local/refactor
|
||||
description: Refactor a module to follow SOLID principles
|
||||
strategy_actor: openai/gpt-4
|
||||
execution_actor: anthropic/claude-3
|
||||
estimation_actor: openai/gpt-4
|
||||
invariant_actor: anthropic/claude-3
|
||||
automation_profile: trusted
|
||||
definition_of_done: |
|
||||
All classes follow Single Responsibility Principle.
|
||||
No method exceeds 20 lines.
|
||||
invariants:
|
||||
- "No reduction in test coverage"
|
||||
- "All public APIs remain backward compatible"
|
||||
arguments:
|
||||
- name: module_path
|
||||
type: string
|
||||
required: true
|
||||
```
|
||||
|
||||
### `agents action list` / `show` / `archive`
|
||||
|
||||
```bash
|
||||
agents action list # all actions
|
||||
agents action list --namespace local # filter by namespace
|
||||
agents action list --state available # filter by state
|
||||
agents action show local/code-coverage # show action details
|
||||
agents action archive local/old-action # soft-delete action
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Resource Management (`agents resource`)
|
||||
|
||||
Resources are external entities (git repos, directories, mounts) that plans
|
||||
can read or modify.
|
||||
|
||||
### `agents resource add`
|
||||
|
||||
```bash
|
||||
agents resource add git-checkout local/my-repo --path /home/user/projects/my-repo
|
||||
agents resource add fs-directory local/data --path /data --read-only
|
||||
```
|
||||
|
||||
**Built-in resource types (v3.0.0):** `git-checkout`, `fs-directory`, `fs-mount`
|
||||
|
||||
### `agents resource list` / `show` / `remove`
|
||||
|
||||
```bash
|
||||
agents resource list # all resources
|
||||
agents resource show local/my-repo # resource details
|
||||
agents resource remove --yes local/my-repo # remove resource
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Project Management (`agents project`)
|
||||
|
||||
Projects group resources together for plan execution context.
|
||||
|
||||
### `agents project create` / `link-resource` / `list` / `show`
|
||||
|
||||
```bash
|
||||
agents project create local/my-project --description "My application"
|
||||
agents project link-resource local/my-project local/my-repo
|
||||
agents project list
|
||||
agents project show local/my-project
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Plan Lifecycle (`agents plan`)
|
||||
|
||||
Plans are the central execution unit. Lifecycle: **Strategize -> Execute -> Apply**.
|
||||
|
||||
### `agents plan use` — Create a plan
|
||||
|
||||
```bash
|
||||
agents plan use local/code-coverage my-project --arg target_coverage=80
|
||||
agents plan use local/lint proj-1 proj-2 # multiple projects
|
||||
agents plan use local/refactor my-project \ # with automation profile & invariants
|
||||
--automation-profile trusted --invariant "No new warnings"
|
||||
agents plan use local/code-coverage my-project \ # with actor overrides
|
||||
--strategy-actor openai/gpt-4 --execution-actor anthropic/claude-3
|
||||
```
|
||||
|
||||
### `agents plan execute` / `diff` / `apply` — Run and merge changes
|
||||
|
||||
```bash
|
||||
agents plan execute # Strategize -> Execute phases
|
||||
agents plan diff 01HXYZ... # preview proposed changes
|
||||
agents plan apply --yes # merge changes (skip confirmation)
|
||||
```
|
||||
|
||||
### `agents plan list` / `status` / `cancel` — Manage plans
|
||||
|
||||
```bash
|
||||
agents plan list --state complete # filter plans
|
||||
agents plan status 01HXYZ... # plan details
|
||||
agents plan cancel 01HXYZ... --reason "Requirements changed"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [CLI overview (v3.0.0)](index.md) — Goals, architecture notes, feature table
|
||||
- [Deep Dive](deep-dive.md) — Sandbox, persistence, domain model internals
|
||||
@@ -0,0 +1,95 @@
|
||||
# v3.0.0 — Deep Dive: Sandbox, Persistence & Domain Model
|
||||
|
||||
Internal architecture details for CleverAgents v3.0.0.
|
||||
|
||||
---
|
||||
|
||||
## Git Worktree Sandbox
|
||||
|
||||
The git worktree sandbox isolates LLM-generated changes in a dedicated branch
|
||||
and worktree until the user approves them via `agents plan apply`.
|
||||
|
||||
See the full [Git Worktree Sandbox](../../modules/git-worktree-sandbox.md) module docs.
|
||||
|
||||
### How It Works
|
||||
|
||||
1. Plan execute creates branch `cleveragents/plan-<id>` from current HEAD.
|
||||
2. A temporary git worktree is created in a system temp directory.
|
||||
3. The LLM writes changes only inside the worktree (not the real repo).
|
||||
4. `plan diff` shows proposed changes as unified diff.
|
||||
5. `plan apply` merges sandbox into original via `git merge`, then cleans up.
|
||||
6. `plan cancel` discards the worktree — no real repo changes.
|
||||
|
||||
Non-git resources (e.g. `fs-directory`) fall back to `shutil.copy2`.
|
||||
|
||||
### Apply Summary Output
|
||||
|
||||
```
|
||||
╭─ Apply Summary ────────────────────────────────────────╮
|
||||
│ Plan ID: plan-01HZ... │
|
||||
│ Artifacts: 3 files changed │
|
||||
│ Insertions: +142 │
|
||||
│ Deletions: -17 │
|
||||
╰───────────────────────────────────────────────────────╯
|
||||
╭─ Sandbox Cleanup ──────────────────────────────────────╮
|
||||
│ [x] Worktree removed │
|
||||
│ [x] Branch deleted │
|
||||
╰───────────────────────────────────────────────────────╯
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## SQLite Persistence with Alembic
|
||||
|
||||
All entities persist in `~/.cleveragents/cleveragents.db`.
|
||||
|
||||
### Managed Tables
|
||||
|
||||
| Table | Description |
|
||||
|-------|-------------|
|
||||
| `actions` | Action templates |
|
||||
| `action_arguments`, `action_invariants` | Per-action metadata |
|
||||
| `resources` | Resource instances |
|
||||
| `projects`, `project_resource_links` | Projects and links |
|
||||
| `plans`, `plan_project_links` | Plans and project bindings |
|
||||
| `decisions` | Decision tree nodes per plan |
|
||||
|
||||
### Migration Commands
|
||||
|
||||
```bash
|
||||
alembic upgrade head # apply pending migrations
|
||||
alembic current # show current revision
|
||||
alembic history # migration history
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Domain Model (Pydantic v2)
|
||||
|
||||
All domain entities use Pydantic v2 with `frozen=True` for immutability.
|
||||
|
||||
```python
|
||||
from cleveragents.domain.models import Action, Resource, Project, Plan
|
||||
|
||||
action = Action(
|
||||
name="local/code-coverage",
|
||||
description="Increase code coverage",
|
||||
strategy_actor="openai/gpt-4",
|
||||
execution_actor="openai/gpt-4",
|
||||
definition_of_done="Coverage reaches target.",
|
||||
)
|
||||
# action.name = "other" # raises ValidationError (frozen)
|
||||
```
|
||||
|
||||
ULIDs (Universally Unique Lexicographically Sortable Identifiers) are the primary keys.
|
||||
|
||||
See [ADR-019](../../adr/ADR-019-storage-and-persistence.md) for storage details.
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [CLI Reference](cli-reference.md) — All command documentation
|
||||
- [Git Worktree Sandbox](../../modules/git-worktree-sandbox.md)
|
||||
- [ADR-006](../../adr/ADR-006-plan-lifecycle.md): Plan Lifecycle
|
||||
- [ADR-019](../../adr/ADR-019-storage-and-persistence.md): Storage & Persistence
|
||||
@@ -0,0 +1,87 @@
|
||||
# v3.0.0 — Minimal Local Source-Code Workflow (M1) — Overview
|
||||
|
||||
**Released:** v3.0.0
|
||||
**Milestone:** M1 — Minimal Local Source-Code Workflow
|
||||
**Theme:** Establish the foundational CLI, domain model, and persistence layer for local agent-driven code workflows.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
v3.0.0 delivers the first complete end-to-end workflow for using CleverAgents to
|
||||
manage source-code changes on a local machine. It introduces the core CLI command
|
||||
groups (`agents action`, `agents resource`, `agents project`, `agents plan`),
|
||||
a SQLite-backed persistence layer with Alembic migrations, and a git worktree
|
||||
sandbox that isolates LLM-generated changes until the user approves them.
|
||||
|
||||
This milestone establishes the **specification-first** development model: every
|
||||
feature is driven by a YAML-defined action template, executed through a structured
|
||||
plan lifecycle (Strategize -> Execute -> Apply), and persisted in a local SQLite
|
||||
database.
|
||||
|
||||
---
|
||||
|
||||
## Goals
|
||||
|
||||
1. Provide a minimal but complete CLI for local source-code automation.
|
||||
2. Introduce a typed, versioned domain model using Pydantic v2.
|
||||
3. Persist all entities (actions, resources, projects, plans) in SQLite via Alembic.
|
||||
4. Isolate LLM-generated changes in a git worktree sandbox before applying them.
|
||||
5. Enable plan lifecycle management: create, execute, diff, and apply plans.
|
||||
|
||||
---
|
||||
|
||||
## Delivered Features
|
||||
|
||||
v3.0.0 delivers the following feature groups:
|
||||
|
||||
| Feature Group | CLI Commands | Description | Page |
|
||||
|---------------|-------------|-------------|------|
|
||||
| **Action Mgmt** | `agents action` | YAML plan templates | [CLI Reference](cli-reference.md) |
|
||||
| **Resource Mgmt** | `agents resource` | Git repos, filesystems | [CLI Reference](cli-reference.md) |
|
||||
| **Project Mgmt** | `agents project` | Group resources for plans | [CLI Reference](cli-reference.md) |
|
||||
| **Plan Lifecycle** | `agents plan` | Strategy-Execute-Apply cycle | [CLI Reference](cli-reference.md) |
|
||||
| **Git Worktree Sandbox** | (internal) | Isolated LLM change sandbox | [Deep Dive](deep-dive.md) |
|
||||
| **SQLite Persistence** | (internal) | SQLite + Alembic with 9 tables | [Deep Dive](deep-dive.md) |
|
||||
| **Pydantic v2 Domain Model** | (internal) | Frozen=True, ULID keys | [Deep Dive](deep-dive.md) |
|
||||
|
||||
---
|
||||
|
||||
## Architecture Notes
|
||||
|
||||
### Layered Architecture
|
||||
|
||||
v3.0.0 follows a strict layered architecture (see [ADR-001](../../adr/ADR-001-layered-architecture.md)):
|
||||
|
||||
```
|
||||
CLI Layer agents action/resource/project/plan
|
||||
|
|
||||
Application Layer PlanLifecycleService, ActionService, ResourceService
|
||||
|
|
||||
Domain Layer Action, Resource, Project, Plan (Pydantic v2, frozen=True)
|
||||
|
|
||||
Infrastructure SQLite + Alembic, GitWorktreeSandbox, ToolRegistry
|
||||
```
|
||||
|
||||
### Dependency Injection
|
||||
|
||||
All services are wired via a DI container (see [ADR-003](../../adr/ADR-003-dependency-injection.md)).
|
||||
|
||||
### Plan Lifecycle State Machine
|
||||
|
||||
Plans follow a strict state machine (see [ADR-006](../../adr/ADR-006-plan-lifecycle.md)):
|
||||
|
||||
```
|
||||
CREATED -> STRATEGIZE/queued -> STRATEGIZE/complete
|
||||
-> EXECUTE/queued -> EXECUTE/complete
|
||||
-> APPLY/queued -> APPLY/complete -> CANCELLED / ERRORED (terminal)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [CLI Reference](cli-reference.md) — Full command-by-command documentation
|
||||
- [Deep Dive](deep-dive.md) — Sandbox, persistence, domain model internals
|
||||
- [ADR-001](../../adr/ADR-001-layered-architecture.md): Layered Architecture
|
||||
- [ADR-006](../../adr/ADR-006-plan-lifecycle.md): Plan Lifecycle
|
||||
@@ -0,0 +1,131 @@
|
||||
# v3.1.0 — Actor YAML Format & Compiler
|
||||
|
||||
Declarative actor definitions compiled into LangGraph graphs.
|
||||
|
||||
---
|
||||
|
||||
## Actor YAML Format
|
||||
|
||||
Every actor file must specify `version: "3"` and one of three types: `llm`, `tool`, or `graph`.
|
||||
|
||||
See [Actor YAML Schema Reference](../../reference/actors_schema.md) for full fields.
|
||||
|
||||
### LLM Actor
|
||||
|
||||
```yaml
|
||||
version: "3"
|
||||
name: assistants/code-reviewer
|
||||
type: llm
|
||||
model: gpt-4
|
||||
system_prompt: |
|
||||
You are an expert Python code reviewer.
|
||||
context_view: reviewer
|
||||
memory:
|
||||
enabled: true
|
||||
max_messages: 20
|
||||
```
|
||||
|
||||
### Tool Actor
|
||||
|
||||
```yaml
|
||||
version: "3"
|
||||
name: utilities/file-ops
|
||||
type: tool
|
||||
tools:
|
||||
- files/read_file
|
||||
- files/write_file
|
||||
- files/list_directory
|
||||
```
|
||||
|
||||
### Graph Actor
|
||||
|
||||
```yaml
|
||||
version: "3"
|
||||
name: workflows/tdd-cycle
|
||||
type: graph
|
||||
model: gpt-4
|
||||
route:
|
||||
nodes:
|
||||
- id: planner; type: agent; name: Task Planner
|
||||
config: { model: gpt-4, prompt: "Plan implementation tasks." }
|
||||
- id: implementer; type: agent; name: Code Implementer
|
||||
config: { model: gpt-4, prompt: "Implement planned tasks." }
|
||||
- id: verifier; type: tool; name: Test Runner
|
||||
config: { tool_name: testing/run_pytest }
|
||||
- id: check_results; type: conditional
|
||||
config: { conditions: [{check: "passed==True", route_to: done}, {check: "passed==False", route_to: implementer}] }
|
||||
- id: done; type: agent; name: Summary Writer
|
||||
edges:
|
||||
- {from_node: planner, to_node: implementer}
|
||||
- {from_node: implementer, to_node: verifier}
|
||||
- {from_node: verifier, to_node: check_results}
|
||||
entry_node: planner
|
||||
exit_nodes: [done]
|
||||
```
|
||||
|
||||
### Actor Name Format
|
||||
|
||||
Must use `namespace/name`:
|
||||
- `assistants/code-reviewer` — valid
|
||||
- `code-reviewer` — invalid (no namespace)
|
||||
- `a/b/c` — invalid (too many slashes)
|
||||
|
||||
---
|
||||
|
||||
## Actor Compiler (LangGraph Integration)
|
||||
|
||||
Translates GRAPH-type YAML into LangGraph `StateGraph` structures.
|
||||
|
||||
See [Actor Compiler Reference](../../reference/actor_compiler.md).
|
||||
|
||||
### Compilation Pipeline
|
||||
|
||||
```
|
||||
ActorConfigSchema (type=GRAPH)
|
||||
-> validate type, route
|
||||
-> reference validation (all node IDs exist)
|
||||
-> intra-graph cycle detection
|
||||
-> cross-actor subgraph cycle detection
|
||||
-> node mapping (NodeDef -> LangGraph NodeConfig)
|
||||
-> edge mapping
|
||||
-> metadata assembly
|
||||
=> CompiledActor (nodes, edges, entry_point, metadata)
|
||||
```
|
||||
|
||||
### Node Type Mapping
|
||||
|
||||
| Actor Node | LangGraph Type |
|
||||
|------------|----------------|
|
||||
| `agent` | AGENT |
|
||||
| `tool` | TOOL |
|
||||
| `conditional` | CONDITIONAL |
|
||||
| `subgraph` | SUBGRAPH |
|
||||
|
||||
### Compilation Errors
|
||||
|
||||
| Error | Class | When |
|
||||
|-------|-------|------|
|
||||
| Non-GRAPH type | `ActorCompilationError` | config.type != GRAPH |
|
||||
| Missing route | `ActorCompilationError` | config.route is None |
|
||||
| Missing node | `MissingNodeError` | edge refs unknown node |
|
||||
| Invalid entry/exit | `InvalidEntryExitError` | node not in graph |
|
||||
| Cycle detected | `SubgraphCycleError` | intra or cross-actor cycle |
|
||||
|
||||
### Graph Validation Rules
|
||||
|
||||
1. Unique node IDs within graph
|
||||
2. Entry node exists and is valid
|
||||
3. All exit_nodes reference valid nodes
|
||||
4. All edge from_node/to_node exist
|
||||
5. No cycles (graphs are acyclic)
|
||||
6. All nodes reachable from entry_node
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [Actor YAML Schema Reference](../../reference/actors_schema.md)
|
||||
- [Actor Compiler Reference](../../reference/actor_compiler.md)
|
||||
- [ADR-010](../../adr/ADR-010-actor-and-agent-architecture.md): Actor Architecture
|
||||
- [ADR-022](../../adr/ADR-022-langchain-langgraph-integration.md): LangGraph Integration
|
||||
- [ADR-031](../../adr/ADR-031-actor-abstraction-definition.md): Actor Abstraction Definition
|
||||
@@ -0,0 +1,69 @@
|
||||
# v3.1.0 — Actor Compiler + Full LLM Integration (M2) — Overview
|
||||
|
||||
**Released:** v3.1.0
|
||||
**Milestone:** M2 — Actor Compiler + Full LLM Integration
|
||||
**Theme:** Introduce the Actor system, compile YAML-defined actors into executable LangGraph graphs, and wire full LLM integration through MCP adapter, tool router, skill registry, and validation runner.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
v3.1.0 builds on the v3.0.0 foundation by introducing the **Actor system** —
|
||||
declarative YAML definitions for AI agents compiled into LangGraph `StateGraph`
|
||||
structures at runtime. It also delivers MCP adapter (external tool server
|
||||
connectivity), tool router (provider normalization), skill registry (reusable
|
||||
tool collections), and validation runner (pre-apply quality gates).
|
||||
|
||||
---
|
||||
|
||||
## Goals
|
||||
|
||||
1. Define a declarative YAML format for actors (`llm`, `tool`, `graph`).
|
||||
2. Compile GRAPH-type actors into executable LangGraph `StateGraph` structures.
|
||||
3. Connect to external MCP tool servers for tool discovery and invocation.
|
||||
4. Route LLM tool calls across OpenAI, Anthropic, and LangChain formats.
|
||||
5. Manage reusable tool collections via the skill registry.
|
||||
6. Enforce validation gates before plan apply via the validation runner.
|
||||
|
||||
---
|
||||
|
||||
## Feature Index
|
||||
|
||||
| Feature | Description | Documentation |
|
||||
|---------|-------------|---------------|
|
||||
| **Actor YAML** | Declarative defs: llm, tool, graph types | [Actor Format](actor-yaml.md) |
|
||||
| **Actor Compiler** | Compile to LangGraph StateGraph | [Actor Format](actor-yaml.md) |
|
||||
| **MCP Adapter** | External MCP tool server connectivity | [Integration](integration.md) |
|
||||
| **Tool Router** | Normalize across OpenAI/Anthropic/LC formats | [Integration](integration.md) |
|
||||
| **Skill Registry** | Persistent, composable tool collections | [Skills](skills.md) |
|
||||
| **Validation Runner** | Pre-apply required/informational gates | [Skills](skills.md) |
|
||||
|
||||
---
|
||||
|
||||
## Architecture Notes
|
||||
|
||||
### Actor System Design
|
||||
|
||||
Follows the **Actor Abstraction Definition** (see [ADR-031](../../adr/ADR-031-actor-abstraction-definition.md)):
|
||||
- **Declarative** — defined in YAML, not code.
|
||||
- **Composable** — GRAPH actors embed other actors as subgraphs.
|
||||
- **Compiled** — validated and compiled to LangGraph at load time.
|
||||
- **Namespaced** — `namespace/name` format required.
|
||||
|
||||
### LangGraph Integration
|
||||
|
||||
See [ADR-022](../../adr/ADR-022-langchain-langgraph-integration.md). The compiler
|
||||
takes GRAPH actors through an 8-step pipeline, producing LangGraph `StateGraph` structures.
|
||||
|
||||
### MCP & Skills
|
||||
|
||||
MCP adapter implements Model Context Protocol (see [ADR-029](../../adr/ADR-029-model-context-protocol.md)). Skill registry implements the Skill Abstraction Definition (see [ADR-030](../../adr/ADR-030-skill-abstraction-definition.md) and [ADR-012](../../adr/ADR-012-skill-system.md)).
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [Actor Format & Compiler](actor-yaml.md)
|
||||
- [MCP + Tool Router](integration.md)
|
||||
- [Skills + Validation Runner](skills.md)
|
||||
- [Quick Start Guide](quickstart.md) — common workflows
|
||||
@@ -0,0 +1,91 @@
|
||||
# v3.1.0 — MCP Adapter & Tool Router
|
||||
|
||||
External tool server integration and LLM provider normalization.
|
||||
|
||||
---
|
||||
|
||||
## MCP Adapter
|
||||
|
||||
The Model Context Protocol adapter connects CleverAgents to external tool servers.
|
||||
|
||||
### Supported Transports
|
||||
|
||||
| Transport | Description | Config Required |
|
||||
|-----------|-------------|-----------------|
|
||||
| `stdio` | Spawns subprocess | `command` |
|
||||
| `sse` | Server-Sent Events HTTP | `url` |
|
||||
| `streamable-http` | Streamable HTTP | `url` |
|
||||
|
||||
### Configuration & Usage
|
||||
|
||||
```python
|
||||
from cleveragents.mcp.adapter import MCPToolAdapter, MCPServerConfig
|
||||
|
||||
config = MCPServerConfig(
|
||||
name="my-tools", transport="stdio", command="python",
|
||||
args=["-m", "my_mcp_server"], env={"API_KEY": "secret"},
|
||||
)
|
||||
adapter = MCPToolAdapter(config)
|
||||
adapter.connect(timeout=10.0)
|
||||
tools = adapter.discover_tools()
|
||||
result = adapter.invoke("create_issue", {"title": "Bug"})
|
||||
# result: {success: true, data: {"issue_id": 42}, duration_ms: 145.3}
|
||||
adapter.disconnect()
|
||||
```
|
||||
|
||||
### ToolRegistry Integration
|
||||
|
||||
```python
|
||||
from cleveragents.tool.registry import ToolRegistry
|
||||
registry = ToolRegistry()
|
||||
adapter.register_tools(registry, namespace="my-tools")
|
||||
# Tools available as "my-tools/<tool_name>"
|
||||
```
|
||||
|
||||
### Capability Inference (name-based heuristics)
|
||||
|
||||
- `read`, `get`, `list`, `search`, `find` — inferred as read-only
|
||||
- `write`, `create`, `update`, `delete`, `set` — inferred as writable
|
||||
|
||||
### Reconnect on Failure
|
||||
|
||||
```python
|
||||
try: adapter.invoke("tool", {"arg": "x"})
|
||||
except RuntimeError:
|
||||
adapter.reconnect(timeout=10.0)
|
||||
adapter.discover_tools()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tool Router
|
||||
|
||||
Normalizes LLM provider tool calls into a unified internal representation.
|
||||
|
||||
### Supported Provider Formats
|
||||
|
||||
| Provider | Shape | Arguments Key | Type |
|
||||
|----------|-------|---------------|------|
|
||||
| OpenAI | `{"name": "...", "arguments": "..."}` | `arguments` | JSON string |
|
||||
| Anthropic | `{"name": "...", "input": {...}}` | `input` | dict |
|
||||
| LangChain | `{"name": "...", "type": "tool_call", "args": {...}}` | `args` | dict |
|
||||
|
||||
### Usage
|
||||
|
||||
```python
|
||||
from cleveragents.tool.router import ToolCallRouter
|
||||
router = ToolCallRouter(registry, runner, plan_id="plan-001")
|
||||
result = router.route({"name": "files/read_file", 'arguments': '{"path": "a.py"}'})
|
||||
# result.provider_format: "openai" result.tool_call_id: "tc_<24-hex>"
|
||||
|
||||
# Stable IDs: generate_tool_call_id("plan-001", 0) always returns same value
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [MCP Tool Adapter Reference](../../reference/mcp_adapter.md)
|
||||
- [Tool Call Router Reference](../../reference/tool_router.md)
|
||||
- [ADR-011](../../adr/ADR-011-tool-system.md): Tool System
|
||||
- [ADR-029](../../adr/ADR-029-model-context-protocol.md): MCP Adoption
|
||||
@@ -0,0 +1,66 @@
|
||||
# v3.1.0 — Quick Start Guide
|
||||
|
||||
Common workflows: actor definitions, MCP setup, skills, & validation gates.
|
||||
|
||||
---
|
||||
|
||||
## Step-by-Step Workflow
|
||||
|
||||
```bash
|
||||
# 1. Define a graph actor YAML
|
||||
cat > actors/tdd.yaml << 'EOF'
|
||||
version: "3"
|
||||
name: assistants/tdd
|
||||
type: graph
|
||||
model: gpt-4
|
||||
route:
|
||||
nodes:
|
||||
- id: planner; type: agent; name: Planner
|
||||
config: { model: gpt-4, prompt: "Plan tasks from requirements." }
|
||||
- id: implementer; type: agent; name: Implementer
|
||||
config: { model: gpt-4, prompt: "Write clean tested code." }
|
||||
edges: [{from_node: planner, to_node: implementer}]
|
||||
entry_node: planner
|
||||
exit_nodes: [implementer]
|
||||
EOF
|
||||
|
||||
# 2. Create an action using the actor
|
||||
cat > review-action.yaml << 'EOF'
|
||||
name: local/code-review
|
||||
description: Review code changes
|
||||
execution_actor: assistants/tdd
|
||||
definition_of_done: |
|
||||
All changes reviewed for correctness and style.
|
||||
EOF
|
||||
agents action create --config review-action.yaml
|
||||
|
||||
# 3. Configure MCP tool server in ~/.cleveragents/config.toml
|
||||
|
||||
# 4. Register a skill
|
||||
agents skill add --config my-skill.yaml
|
||||
|
||||
# 5. Execute plan with validation gate
|
||||
agents plan use local/code-review my-project
|
||||
agents plan execute # runs validation before apply
|
||||
agents plan diff
|
||||
agents plan apply --yes # blocked if required validations fail
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Actor Creation Checklist
|
||||
|
||||
- [ ] `version: "3"` specified
|
||||
- [ ] Valid type: `llm`, `tool`, or `graph`
|
||||
- [ ] Name follows `namespace/name` format
|
||||
- [ ] All referenced actors and tools are resolvable
|
||||
- [ ] Graph actors have valid entry/exit nodes and acyclic edges
|
||||
|
||||
---
|
||||
|
||||
## Skill & Validation Quick Tips
|
||||
|
||||
- Skills can include other skills for composition
|
||||
- Agent Skills Standard discovers skills from filesystem paths in `agent_skills_paths`
|
||||
- Validations run deterministically: alphabetical (resource, mode, name) order
|
||||
- Run with `max_workers=1` in CI for predictable sequential validation
|
||||
@@ -0,0 +1,127 @@
|
||||
# v3.1.0 — Skill Registry & Validation Runner
|
||||
|
||||
Reusable tool collections and pre-apply validation quality gates.
|
||||
|
||||
---
|
||||
|
||||
## Skill Registry
|
||||
|
||||
Persistent storage for named, reusable tool collections in SQLite.
|
||||
|
||||
### Item Types
|
||||
|
||||
| Type | Description |
|
||||
|------|-------------|
|
||||
| `tool_ref` | Reference to a named tool |
|
||||
| `include` | Recursive inclusion of another skill |
|
||||
| `inline_tool` | Anonymous tool defined inline |
|
||||
| `mcp_source` | MCP server tool source |
|
||||
| `agent_source` | Agent Skills Standard folder source |
|
||||
|
||||
### CLI Commands
|
||||
|
||||
```bash
|
||||
agents skill list # all skills
|
||||
agents skill tools local/devops-toolkit # show skill's tools
|
||||
agents skill add --config ./my-skill.yaml # register from YAML
|
||||
agents skill remove local/my-skill # remove a skill
|
||||
```
|
||||
|
||||
### Service API
|
||||
|
||||
```python
|
||||
from cleveragents.application.services import SkillRegistryService
|
||||
svc = SkillRegistryService(skill_repo=SkillRepository(session_factory))
|
||||
svc.add_skill(skill) # register
|
||||
skill = svc.get_skill("local/code-tools") # retrieve
|
||||
skills = svc.list_skills(namespace="local") # filter
|
||||
svc.remove_skill("local/code-tools") # delete
|
||||
```
|
||||
|
||||
### Agent Skills Standard Integration
|
||||
|
||||
Filesystem-based tool bundles with `SKILL.md` + YAML front-matter.
|
||||
|
||||
**Default discovery path:** `~/.cleveragents/agent_skills`
|
||||
|
||||
**SKILL.md format:**
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: my-tool
|
||||
description: A custom agent skill
|
||||
input_schema: {type: object, properties: {query: {type: string}}}
|
||||
---
|
||||
|
||||
# My Tool
|
||||
Additional documentation.
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
```toml
|
||||
[skills]
|
||||
agent_skills_paths = "/home/user/.cleveragents/agent_skills,/opt/skills"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation Runner
|
||||
|
||||
Executes validations against plan resources before Apply phase.
|
||||
|
||||
### Modes
|
||||
|
||||
| Mode | Pass Behavior | Fail Behavior |
|
||||
|------|---------------|----------------|
|
||||
| `required` | OK | **Blocks apply** — sets `all_required_passed = False` |
|
||||
| `informational` | Logged | Logged only — does NOT block apply |
|
||||
|
||||
### Attachment Scopes
|
||||
|
||||
| Scope | Active When |
|
||||
|-------|-------------|
|
||||
| direct | Always for the resource |
|
||||
| project | Resource accessed via a project |
|
||||
| plan | Only for specific plan |
|
||||
|
||||
### Apply Validation Gate
|
||||
|
||||
```python
|
||||
from cleveragents.validation.gate import ApplyValidationGate
|
||||
gate = ApplyValidationGate(runner=DefaultValidationRunner())
|
||||
summary = gate.run(plan_id, attachments, context)
|
||||
if gate.should_block_apply(summary):
|
||||
# report failure reasons, block apply
|
||||
else:
|
||||
# proceed
|
||||
```
|
||||
|
||||
### Blocking vs Allowed Output
|
||||
|
||||
**Blocked:**
|
||||
```
|
||||
Validation Gate: BLOCKED
|
||||
Required: 1 passed, 1 failed
|
||||
Failures: lint-check on res-001: 3 lint errors found
|
||||
```
|
||||
|
||||
**Allowed:**
|
||||
```
|
||||
Validation Gate: PASSED
|
||||
Required: 2 passed, 0 failed
|
||||
Informational: 1 passed, 0 failed
|
||||
```
|
||||
|
||||
### Concurrency
|
||||
|
||||
Uses `ThreadPoolExecutor` with configurable `max_workers` (default 4).
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [Skill Registry Reference](../../reference/skill_registry.md)
|
||||
- [Validation Pipeline Reference](../../reference/validation_pipeline.md)
|
||||
- [ADR-012](../../adr/ADR-012-skill-system.md): Skill System
|
||||
- [ADR-013](../../adr/ADR-013-validation-abstraction.md): Validation Abstraction
|
||||
+67
-54
@@ -88,8 +88,8 @@ via the `getExtendedAgentCard` operation.
|
||||
|
||||
## Transport Modes
|
||||
|
||||
| Mode | Transport | Class / SDK Component | Behaviour |
|
||||
|--------|------------------------|------------------------------------|--------------------------------------------------------------------|
|
||||
| Mode | Transport | Class / SDK Component | Behaviour |
|
||||
|--------|------------------------|------------------------------------|--------------------------------------------------------|
|
||||
| Local | A2A JSON-RPC over stdio | `A2aLocalFacade` + JSON-RPC stdio | Agent as subprocess; extensions resolved in-process |
|
||||
| Server | A2A JSON-RPC over HTTP | A2A SDK HTTP transport | JSON-RPC 2.0 to CleverAgents server |
|
||||
|
||||
@@ -165,7 +165,7 @@ Pattern applies to all entity types (`actor`, `skill`, `tool`, `validation`,
|
||||
### Context Operations
|
||||
|
||||
| Method | Service Method | Required Params |
|
||||
|--------|----------------|-----------------|
|
||||
|--------|---------------|-----------------|
|
||||
| `_cleveragents/context/show` | `ContextService.show()` | `project_name` |
|
||||
| `_cleveragents/context/inspect` | `ContextService.inspect()` | `project_name` |
|
||||
| `_cleveragents/context/simulate` | `ContextService.simulate()` | `project_name`, simulation params |
|
||||
@@ -174,7 +174,7 @@ Pattern applies to all entity types (`actor`, `skill`, `tool`, `validation`,
|
||||
### Sync Operations
|
||||
|
||||
| Method | Service Method | Required Params |
|
||||
|--------|----------------|-----------------|
|
||||
|--------|---------------|-----------------|
|
||||
| `_cleveragents/sync/pull` | `SyncService.pull()` | `namespace` |
|
||||
| `_cleveragents/sync/push` | `SyncService.push()` | `namespace`, `entities` |
|
||||
| `_cleveragents/sync/status` | `SyncService.status()` | `namespace` (optional) |
|
||||
@@ -182,7 +182,7 @@ Pattern applies to all entity types (`actor`, `skill`, `tool`, `validation`,
|
||||
### Namespace Operations
|
||||
|
||||
| Method | Service Method | Required Params |
|
||||
|--------|----------------|-----------------|
|
||||
|--------|---------------|-----------------|
|
||||
| `_cleveragents/namespace/list` | `NamespaceService.list()` | -- |
|
||||
| `_cleveragents/namespace/show` | `NamespaceService.show()` | `namespace` |
|
||||
| `_cleveragents/namespace/members` | `NamespaceService.members()` | `namespace` |
|
||||
@@ -198,20 +198,13 @@ Pattern applies to all entity types (`actor`, `skill`, `tool`, `validation`,
|
||||
|
||||
## Streaming Events
|
||||
|
||||
A2A streaming uses SSE via the `message/stream` operation (client → server
|
||||
request), delivering typed events as the task progresses. The SSE data
|
||||
payloads are JSON-RPC 2.0 **notifications** (no `id` field):
|
||||
A2A streaming uses SSE via the `message/stream` operation, delivering typed
|
||||
events as the task progresses:
|
||||
|
||||
| Event Type | JSON-RPC Method | Emitted When |
|
||||
|------------|-----------------|--------------|
|
||||
| `TaskStatusUpdateEvent` | `task/statusUpdate` | Task state changes (submitted -> working -> completed, etc.) |
|
||||
| `TaskArtifactUpdateEvent` | `task/artifactUpdate` | Agent produces output (response chunks, plan entries, tool results) |
|
||||
|
||||
> **Note:** `message/stream` is the **request** method (client → server) that
|
||||
> initiates streaming. The SSE data payloads use `task/statusUpdate` or
|
||||
> `task/artifactUpdate` as the notification method name — not `message/stream`.
|
||||
> Non-spec fields (`event_id`, `event_type`, `timestamp`) are carried in SSE
|
||||
> envelope headers (`event:` and `id:` lines), not in the data payload.
|
||||
| Event Type | Emitted When |
|
||||
|-----------|-------------|
|
||||
| `TaskStatusUpdateEvent` | Task state changes (submitted -> working -> completed, etc.) |
|
||||
| `TaskArtifactUpdateEvent` | Agent produces output (response chunks, plan entries, tool results) |
|
||||
|
||||
### Task Lifecycle States
|
||||
|
||||
@@ -227,30 +220,16 @@ payloads are JSON-RPC 2.0 **notifications** (no `id` field):
|
||||
|
||||
### Example Streaming Event
|
||||
|
||||
SSE data payload for a task status update (JSON-RPC 2.0 notification):
|
||||
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"method": "task/statusUpdate",
|
||||
"method": "message/stream",
|
||||
"params": {
|
||||
"taskId": "task_01HXR...",
|
||||
"status": { "state": "working" }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
SSE data payload for an artifact update:
|
||||
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"method": "task/artifactUpdate",
|
||||
"params": {
|
||||
"taskId": "task_01HXR...",
|
||||
"artifact": {
|
||||
"id": "task_01HXR...",
|
||||
"status": { "state": "working" },
|
||||
"artifacts": [{
|
||||
"parts": [{ "text": "I'll start by extracting..." }]
|
||||
}
|
||||
}]
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -298,9 +277,9 @@ response = facade.dispatch(A2aRequest(
|
||||
|
||||
### Constructor
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|------------|----------------------------------|---------|--------------------------------|
|
||||
| `services` | `dict[str, Any] \| None` | `None` | Named services for routing |
|
||||
| Parameter | Type | Default | Description |
|
||||
|------------|----------------------------|---------|-------------------------------|
|
||||
| `services` | `dict[str, Any] \| None` | `None` | Named services for routing |
|
||||
|
||||
### Methods
|
||||
|
||||
@@ -320,15 +299,15 @@ registered later with `register_service()`.
|
||||
|
||||
### Service Keys
|
||||
|
||||
| Key | Type | Wired Methods |
|
||||
|----------------------------------|----------------------------------|--------------------------------------------|
|
||||
| `session_service` | `SessionWorkflow` | Standard message operations |
|
||||
| `plan_lifecycle_service` | `PlanLifecycleService` | `_cleveragents/plan/*` |
|
||||
| `tool_registry` | `ToolRegistry` | `_cleveragents/registry/tool/*` |
|
||||
| `resource_registry_service` | `ResourceRegistryService` | `_cleveragents/registry/resource/*` |
|
||||
| `sync_service` | `SyncService` | `_cleveragents/sync/*` |
|
||||
| `namespace_service` | `NamespaceService` | `_cleveragents/namespace/*` |
|
||||
| `context_service` | `ContextService` | `_cleveragents/context/*` |
|
||||
| Key | Type | Wired Methods |
|
||||
|------------------------------|-----------------------------|-------------------------------------|
|
||||
| `session_service` | `SessionWorkflow` | Standard message operations |
|
||||
| `plan_lifecycle_service` | `PlanLifecycleService` | `_cleveragents/plan/*` |
|
||||
| `tool_registry` | `ToolRegistry` | `_cleveragents/registry/tool/*` |
|
||||
| `resource_registry_service` | `ResourceRegistryService` | `_cleveragents/registry/resource/*` |
|
||||
| `sync_service` | `SyncService` | `_cleveragents/sync/*` |
|
||||
| `namespace_service` | `NamespaceService` | `_cleveragents/namespace/*` |
|
||||
| `context_service` | `ContextService` | `_cleveragents/context/*` |
|
||||
|
||||
---
|
||||
|
||||
@@ -423,8 +402,42 @@ CleverAgentsError
|
||||
└── A2aOperationNotFoundError
|
||||
```
|
||||
|
||||
| Exception | When Raised |
|
||||
|------------------------------|--------------------------------------------------------------------|
|
||||
| `A2aNotAvailableError` | Server-mode operation attempted without connection |
|
||||
| `A2aVersionMismatchError` | Unsupported A2A version |
|
||||
| `A2aOperationNotFoundError` | Unknown method dispatched |
|
||||
| Exception | When Raised |
|
||||
|-----------------------------|----------------------------------------------------|
|
||||
| `A2aNotAvailableError` | Server-mode operation attempted without connection |
|
||||
| `A2aVersionMismatchError` | Unsupported A2A version |
|
||||
| `A2aOperationNotFoundError` | Unknown method dispatched |
|
||||
|
||||
---
|
||||
|
||||
## ACP to A2A Migration
|
||||
|
||||
> **v3.6.0 breaking change.** The `cleveragents.acp` module was renamed to
|
||||
> `cleveragents.a2a` and all `Acp*` symbols were renamed to `A2a*`. The
|
||||
> request/response wire format was updated to comply with JSON-RPC 2.0.
|
||||
|
||||
For a complete migration guide including symbol rename tables, field name
|
||||
changes, operation name mapping, and YAML configuration updates, see
|
||||
[`docs/development/acp-to-a2a-migration.md`](../development/acp-to-a2a-migration.md).
|
||||
|
||||
### Quick Reference: Deprecated Legacy Operations
|
||||
|
||||
The following ACP operation names are accepted by `A2aLocalFacade.dispatch()`
|
||||
in `cleveragents.a2a.facade.A2aLocalFacade` (commit `449c33b7`) for backward
|
||||
compatibility but are **deprecated**:
|
||||
|
||||
| Deprecated (ACP) | Replacement (A2A) |
|
||||
|---|---|
|
||||
| `session.create` | `_cleveragents/session/create` |
|
||||
| `session.close` | `_cleveragents/session/close` |
|
||||
| `plan.create` | `_cleveragents/plan/use` |
|
||||
| `plan.execute` | `_cleveragents/plan/execute` |
|
||||
| `plan.status` | `_cleveragents/plan/status` |
|
||||
| `plan.diff` | `_cleveragents/plan/diff` |
|
||||
| `plan.apply` | `_cleveragents/plan/apply` |
|
||||
| `registry.list_tools` | `_cleveragents/registry/tool/list` |
|
||||
| `registry.list_resources` | `_cleveragents/registry/resource/list` |
|
||||
| `context.get` | `_cleveragents/context/show` |
|
||||
|
||||
See [ADR-047](../adr/ADR-047-acp-standard-adoption.md) for the architectural
|
||||
rationale behind this change.
|
||||
|
||||
@@ -1,71 +0,0 @@
|
||||
@acms-parallel-indexing
|
||||
Feature: ACMS parallel indexing optimization (#9981)
|
||||
Verifies that the FileTraversalEngine uses ThreadPoolExecutor-based
|
||||
parallel processing to efficiently index large projects with 10,000+ files.
|
||||
|
||||
This satisfies the PR #9981 acceptance criterion: "Parallel processing
|
||||
reduces indexing wall-clock time compared to sequential chunking."
|
||||
|
||||
Background:
|
||||
The parallel engine uses configurable worker threads (max_workers)
|
||||
distributed across file stat/reading operations while preserving
|
||||
thread-safe progress tracking and deterministic chunk boundaries.
|
||||
|
||||
Scenario: Parallel FileTraversalEngine processes 100 files without timeout
|
||||
Given a test directory with 100 Python files
|
||||
And a parallel file traversal engine with 4 workers and chunk size 25
|
||||
When I traverse and index the directory in parallel
|
||||
Then the parallel index should contain 100 entries
|
||||
And the indexing should complete within 30 seconds
|
||||
|
||||
Scenario: Parallel FileTraversalEngine skips binary files via null-byte check
|
||||
Given a test directory with 50 text files and 50 binary PNG files
|
||||
And a parallel file traversal engine with 2 workers
|
||||
When I traverse and index the directory in parallel
|
||||
Then the parallel index should contain approximately 50 entries
|
||||
And all indexed entries should have text (not binary) content
|
||||
|
||||
Scenario: Parallel FileTraversalEngine respects exclusion patterns from .gitignore
|
||||
Given a test directory with Python files and a node_modules subdirectory
|
||||
And a parallel file traversal engine with 2 workers
|
||||
When I traverse and index the directory in parallel with exclusions
|
||||
Then the parallel index should not contain any node_modules paths
|
||||
|
||||
Scenario: Parallel FileTraversalEngine respects exclusion patterns from .acmsignore
|
||||
Given a test directory with Python files and a __pycache__ subdirectory
|
||||
And a parallel file traversal engine with 2 workers
|
||||
When I traverse and index the directory in parallel with acms exclusions
|
||||
Then the parallel index should not contain any __pycache__ paths
|
||||
|
||||
Scenario: Parallel FileTraversalEngine with max_workers=1 runs sequentially
|
||||
Given a test directory with 50 Python files
|
||||
And a sequential file traversal engine (max_workers=1) with chunk size 25
|
||||
When I traverse and index the directory in parallel
|
||||
Then the sequential index should contain 50 entries
|
||||
|
||||
Scenario: Parallel FileTraversalEngine tracks thread-safe progress atomically
|
||||
Given a test directory with 200 Python files
|
||||
And a parallel file traversal engine with 4 workers with progress tracking
|
||||
When I traverse and index the directory in parallel
|
||||
Then the progress snapshot should show all 200 files indexed
|
||||
And no errors should have occurred during indexing
|
||||
|
||||
Scenario: Parallel FileTraversalEngine saves JSON cache to disk
|
||||
Given a test directory with 100 Python files
|
||||
And a parallel file traversal engine with 4 workers and cache at "/tmp/parallel_cache.json"
|
||||
When I traverse and index the directory in parallel
|
||||
Then the cache file should exist at the specified path
|
||||
And the cache should contain valid JSON with all indexed entries
|
||||
|
||||
Scenario: Parallel FileTraversalEngine handles permission errors gracefully
|
||||
Given a test directory with 50 Python files and one unreadable subdirectory
|
||||
And a parallel file traversal engine with 4 workers
|
||||
When I traverse and index the directory in parallel
|
||||
Then the parallel index should contain at least 48 entries
|
||||
And the progress snapshot should show zero or few errors
|
||||
|
||||
Scenario: Parallel FileTraversalEngine processes chunk order deterministically
|
||||
Given a test directory with files named "file_00.py" through "file_99.py"
|
||||
And a parallel file traversal engine with 4 workers and chunk size 50
|
||||
When I traverse and index the directory in parallel
|
||||
Then all entries should have valid file paths matching the naming pattern
|
||||
@@ -1,376 +0,0 @@
|
||||
"""Step definitions for acms_parallel_indexing.feature.
|
||||
|
||||
Validates that the parallel FileTraversalEngine correctly uses ThreadPoolExecutor
|
||||
to index large projects with multiple worker threads, while preserving thread safety
|
||||
and deterministic results.
|
||||
|
||||
References:
|
||||
- src/cleveragents/acms/index.py (FileTraversalEngine)
|
||||
- Issue #9981 / PR #9981
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from behave import given, then, when
|
||||
|
||||
from cleveragents.acms.index import (
|
||||
FileTraversalEngine,
|
||||
IndexProgress,
|
||||
)
|
||||
|
||||
|
||||
def _create_test_directory_with_py_files(
|
||||
directory: str, count: int, prefix: str = "file"
|
||||
) -> None:
|
||||
"""Create ``count`` Python source files spread across subdirectories."""
|
||||
root = Path(directory)
|
||||
for i in range(count):
|
||||
subdir = root / f"pkg_{i % 10}"
|
||||
subdir.mkdir(parents=True, exist_ok=True)
|
||||
(subdir / f"{prefix}_{i:03d}.py").write_text(
|
||||
f"# Module {i}\nVALUE = {i}\nx = {i} * 2\n", encoding="utf-8"
|
||||
)
|
||||
|
||||
|
||||
def _create_test_directory_with_binary_files(
|
||||
directory: str, text_count: int, binary_count: int
|
||||
) -> None:
|
||||
"""Create a mix of text and binary (PNG) files."""
|
||||
root = Path(directory)
|
||||
half = text_count // 2
|
||||
for b in range(half):
|
||||
(root / f"text_{b:02d}").mkdir(parents=True, exist_ok=True)
|
||||
for i in range(text_count):
|
||||
bucket = i % half
|
||||
(root / f"text_{bucket:02d}" / f"file_{i:03d}.py").write_text(
|
||||
f"# text file {i}\n", encoding="utf-8"
|
||||
)
|
||||
half_bin = binary_count // 2
|
||||
for b in range(half_bin):
|
||||
(root / f"bin_{b:02d}").mkdir(parents=True, exist_ok=True)
|
||||
for i in range(binary_count):
|
||||
bucket = i % half_bin
|
||||
(root / f"bin_{bucket:02d}" / f"image_{i:03d}.png").write_bytes(
|
||||
b"\x89PNG\r\n\x1a\n" + bytes(range(256)) * 4
|
||||
)
|
||||
|
||||
|
||||
def _create_test_directory_with_gitignore(
|
||||
directory: str, py_count: int, node_modules_entries: int = 30
|
||||
) -> None:
|
||||
"""Create Python files plus a node_modules dir with fake JS files + .gitignore."""
|
||||
root = Path(directory)
|
||||
for i in range(py_count):
|
||||
subdir = root / f"src_{i % 5}"
|
||||
subdir.mkdir(parents=True, exist_ok=True)
|
||||
(subdir / f"module_{i:03d}.py").write_text(
|
||||
f"# src module {i}\n", encoding="utf-8"
|
||||
)
|
||||
nm = root / "node_modules"
|
||||
nm.mkdir(parents=True, exist_ok=True)
|
||||
for i in range(node_modules_entries):
|
||||
(nm / f"pkg_{i}.js").write_text(f"console.log({i});\n", encoding="utf-8")
|
||||
gitignore = root / ".gitignore"
|
||||
gitignore.write_text("node_modules\n", encoding="utf-8")
|
||||
|
||||
|
||||
def _create_test_directory_with_acmsignore(
|
||||
directory: str, py_count: int, pycache_entries: int = 20
|
||||
) -> None:
|
||||
"""Create Python files plus __pycache__ dirs with fake bytecode + .acmsignore."""
|
||||
root = Path(directory)
|
||||
for i in range(py_count):
|
||||
subdir = root / f"src_{i % 5}"
|
||||
subdir.mkdir(parents=True, exist_ok=True)
|
||||
(subdir / f"module_{i:03d}.py").write_text(
|
||||
f"# src module {i}\n", encoding="utf-8"
|
||||
)
|
||||
for i in range(pycache_entries):
|
||||
(root / "__pycache__" / f"module_{i:02d}.cpython-312.pyc").write_bytes(
|
||||
bytes([i] * 64)
|
||||
)
|
||||
acmsignore = root / ".acmsignore"
|
||||
acmsignore.write_text("__pycache__\n", encoding="utf-8")
|
||||
|
||||
|
||||
def _create_test_directory_with_unreadable_dir(directory: str, py_count: int):
|
||||
"""Create Python files plus one subdirectory we cannot read."""
|
||||
import platform
|
||||
|
||||
if platform.system() == "Windows":
|
||||
# On Windows we create a regular dir but the engine handles PermissionError.
|
||||
root = Path(directory)
|
||||
forbidden = root / "forbidden"
|
||||
forbidden.mkdir(parents=True, exist_ok=True)
|
||||
return
|
||||
root = Path(directory)
|
||||
for i in range(py_count):
|
||||
subdir = root / f"src_{i % 5}"
|
||||
subdir.mkdir(parents=True, exist_ok=True)
|
||||
(subdir / f"module_{i:03d}.py").write_text(
|
||||
f"# src module {i}\n", encoding="utf-8"
|
||||
)
|
||||
# Create a subdirectory that we can't read.
|
||||
forbidden = root / "forbidden_dir"
|
||||
forbidden.mkdir(parents=True, exist_ok=True)
|
||||
(forbidden / "secret.py").write_text("# secret\n", encoding="utf-8")
|
||||
os.chmod(forbidden, 0o000)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Given - test data fixtures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@given("a test directory with {count:d} Python files")
|
||||
def step_create_py_dir(context: Any, count: int) -> None:
|
||||
_create_test_directory_with_py_files(
|
||||
directory=tempfile.mkdtemp(prefix="parallel-idx-"),
|
||||
count=count,
|
||||
prefix="file",
|
||||
)
|
||||
|
||||
|
||||
@given(
|
||||
"a test directory with {text_count:d} text files and {binary_count:d} binary PNG files"
|
||||
)
|
||||
def step_create_mixed_file_dir(
|
||||
context: Any, text_count: int, binary_count: int
|
||||
) -> None:
|
||||
d = tempfile.mkdtemp(prefix="parallel-idx-mixed-")
|
||||
_create_test_directory_with_binary_files(d, text_count, binary_count)
|
||||
context.temp_dir_path = d
|
||||
context.add_cleanup(shutil.rmtree, d, True)
|
||||
|
||||
|
||||
@given("a test directory with Python files and a node_modules subdirectory")
|
||||
def step_create_gitignore_dir(context: Any) -> None:
|
||||
d = tempfile.mkdtemp(prefix="parallel-idx-giti-")
|
||||
_create_test_directory_with_gitignore(d, py_count=50)
|
||||
context.temp_dir_path = d
|
||||
context.add_cleanup(shutil.rmtree, d, True)
|
||||
|
||||
|
||||
@given("a test directory with Python files and a __pycache__ subdirectory")
|
||||
def step_create_acmsignore_dir(context: Any) -> None:
|
||||
d = tempfile.mkdtemp(prefix="parallel-idx-acms-")
|
||||
_create_test_directory_with_acmsignore(d, py_count=50)
|
||||
context.temp_dir_path = d
|
||||
context.add_cleanup(shutil.rmtree, d, True)
|
||||
|
||||
|
||||
@given('a test directory with files named "file_00.py" through "file_99.py"')
|
||||
def step_create_named_files(context: Any) -> None:
|
||||
d = tempfile.mkdtemp(prefix="parallel-idx-named-")
|
||||
_create_test_directory_with_py_files(d, count=100)
|
||||
context.temp_dir_path = d
|
||||
context.add_cleanup(shutil.rmtree, d, True)
|
||||
|
||||
|
||||
@given(
|
||||
"a parallel file traversal engine with {workers:d} workers and chunk size {chunk:d}"
|
||||
)
|
||||
def step_create_parallel_engine(context: Any, workers: int, chunk: int) -> None:
|
||||
engine = FileTraversalEngine(chunk_size=chunk, max_workers=workers)
|
||||
context.parallel_engine = engine
|
||||
|
||||
|
||||
@given("a parallel file traversal engine with {workers:d} workers")
|
||||
def step_create_parallel_engine_simple(context: Any, workers: int) -> None:
|
||||
engine = FileTraversalEngine(chunk_size=25, max_workers=workers)
|
||||
context.parallel_engine = engine
|
||||
|
||||
|
||||
@given("a sequential file traversal engine (max_workers=1) with chunk size {chunk:d}")
|
||||
def step_create_sequential_engine(context: Any, chunk: int) -> None:
|
||||
engine = FileTraversalEngine(chunk_size=chunk, max_workers=1)
|
||||
context.parallel_engine = engine
|
||||
|
||||
|
||||
@given(
|
||||
"a parallel file traversal engine with {workers:d} workers with progress tracking"
|
||||
)
|
||||
def step_create_parallel_engine_with_progress(context: Any, workers: int) -> None:
|
||||
progress = IndexProgress()
|
||||
engine = FileTraversalEngine(chunk_size=25, max_workers=workers, progress=progress)
|
||||
context.parallel_engine = engine
|
||||
context.progress_tracker = progress
|
||||
|
||||
|
||||
@given(
|
||||
'a parallel file traversal engine with {workers:d} workers and cache at "{cache_path}"'
|
||||
)
|
||||
def step_create_parallel_engine_with_cache(
|
||||
context: Any, workers: int, cache_path: str
|
||||
) -> None:
|
||||
engine = FileTraversalEngine(
|
||||
chunk_size=25, max_workers=workers, cache_path=cache_path
|
||||
)
|
||||
context.parallel_engine = engine
|
||||
|
||||
|
||||
@given("a test directory with {count:d} Python files and one unreadable subdirectory")
|
||||
def step_create_partial_forbidden_dir(context: Any, count: int) -> None:
|
||||
d = tempfile.mkdtemp(prefix="parallel-idx-fb-")
|
||||
_create_test_directory_with_unreadable_dir(d, py_count=count)
|
||||
context.temp_dir_path = d
|
||||
context.add_cleanup(shutil.rmtree, d, True)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# When - indexing actions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@when("I traverse and index the directory in parallel")
|
||||
def step_traverse_parallel(context: Any) -> None:
|
||||
"""Run traversal with the pre-configured engine."""
|
||||
if not hasattr(context, "temp_dir_path"):
|
||||
# Create inline temp dir for scenarios using generic Given
|
||||
d = tempfile.mkdtemp(prefix="parallel-idx-wf-")
|
||||
_create_test_directory_with_py_files(d, count=100)
|
||||
context.temp_dir_path = d
|
||||
start = time.monotonic()
|
||||
engine = context.parallel_engine
|
||||
engine.reset_index()
|
||||
context.index = engine.traverse_and_index(context.temp_dir_path)
|
||||
context.index_elapsed = time.monotonic() - start
|
||||
if hasattr(context, "progress_tracker"):
|
||||
context.progress_snapshot = context.progress_tracker.snapshot()
|
||||
|
||||
|
||||
@when("I traverse and index the directory in parallel with exclusions")
|
||||
def step_traverse_parallel_exclusions(context: Any) -> None:
|
||||
engine = context.parallel_engine
|
||||
engine.reset_index()
|
||||
context.index = engine.traverse_and_index(
|
||||
context.temp_dir_path, exclude_patterns=["node_modules"]
|
||||
)
|
||||
|
||||
|
||||
@when("I traverse and index the directory in parallel with acms exclusions")
|
||||
def step_traverse_parallel_acms_exclusions(context: Any) -> None:
|
||||
engine = context.parallel_engine
|
||||
engine.reset_index()
|
||||
context.index = engine.traverse_and_index(
|
||||
context.temp_dir_path, exclude_patterns=["__pycache__"]
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Then - assertions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@then("the parallel index should contain {count:d} entries")
|
||||
def step_check_parallel_count(context: Any, count: int) -> None:
|
||||
actual = context.index.get_entry_count()
|
||||
assert actual == count, f"Expected {count} entries, got {actual}"
|
||||
|
||||
|
||||
@then("the parallel index should contain approximately {count:d} entries")
|
||||
def step_check_parallel_approx_count(context: Any, count: int) -> None:
|
||||
actual = context.index.get_entry_count()
|
||||
assert abs(actual - count) <= count * 0.15, (
|
||||
f"Expected ~{count} entries, got {actual}"
|
||||
)
|
||||
|
||||
|
||||
@then("the indexing should complete within {seconds:d} seconds")
|
||||
def step_check_indexing_time(context: Any, seconds: int) -> None:
|
||||
elapsed = context.index_elapsed
|
||||
assert elapsed < seconds, (
|
||||
f"Indexing took {elapsed:.2f}s, exceeding {seconds}s limit."
|
||||
)
|
||||
|
||||
|
||||
@then("all indexed entries should have text (not binary) content")
|
||||
def step_check_all_text_entries(context: Any) -> None:
|
||||
for entry in context.index.get_all_entries():
|
||||
assert not str(entry.path).endswith(".png"), (
|
||||
f"Binary PNG file was incorrectly indexed: {entry.path}"
|
||||
)
|
||||
|
||||
|
||||
@then("the parallel index should not contain any {pattern:s} paths")
|
||||
def step_check_no_pattern_paths(context: Any, pattern: str) -> None:
|
||||
for entry in context.index.get_all_entries():
|
||||
assert pattern not in entry.path, (
|
||||
f"Excluded path '{pattern}' found in index: {entry.path}"
|
||||
)
|
||||
|
||||
|
||||
@then("the sequential index should contain {count:d} entries")
|
||||
def step_check_sequential_count(context: Any, count: int) -> None:
|
||||
actual = context.index.get_entry_count()
|
||||
assert actual == count, f"Expected {count} entries, got {actual}"
|
||||
|
||||
|
||||
@then("the progress snapshot should show all 200 files indexed")
|
||||
def step_check_progress_200(context: Any) -> None:
|
||||
snap = context.progress_snapshot
|
||||
assert snap["files_indexed"] == 200, (
|
||||
f"Expected 200 indexed, got {snap['files_indexed']}"
|
||||
)
|
||||
|
||||
|
||||
@then("no errors should have occurred during indexing")
|
||||
def step_check_no_errors(context: Any) -> None:
|
||||
snap = context.progress_snapshot
|
||||
assert snap["errors_occurred"] == 0, (
|
||||
f"Expected 0 errors, got {snap['errors_occurred']}"
|
||||
)
|
||||
|
||||
|
||||
@then("the cache file should exist at the specified path")
|
||||
def step_check_cache_exists(context: Any) -> None:
|
||||
# The cache path is determined from the engine.
|
||||
assert context.parallel_engine.cache_path is not None
|
||||
assert Path(context.parallel_engine.cache_path).exists(), (
|
||||
f"Cache file does not exist at {context.parallel_engine.cache_path}"
|
||||
)
|
||||
|
||||
|
||||
@then("the cache should contain valid JSON with all indexed entries")
|
||||
def step_check_cache_valid_json(context: Any) -> None:
|
||||
cp = context.parallel_engine.cache_path
|
||||
assert cp is not None
|
||||
data = json.loads(Path(cp).read_text(encoding="utf-8"))
|
||||
entries = data.get("entries", [])
|
||||
count = data.get("count", 0)
|
||||
actual_indexed = context.index.get_entry_count()
|
||||
assert len(entries) == actual_indexed, (
|
||||
f"Cache has {len(entries)} entries but index has {actual_indexed}"
|
||||
)
|
||||
assert count == actual_indexed
|
||||
|
||||
|
||||
@then("the parallel index should contain at least {count:d} entries")
|
||||
def step_check_parallel_min_count(context: Any, count: int) -> None:
|
||||
actual = context.index.get_entry_count()
|
||||
assert actual >= count, f"Expected at least {count} entries, got {actual}"
|
||||
|
||||
|
||||
@then("the progress snapshot should show zero or few errors")
|
||||
def step_check_few_errors(context: Any) -> None:
|
||||
snap = context.progress_snapshot
|
||||
assert snap["errors_occurred"] <= 5, (
|
||||
f"Expected at most 5 errors, got {snap['errors_occurred']}"
|
||||
)
|
||||
|
||||
|
||||
@then("all entries should have valid file paths matching the naming pattern")
|
||||
def step_check_named_entries(context: Any) -> None:
|
||||
for entry in context.index.get_all_entries():
|
||||
path = entry.path
|
||||
assert "/file_" in path or "file_" in path, (
|
||||
f"Entry path does not match expected naming pattern: {path}"
|
||||
)
|
||||
+94
-91
@@ -1,13 +1,8 @@
|
||||
# yaml-language-server: $schema=https://json.schemastore.org/mkdocs-1.6.json
|
||||
# yaml-language-server: customTags:
|
||||
# - !ENV scalar
|
||||
# - !ENV sequence
|
||||
site_name: CleverAgents Documentation
|
||||
site_description: Documentation for CleverAgents.
|
||||
site_author: CleverThis, Inc.
|
||||
site_url: https://docs.cleverthis.com/cleveragents
|
||||
site_dir: build/site
|
||||
|
||||
nav:
|
||||
- Specification: specification.md
|
||||
- Architecture: architecture.md
|
||||
@@ -52,6 +47,19 @@ nav:
|
||||
- Automation Tracking: development/automation-tracking.md
|
||||
- Custom Sandbox Strategy: development/custom_sandbox_strategy.md
|
||||
- Documentation Writer: development/docs-writer.md
|
||||
- ACP to A2A Migration: development/acp-to-a2a-migration.md
|
||||
- Milestones:
|
||||
- Overview: milestones/index.md
|
||||
- "v3.0.0 \u2014 Minimal Local Source-Code Workflow (M1)":
|
||||
- milestones/v3.0.0/index.md
|
||||
- milestones/v3.0.0/cli-reference.md
|
||||
- milestones/v3.0.0/deep-dive.md
|
||||
- "v3.1.0 \u2014 Actor Compiler + Full LLM Integration (M2)":
|
||||
- milestones/v3.1.0/index.md
|
||||
- milestones/v3.1.0/actor-yaml.md
|
||||
- milestones/v3.1.0/integration.md
|
||||
- milestones/v3.1.0/skills.md
|
||||
- milestones/v3.1.0/quickstart.md
|
||||
- Implementation Timeline: timeline.md
|
||||
- Advanced Concepts (v3.6.0):
|
||||
- Overview & Context Strategies: advanced-concepts/index.md
|
||||
@@ -118,105 +126,100 @@ nav:
|
||||
- ADR-046 TUI Reference and Command System: adr/ADR-046-tui-reference-and-command-system.md
|
||||
- ADR-047 A2A Standard Adoption: adr/ADR-047-acp-standard-adoption.md
|
||||
- ADR-048 Server Application Architecture: adr/ADR-048-server-application-architecture.md
|
||||
|
||||
theme:
|
||||
name: material
|
||||
custom_dir: docs/overrides
|
||||
features:
|
||||
- navigation.tabs
|
||||
- navigation.tabs.sticky
|
||||
- toc.integrate
|
||||
- toc.follow
|
||||
- navigation.tabs
|
||||
- navigation.tabs.sticky
|
||||
- toc.integrate
|
||||
- toc.follow
|
||||
palette:
|
||||
- scheme: default
|
||||
primary: blue
|
||||
accent: cyan
|
||||
toggle:
|
||||
icon: material/brightness-7
|
||||
name: Switch to dark mode
|
||||
- scheme: slate
|
||||
primary: blue
|
||||
accent: cyan
|
||||
toggle:
|
||||
icon: material/brightness-4
|
||||
name: Switch to light mode
|
||||
|
||||
- scheme: default
|
||||
primary: blue
|
||||
accent: cyan
|
||||
toggle:
|
||||
icon: material/brightness-7
|
||||
name: Switch to dark mode
|
||||
- scheme: slate
|
||||
primary: blue
|
||||
accent: cyan
|
||||
toggle:
|
||||
icon: material/brightness-4
|
||||
name: Switch to light mode
|
||||
hooks:
|
||||
- hooks/adr_hooks.py
|
||||
|
||||
- hooks/adr_hooks.py
|
||||
extra_css:
|
||||
- stylesheets/extra.css
|
||||
|
||||
- stylesheets/extra.css
|
||||
extra_javascript:
|
||||
- javascripts/adr-page.js
|
||||
- javascripts/toc-collapse.js
|
||||
- javascripts/diagram-lightbox.js
|
||||
|
||||
- javascripts/adr-page.js
|
||||
- javascripts/toc-collapse.js
|
||||
- javascripts/diagram-lightbox.js
|
||||
extra:
|
||||
adr_tiers:
|
||||
1:
|
||||
title: "Foundational"
|
||||
description: "Structural and technological foundation upon which all other decisions rest."
|
||||
title: Foundational
|
||||
description: Structural and technological foundation upon which all other decisions
|
||||
rest.
|
||||
2:
|
||||
title: "Core Domain"
|
||||
description: "Domain model — entities, lifecycles, and relationships that constitute core logic."
|
||||
title: Core Domain
|
||||
description: "Domain model \u2014 entities, lifecycles, and relationships that\
|
||||
\ constitute core logic."
|
||||
3:
|
||||
title: "Infrastructure and Behavior"
|
||||
description: "Cross-cutting behavioral systems and infrastructure concerns."
|
||||
title: Infrastructure and Behavior
|
||||
description: Cross-cutting behavioral systems and infrastructure concerns.
|
||||
4:
|
||||
title: "Integration and Operations"
|
||||
description: "External integrations, operational interfaces, and deployment concerns."
|
||||
|
||||
title: Integration and Operations
|
||||
description: External integrations, operational interfaces, and deployment concerns.
|
||||
plugins:
|
||||
- search
|
||||
- gen-files:
|
||||
scripts:
|
||||
- docs/gen_ref_pages.py
|
||||
- literate-nav:
|
||||
nav_file: SUMMARY.md
|
||||
- mkdocstrings:
|
||||
handlers:
|
||||
python:
|
||||
paths: [src]
|
||||
options:
|
||||
docstring_style: google
|
||||
docstring_section_style: table
|
||||
show_root_heading: true
|
||||
show_source: true
|
||||
merge_init_into_class: true
|
||||
separate_signature: true
|
||||
show_signature_annotations: true
|
||||
signature_crossrefs: true
|
||||
summary: true
|
||||
extensions:
|
||||
- griffe_pydantic:
|
||||
schema: true
|
||||
- kroki:
|
||||
server_url: https://kroki.qoto.org
|
||||
#server_url: https://kroki.io
|
||||
enable_mermaid: true
|
||||
fence_prefix: kroki-
|
||||
http_method: POST
|
||||
|
||||
- search
|
||||
- gen-files:
|
||||
scripts:
|
||||
- docs/gen_ref_pages.py
|
||||
- literate-nav:
|
||||
nav_file: SUMMARY.md
|
||||
- mkdocstrings:
|
||||
handlers:
|
||||
python:
|
||||
paths:
|
||||
- src
|
||||
options:
|
||||
docstring_style: google
|
||||
docstring_section_style: table
|
||||
show_root_heading: true
|
||||
show_source: true
|
||||
merge_init_into_class: true
|
||||
separate_signature: true
|
||||
show_signature_annotations: true
|
||||
signature_crossrefs: true
|
||||
summary: true
|
||||
extensions:
|
||||
- griffe_pydantic:
|
||||
schema: true
|
||||
- kroki:
|
||||
server_url: https://kroki.qoto.org
|
||||
enable_mermaid: true
|
||||
fence_prefix: kroki-
|
||||
http_method: POST
|
||||
markdown_extensions:
|
||||
- admonition
|
||||
- pymdownx.details
|
||||
- pymdownx.superfences
|
||||
- pymdownx.tabbed:
|
||||
alternate_style: true
|
||||
- pymdownx.critic
|
||||
- pymdownx.caret
|
||||
- pymdownx.keys
|
||||
- pymdownx.mark
|
||||
- pymdownx.tilde
|
||||
- pymdownx.tasklist:
|
||||
custom_checkbox: true
|
||||
- attr_list
|
||||
- def_list
|
||||
- footnotes
|
||||
- md_in_html
|
||||
- tables
|
||||
- codehilite
|
||||
- toc:
|
||||
permalink: true
|
||||
toc_depth: 1-6
|
||||
- admonition
|
||||
- pymdownx.details
|
||||
- pymdownx.superfences
|
||||
- pymdownx.tabbed:
|
||||
alternate_style: true
|
||||
- pymdownx.critic
|
||||
- pymdownx.caret
|
||||
- pymdownx.keys
|
||||
- pymdownx.mark
|
||||
- pymdownx.tilde
|
||||
- pymdownx.tasklist:
|
||||
custom_checkbox: true
|
||||
- attr_list
|
||||
- def_list
|
||||
- footnotes
|
||||
- md_in_html
|
||||
- tables
|
||||
- codehilite
|
||||
- toc:
|
||||
permalink: true
|
||||
toc_depth: 1-6
|
||||
|
||||
-11
@@ -1201,17 +1201,6 @@ def benchmark_regression(session: nox.Session):
|
||||
"""Run Airspeed Velocity benchmarks regression test."""
|
||||
session.install("-e", ".[tests]")
|
||||
config_path = "asv.conf.json"
|
||||
|
||||
# Ensure ASV result directories exist (handles cases where S3 sync is skipped)
|
||||
results_dir = os.path.join("build", "asv", "results")
|
||||
machine_dir = os.path.join(results_dir, "forgejo-runner")
|
||||
session.run(
|
||||
"mkdir",
|
||||
"-p",
|
||||
machine_dir,
|
||||
external=True,
|
||||
)
|
||||
|
||||
asv_base_sha = os.environ.get("ASV_BASE_SHA", "master")
|
||||
session.run(
|
||||
"asv",
|
||||
|
||||
+122
-31
@@ -1,48 +1,139 @@
|
||||
*** Settings ***
|
||||
Documentation E2E acceptance test for M2 (v3.1.0): Actor Compiler + Full LLM Integration.
|
||||
Documentation E2E acceptance test for M2 (v3.1.0): Actor Compiler + Full LLM Integration.
|
||||
...
|
||||
... Exercises actor YAML compilation into functional graphs, skill registry,
|
||||
... tool lifecycle, and plan execution with a custom actor using real LLM keys.
|
||||
... Zero mocking — all CLI invocations hit the real CleverAgents binary with
|
||||
... real provider keys.
|
||||
Resource common_e2e.resource
|
||||
Suite Setup E2E Suite Setup
|
||||
... Exercises actor YAML compilation into functional graphs, skill registry,
|
||||
... tool lifecycle, and plan execution with a custom actor using real LLM keys.
|
||||
... Zero mocking — all CLI invocations hit the real CleverAgents binary with
|
||||
... real provider keys.
|
||||
Resource common_e2e.resource
|
||||
Suite Setup E2E Suite Setup
|
||||
Suite Teardown E2E Suite Teardown
|
||||
|
||||
*** Variables ***
|
||||
${ACTOR_NAME} local/m2-e2e-actor
|
||||
${ACTION_NAME} local/m2-e2e-action
|
||||
${RESOURCE_NAME} local/m2-e2e-repo
|
||||
${PROJECT_NAME} local/m2-e2e-project
|
||||
${ACTOR_NAME} local/m2-e2e-actor
|
||||
${ACTION_NAME} local/m2-e2e-action
|
||||
${RESOURCE_NAME} local/m2-e2e-repo
|
||||
${PROJECT_NAME} local/m2-e2e-project
|
||||
|
||||
*** Test Cases ***
|
||||
M2 Full Actor Compiler And LLM Integration
|
||||
[Documentation] End-to-end acceptance test for the M2 milestone.
|
||||
...
|
||||
... Exercises actor YAML compilation, registration via CLI,
|
||||
... resource and project setup, action creation referencing a
|
||||
... custom actor, and the full plan lifecycle (use, execute
|
||||
... strategize, execute, diff, apply) with real LLM API keys.
|
||||
[Tags] E2E tdd_issue tdd_issue_4189 tdd_expected_fail
|
||||
Skip If No LLM Keys
|
||||
# ---- Step 1: Create temp git repo with sample project files ----
|
||||
${repo_dir}= Create Temp Git Repo m2-e2e-repo
|
||||
Create Directory ${repo_dir}${/}src
|
||||
Create File ${repo_dir}${/}src${/}main.py print("hello world")\n
|
||||
${r_add}= Run Process git add . cwd=${repo_dir} timeout=60s on_timeout=kill
|
||||
Should Be Equal As Integers ${r_add.rc} 0 msg=git add failed (rc=${r_add.rc}). Check DEBUG logs above.
|
||||
${r_commit}= Run Process git commit -m Add source files cwd=${repo_dir} timeout=60s on_timeout=kill
|
||||
Should Be Equal As Integers ${r_commit.rc} 0 msg=git commit failed (rc=${r_commit.rc}). Check DEBUG logs above.
|
||||
# Detect the default branch name created by git init
|
||||
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir} timeout=60s on_timeout=kill
|
||||
${branch}= Strip String ${branch_result.stdout}
|
||||
Log Detected branch: ${branch}
|
||||
[Documentation] End-to-end acceptance test for the M2 milestone.
|
||||
...
|
||||
... Exercises actor YAML compilation, registration via CLI,
|
||||
... resource and project setup, action creation referencing a
|
||||
... custom actor, and the full plan lifecycle (use, execute
|
||||
... strategize, execute, diff, apply) with real LLM API keys.
|
||||
[Tags] E2E
|
||||
Skip If No LLM Keys
|
||||
# ---- Step 1: Create temp git repo with sample project files ----
|
||||
${repo_dir}= Create Temp Git Repo m2-e2e-repo
|
||||
Create Directory ${repo_dir}${/}src
|
||||
Create File ${repo_dir}${/}src${/}main.py print("hello world")\n
|
||||
${r_add}= Run Process git add . cwd=${repo_dir} timeout=60s on_timeout=kill
|
||||
Should Be Equal As Integers ${r_add.rc} 0 msg=git add failed (rc=${r_add.rc}). Check DEBUG logs above.
|
||||
${r_commit}= Run Process git commit -m Add source files cwd=${repo_dir} timeout=60s on_timeout=kill
|
||||
Should Be Equal As Integers ${r_commit.rc} 0 msg=git commit failed (rc=${r_commit.rc}). Check DEBUG logs above.
|
||||
# Detect the default branch name created by git init
|
||||
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir} timeout=60s on_timeout=kill
|
||||
${branch}= Strip String ${branch_result.stdout}
|
||||
Log Detected branch: ${branch}
|
||||
|
||||
# ---- Resolve LLM actor (OpenAI if available, Anthropic fallback) ----
|
||||
${llm_actor}= Resolve LLM Actor
|
||||
${llm_parts}= Evaluate $llm_actor.split('/', 1)
|
||||
${llm_provider}= Set Variable ${llm_parts}[0]
|
||||
${llm_model}= Set Variable ${llm_parts}[1]
|
||||
|
||||
# ---- Step 2: Create and register custom actor YAML ----
|
||||
${actor_yaml}= Catenate SEPARATOR=\n
|
||||
... name: ${ACTOR_NAME}
|
||||
... type: llm
|
||||
... description: M2 E2E acceptance actor for compiler and LLM integration
|
||||
... version: "1.0"
|
||||
... model: ${llm_model}
|
||||
${actor_yaml_path}= Set Variable ${SUITE_HOME}${/}m2_actor.yaml
|
||||
Create File ${actor_yaml_path} ${actor_yaml}
|
||||
${actor_config}= Catenate SEPARATOR=\n
|
||||
... {
|
||||
... "name": "${ACTOR_NAME}",
|
||||
... "provider": "${llm_provider}",
|
||||
... "model": "${llm_model}",
|
||||
... "options": {"temperature": 0.2}
|
||||
... }
|
||||
${config_path}= Set Variable ${SUITE_HOME}${/}actor_config.json
|
||||
Create File ${config_path} ${actor_config}
|
||||
${r_actor}= Run CleverAgents Command
|
||||
... actor add --config ${config_path} --format plain
|
||||
Should Be Equal As Integers ${r_actor.rc} 0 msg=Actor add failed (rc=${r_actor.rc}). Check DEBUG logs above.
|
||||
Should Not Contain ${r_actor.stdout}${r_actor.stderr} Traceback
|
||||
Log Actor registration: ${r_actor.stdout}
|
||||
|
||||
# ---- Step 3: Register resource and create project ----
|
||||
${r_resource}= Run CleverAgents Command
|
||||
... resource add git-checkout ${RESOURCE_NAME}
|
||||
... --path ${repo_dir} --branch ${branch} --format plain
|
||||
Output Should Contain ${r_resource} ${RESOURCE_NAME}
|
||||
${r_project}= Run CleverAgents Command
|
||||
... project create ${PROJECT_NAME}
|
||||
... --description M2 E2E acceptance project
|
||||
... --resource ${RESOURCE_NAME} --format plain
|
||||
Output Should Contain ${r_project} ${PROJECT_NAME}
|
||||
|
||||
# ---- Step 4: Create action referencing the custom actor ----
|
||||
${action_yaml}= Catenate SEPARATOR=\n
|
||||
... name: ${ACTION_NAME}
|
||||
... description: M2 acceptance test action for actor compiler and LLM integration
|
||||
... definition_of_done: Generate or modify at least one source file
|
||||
... strategy_actor: ${llm_actor}
|
||||
... execution_actor: ${llm_actor}
|
||||
${action_yaml_path}= Set Variable ${SUITE_HOME}${/}action.yaml
|
||||
Create File ${action_yaml_path} ${action_yaml}
|
||||
${r_action}= Run CleverAgents Command
|
||||
... action create --config ${action_yaml_path} --format plain
|
||||
Output Should Contain ${r_action} ${ACTION_NAME}
|
||||
|
||||
# ---- Step 5: Plan use ----
|
||||
${r_use}= Run CleverAgents Command
|
||||
... plan use ${ACTION_NAME} ${PROJECT_NAME} --format plain
|
||||
Should Not Be Empty ${r_use.stdout}
|
||||
${plan_ids}= Get Regexp Matches ${r_use.stdout} [0-9A-Z]{26}
|
||||
Should Not Be Empty ${plan_ids} msg=Expected a ULID plan ID in plan use output
|
||||
${plan_id}= Set Variable ${plan_ids}[0]
|
||||
Log Extracted plan_id: ${plan_id}
|
||||
|
||||
# ---- Step 6: Plan execute — strategize phase ----
|
||||
${r_strategize}= Run CleverAgents Command
|
||||
... plan execute ${plan_id} --format plain
|
||||
... timeout=180s
|
||||
Should Not Contain ${r_strategize.stdout}${r_strategize.stderr} INTERNAL
|
||||
Should Not Contain ${r_strategize.stdout}${r_strategize.stderr} Traceback
|
||||
Log Strategize phase output: ${r_strategize.stdout}
|
||||
|
||||
# ---- Step 7: Plan execute — execute phase ----
|
||||
${r_execute}= Run CleverAgents Command
|
||||
... plan execute ${plan_id} --format plain
|
||||
... timeout=180s
|
||||
Should Not Contain ${r_execute.stdout}${r_execute.stderr} INTERNAL
|
||||
Should Not Contain ${r_execute.stdout}${r_execute.stderr} Traceback
|
||||
Log Execute phase output: ${r_execute.stdout}
|
||||
|
||||
# ---- Step 8: Plan diff ----
|
||||
${r_diff}= Run CleverAgents Command
|
||||
... plan diff ${plan_id} --format plain
|
||||
Should Not Contain ${r_diff.stdout}${r_diff.stderr} INTERNAL
|
||||
Should Not Contain ${r_diff.stdout}${r_diff.stderr} Traceback
|
||||
Log Diff output: ${r_diff.stdout}
|
||||
|
||||
# ---- Step 9: Plan apply ----
|
||||
${r_apply}= Run CleverAgents Command
|
||||
... plan apply --yes ${plan_id} --format plain
|
||||
Should Not Contain ${r_apply.stdout}${r_apply.stderr} INTERNAL
|
||||
Should Not Contain ${r_apply.stdout}${r_apply.stderr} Traceback
|
||||
Log Apply output: ${r_apply.stdout}
|
||||
|
||||
# ---- Step 10: Verify actor compilation and plan integrity ----
|
||||
${r_status}= Run CleverAgents Command
|
||||
... plan status ${plan_id} --format plain
|
||||
Should Not Be Empty ${r_status.stdout}
|
||||
Output Should Contain ${r_status} ${plan_id}
|
||||
Log Final plan status: ${r_status.stdout}
|
||||
|
||||
+34
-406
@@ -2,30 +2,20 @@
|
||||
|
||||
Provides the foundational data model for indexed context entries and a
|
||||
file traversal engine that can handle 10,000+ files without timeout using
|
||||
chunked processing with parallel worker threads.
|
||||
chunked processing.
|
||||
|
||||
Based on issue #9579 and ``docs/specification.md`` ~lines 44405-44420.
|
||||
Optimized for PR #9981: parallel processing reduces indexing time for
|
||||
large projects by distributing file stat/reading work across multiple
|
||||
threads while preserving thread safety and deterministic chunk boundaries.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import threading
|
||||
from collections.abc import Iterator
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from datetime import datetime
|
||||
from enum import StrEnum
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from typing import Any
|
||||
|
||||
|
||||
class FileType(StrEnum):
|
||||
"""File type enumeration for index entries."""
|
||||
@@ -77,7 +67,7 @@ class IndexEntry(BaseModel):
|
||||
modified_at: datetime
|
||||
tags: set[str] = Field(default_factory=set)
|
||||
tier: TierLevel = TierLevel.COLD
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
metadata: dict[str, str] = Field(default_factory=dict)
|
||||
|
||||
def add_tag(self, tag: str) -> None:
|
||||
"""Add a tag to this entry.
|
||||
@@ -114,74 +104,6 @@ class IndexEntry(BaseModel):
|
||||
self.tier = tier
|
||||
|
||||
|
||||
class IndexProgress(BaseModel):
|
||||
"""Thread-safe progress tracker for file traversal indexing.
|
||||
|
||||
Tracks how many files have been processed and their types.
|
||||
Thread-safe via a single lock protecting all counters.
|
||||
"""
|
||||
|
||||
files_processed: int = 0
|
||||
files_skipped: int = 0
|
||||
files_indexed: int = 0
|
||||
bytes_read: int = 0
|
||||
binary_files_found: int = 0
|
||||
errors_occurred: int = 0
|
||||
|
||||
def __init__(self) -> None:
|
||||
"""Initialize progress tracker with a reentrant lock."""
|
||||
super().__init__()
|
||||
self._lock = threading.RLock()
|
||||
|
||||
def record_processed(self, bytes_read: int = 0) -> None:
|
||||
"""Record that a file was processed (stat+read completed)."""
|
||||
with self._lock:
|
||||
self.files_processed += 1
|
||||
self.bytes_read += bytes_read
|
||||
|
||||
def record_skipped(self) -> None:
|
||||
"""Record that a file was skipped (excluded/by pattern)."""
|
||||
with self._lock:
|
||||
self.files_skipped += 1
|
||||
|
||||
def record_indexed(self) -> None:
|
||||
"""Record that an entry was successfully indexed."""
|
||||
with self._lock:
|
||||
self.files_indexed += 1
|
||||
|
||||
def record_binary(self) -> None:
|
||||
"""Record a binary file detection."""
|
||||
with self._lock:
|
||||
self.binary_files_found += 1
|
||||
|
||||
def record_error(self) -> None:
|
||||
"""Record an indexing error."""
|
||||
with self._lock:
|
||||
self.errors_occurred += 1
|
||||
|
||||
@property
|
||||
def progress_percent(self) -> float:
|
||||
"""Return completion percentage (0-100)."""
|
||||
with self._lock:
|
||||
total = self.files_processed + self.files_skipped
|
||||
if total == 0:
|
||||
return 0.0
|
||||
return round((self.files_indexed / total) * 100, 2)
|
||||
|
||||
def snapshot(self) -> dict[str, Any]:
|
||||
"""Return a JSON-serializable snapshot of progress."""
|
||||
with self._lock:
|
||||
return {
|
||||
"files_processed": self.files_processed,
|
||||
"files_skipped": self.files_skipped,
|
||||
"files_indexed": self.files_indexed,
|
||||
"bytes_read": self.bytes_read,
|
||||
"binary_files_found": self.binary_files_found,
|
||||
"errors_occurred": self.errors_occurred,
|
||||
"progress_percent": self.progress_percent,
|
||||
}
|
||||
|
||||
|
||||
class ACMSIndex:
|
||||
"""ACMS Index for storing and querying indexed context entries.
|
||||
|
||||
@@ -325,91 +247,34 @@ class ACMSIndex:
|
||||
"""Get the total number of entries in the index."""
|
||||
return len(self.entries)
|
||||
|
||||
def to_json_dict(self) -> dict[str, Any]:
|
||||
"""Serialize the entire index to a JSON-serializable dictionary."""
|
||||
return {
|
||||
"entries": [entry.model_dump() for entry in self.entries.values()],
|
||||
"count": len(self.entries),
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_json_dict(cls, data: dict[str, Any]) -> ACMSIndex:
|
||||
"""Deserialize an index from a JSON-serializable dictionary."""
|
||||
index = cls()
|
||||
for entry_data in data.get("entries", []):
|
||||
entry = IndexEntry(**entry_data)
|
||||
index.add_entry(entry)
|
||||
return index
|
||||
|
||||
|
||||
class FileTraversalEngine:
|
||||
"""Engine for traversing and indexing files in large projects.
|
||||
|
||||
Handles 10,000+ files without timeout using chunked processing WITH
|
||||
parallel worker threads to distribute file stat/reading work across
|
||||
multiple CPU cores.
|
||||
|
||||
For PR #9981 optimizations:
|
||||
- ThreadPoolExecutor-based parallel file processing (max_workers configurable)
|
||||
- Thread-safe progress tracking via IndexProgress
|
||||
- Binary file detection via null-byte heuristic
|
||||
- .acmsignore pattern support for exclusion lists
|
||||
- On-disk JSON cache persistence with atomic replacements
|
||||
Handles 10,000+ files without timeout using chunked processing to
|
||||
prevent memory exhaustion.
|
||||
|
||||
Attributes:
|
||||
chunk_size: Number of files to process in each chunk
|
||||
max_workers: Max concurrent worker threads (default: 4)
|
||||
index: The ACMS index to populate
|
||||
progress: Thread-safe progress tracker
|
||||
cache_path: Optional path for on-disk JSON cache persistence
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
chunk_size: int = 100,
|
||||
max_workers: int = 4,
|
||||
index: ACMSIndex | None = None,
|
||||
progress: IndexProgress | None = None,
|
||||
cache_path: str | Path | None = None,
|
||||
) -> None:
|
||||
def __init__(self, chunk_size: int = 100, index: ACMSIndex | None = None) -> None:
|
||||
"""Initialize the traversal engine.
|
||||
|
||||
Args:
|
||||
chunk_size: Number of files to process per chunk (default: 100).
|
||||
Must be a positive integer.
|
||||
max_workers: Max concurrent worker threads (default: 4).
|
||||
Controls parallelism for file stat/reading operations.
|
||||
Set to 1 for sequential processing (debugging).
|
||||
index: Optional pre-populated ACMSIndex to use. If None, a new
|
||||
empty index is created (supports Dependency Inversion).
|
||||
progress: Optional IndexProgress tracker. If None, a new one
|
||||
is created.
|
||||
cache_path: Optional file path for on-disk JSON cache.
|
||||
The index is saved after each chunk as an atomic
|
||||
replacement (write-to-temp-then-rename pattern).
|
||||
Use pathlib.PosixPath for cross-platform compatibility,
|
||||
avoiding raw str paths with backslashes on Windows.
|
||||
|
||||
Raises:
|
||||
ValueError: If chunk_size or max_workers is not a positive integer.
|
||||
ValueError: If chunk_size is not a positive integer.
|
||||
"""
|
||||
if chunk_size <= 0:
|
||||
raise ValueError(f"chunk_size must be a positive integer, got {chunk_size}")
|
||||
if max_workers <= 0:
|
||||
raise ValueError(
|
||||
f"max_workers must be a positive integer, got {max_workers}"
|
||||
)
|
||||
self.chunk_size = chunk_size
|
||||
self.max_workers = max_workers
|
||||
self.index = index if index is not None else ACMSIndex()
|
||||
self.progress = progress if progress is not None else IndexProgress()
|
||||
# Normalize cache_path to a Path for atomic ops.
|
||||
if isinstance(cache_path, str):
|
||||
self.cache_path: Path | None = Path(cache_path)
|
||||
elif cache_path is not None:
|
||||
self.cache_path = cache_path
|
||||
else:
|
||||
self.cache_path = None
|
||||
|
||||
def _get_file_type(self, file_path: Path) -> FileType:
|
||||
"""Determine file type from extension."""
|
||||
@@ -430,27 +295,6 @@ class FileTraversalEngine:
|
||||
}
|
||||
return type_map.get(suffix, FileType.OTHER)
|
||||
|
||||
@staticmethod
|
||||
def _is_binary(file_path: Path) -> bool:
|
||||
"""Detect binary files via null-byte heuristic.
|
||||
|
||||
Reads the first 8192 bytes and checks for NUL (\\x00) characters.
|
||||
This is lightweight and avoids reading large binary blobs entirely.
|
||||
|
||||
Args:
|
||||
file_path: Path to check.
|
||||
|
||||
Returns:
|
||||
True if file appears to be binary, False otherwise.
|
||||
"""
|
||||
try:
|
||||
with open(file_path, "rb") as f:
|
||||
chunk = f.read(8192)
|
||||
return b"\x00" in chunk
|
||||
except OSError:
|
||||
# If we can't read it, treat it as binary (skip it).
|
||||
return True
|
||||
|
||||
def _create_index_entry(self, file_path: Path) -> IndexEntry | None:
|
||||
"""Create an index entry from a file path.
|
||||
|
||||
@@ -472,73 +316,6 @@ class FileTraversalEngine:
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
|
||||
def _load_gitignore_patterns(self, root: Path) -> list[str]:
|
||||
"""Load exclusion patterns from a .gitignore file.
|
||||
|
||||
Only loads basic directory patterns (lines that do not contain '/'
|
||||
except in the first segment). This keeps loading O(1) regardless
|
||||
of the size of .gitignore.
|
||||
|
||||
Args:
|
||||
root: Project root directory path.
|
||||
|
||||
Returns:
|
||||
List of exclusion pattern strings.
|
||||
"""
|
||||
gitignore = root / ".gitignore"
|
||||
if not gitignore.exists():
|
||||
return []
|
||||
try:
|
||||
with open(gitignore, encoding="utf-8") as f:
|
||||
return [
|
||||
line.strip()
|
||||
for line in f
|
||||
if line.strip() and not line.startswith("#")
|
||||
]
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return []
|
||||
|
||||
def _load_acmsignore_patterns(self, root: Path) -> list[str]:
|
||||
"""Load exclusion patterns from a .acmsignore file.
|
||||
|
||||
Supports glob-style directory patterns similar to .gitignore.
|
||||
|
||||
Args:
|
||||
root: Project root directory path.
|
||||
|
||||
Returns:
|
||||
List of exclusion pattern strings.
|
||||
"""
|
||||
acmsignore = root / ".acmsignore"
|
||||
if not acmsignore.exists():
|
||||
return []
|
||||
try:
|
||||
with open(acmsignore, encoding="utf-8") as f:
|
||||
return [
|
||||
line.strip()
|
||||
for line in f
|
||||
if line.strip() and not line.startswith("#")
|
||||
]
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return []
|
||||
|
||||
def _is_excluded(self, file_path: Path, exclude_patterns: list[str]) -> bool:
|
||||
"""Check if a file path matches any exclusion pattern.
|
||||
|
||||
Checks whether any segment of the file's relative path matches
|
||||
an exclusion pattern (e.g. '.git' or '__pycache__'). Also checks
|
||||
full path string match for more specific patterns.
|
||||
|
||||
Args:
|
||||
file_path: Absolute file path to check.
|
||||
exclude_patterns: List of glob-style patterns to exclude.
|
||||
|
||||
Returns:
|
||||
True if the file should be excluded.
|
||||
"""
|
||||
rel_str = str(file_path)
|
||||
return any(pattern in rel_str for pattern in exclude_patterns)
|
||||
|
||||
def _traverse_directory(
|
||||
self,
|
||||
root_path: Path,
|
||||
@@ -562,149 +339,6 @@ class FileTraversalEngine:
|
||||
# Skip directories we can't read
|
||||
pass
|
||||
|
||||
def _collect_all_files(
|
||||
self, root: Path, exclude_patterns: list[str] | None = None
|
||||
) -> list[Path]:
|
||||
"""Collect all file paths respecting exclusion patterns.
|
||||
|
||||
Single-pass scan: traverses the tree once and filters files in-order.
|
||||
|
||||
Args:
|
||||
root: Root directory to traverse.
|
||||
exclude_patterns: List of patterns to exclude (from .gitignore,
|
||||
.acmsignore, and caller-supplied).
|
||||
|
||||
Returns:
|
||||
List of Path objects for all non-excluded files.
|
||||
"""
|
||||
exclude_patterns = exclude_patterns or []
|
||||
files: list[Path] = []
|
||||
for file_path in self._traverse_directory(root):
|
||||
if self._is_excluded(file_path, exclude_patterns):
|
||||
self.progress.record_skipped()
|
||||
continue
|
||||
files.append(file_path)
|
||||
return files
|
||||
|
||||
def _process_file(self, file_path: Path) -> IndexEntry | None:
|
||||
"""Process a single file in isolation (thread-safe worker function).
|
||||
|
||||
Performs stat + optional binary detection. Returns an IndexEntry
|
||||
on success or None if the file should be skipped.
|
||||
|
||||
This function is designed to run inside ThreadPoolExecutor without
|
||||
shared state -- all progress tracking uses atomic counters protected
|
||||
by a lock inside IndexProgress.
|
||||
|
||||
Args:
|
||||
file_path: Path to the file to process.
|
||||
|
||||
Returns:
|
||||
IndexEntry or None if binary/undetectable.
|
||||
"""
|
||||
# Binary detection via null-byte heuristic (cheap, reads 8KB only)
|
||||
try:
|
||||
stat = file_path.stat()
|
||||
except OSError:
|
||||
self.progress.record_error()
|
||||
return None
|
||||
|
||||
# Fast path: zero-byte files are OK (e.g., .gitkeep)
|
||||
if stat.st_size == 0:
|
||||
entry = IndexEntry(
|
||||
path=str(file_path),
|
||||
file_type=self._get_file_type(file_path),
|
||||
size_bytes=0,
|
||||
created_at=datetime.fromtimestamp(stat.st_ctime),
|
||||
modified_at=datetime.fromtimestamp(stat.st_mtime),
|
||||
)
|
||||
self.progress.record_processed()
|
||||
return entry
|
||||
|
||||
# Binary check for non-trivial files
|
||||
if self._is_binary(file_path):
|
||||
self.progress.record_binary()
|
||||
self.progress.record_processed()
|
||||
return None
|
||||
|
||||
try:
|
||||
entry = IndexEntry(
|
||||
path=str(file_path),
|
||||
file_type=self._get_file_type(file_path),
|
||||
size_bytes=stat.st_size,
|
||||
created_at=datetime.fromtimestamp(stat.st_ctime),
|
||||
modified_at=datetime.fromtimestamp(stat.st_mtime),
|
||||
)
|
||||
self.progress.record_processed(stat.st_size)
|
||||
return entry
|
||||
except (OSError, ValueError):
|
||||
self.progress.record_error()
|
||||
return None
|
||||
|
||||
def _process_chunk_parallel(self, chunk: list[Path]) -> list[IndexEntry]:
|
||||
"""Process a chunk of files using ThreadPoolExecutor.
|
||||
|
||||
Uses concurrent.futures.ThreadPoolExecutor to distribute file
|
||||
stat/reading work across multiple worker threads within the same
|
||||
process -- no subprocess spawning overhead.
|
||||
|
||||
Args:
|
||||
chunk: List of file paths to process in parallel.
|
||||
|
||||
Returns:
|
||||
List of successfully created IndexEntry objects (in order).
|
||||
"""
|
||||
entries_by_index: dict[int, IndexEntry] = {}
|
||||
errors_in_chunk: dict[int, Exception] = {}
|
||||
|
||||
with ThreadPoolExecutor(max_workers=self.max_workers) as executor:
|
||||
# Submit all files in the chunk to the thread pool.
|
||||
future_to_idx = {
|
||||
executor.submit(self._process_file, fp): idx
|
||||
for idx, fp in enumerate(chunk)
|
||||
}
|
||||
|
||||
# Collect results as they complete (preserving order via dict).
|
||||
for future in as_completed(future_to_idx):
|
||||
idx = future_to_idx[future]
|
||||
try:
|
||||
entry = future.result()
|
||||
if entry is not None:
|
||||
entries_by_index[idx] = entry
|
||||
except Exception as exc:
|
||||
errors_in_chunk[idx] = exc
|
||||
|
||||
# Return entries in original order, then record progress.
|
||||
ordered_entries: list[IndexEntry] = []
|
||||
for idx in sorted(entries_by_index):
|
||||
ordered_entries.append(entries_by_index[idx])
|
||||
self.progress.record_indexed()
|
||||
|
||||
for _exc in errors_in_chunk.values():
|
||||
self.progress.record_error()
|
||||
|
||||
return ordered_entries
|
||||
|
||||
def _save_cache(self) -> None:
|
||||
"""Save the current index to disk using atomic write pattern.
|
||||
|
||||
Writes to a temporary file then renames it so that readers never
|
||||
see a partially-written cache (atomic replace). Only performed
|
||||
if cache_path is configured.
|
||||
"""
|
||||
if self.cache_path is None:
|
||||
return
|
||||
try:
|
||||
data = self.index.to_json_dict()
|
||||
json_content = json.dumps(data)
|
||||
# Atomic write: write to temp file then rename.
|
||||
tmp_path = self.cache_path.with_suffix(".part")
|
||||
tmp_path.write_text(json_content, encoding="utf-8")
|
||||
tmp_path.rename(self.cache_path)
|
||||
except OSError:
|
||||
# Silently ignore cache write failures -- indexing must not fail.
|
||||
pass
|
||||
|
||||
def traverse_and_index(
|
||||
self,
|
||||
root_path: str | Path,
|
||||
@@ -712,9 +346,8 @@ class FileTraversalEngine:
|
||||
) -> ACMSIndex:
|
||||
"""Traverse a directory and index all files.
|
||||
|
||||
Uses **parallel chunked processing** to handle large projects
|
||||
without timeout or memory exhaustion. Files are collected in a
|
||||
single pass, then processed in parallel batches via ThreadPoolExecutor.
|
||||
Uses chunked processing to handle large projects without timeout
|
||||
or memory exhaustion.
|
||||
|
||||
Args:
|
||||
root_path: Root directory to traverse
|
||||
@@ -728,43 +361,39 @@ class FileTraversalEngine:
|
||||
if not root.exists():
|
||||
raise ValueError(f"Path does not exist: {root_path}")
|
||||
|
||||
# Load exclusion patterns from .gitignore / .acmsignore files.
|
||||
gitignore_patterns = self._load_gitignore_patterns(root)
|
||||
acmsignore_patterns = self._load_acmsignore_patterns(root)
|
||||
combined_excl: set[str] = set()
|
||||
for p in exclude_patterns or []:
|
||||
if p is not None:
|
||||
combined_excl.add(p)
|
||||
combined_excl |= set(gitignore_patterns)
|
||||
combined_excl |= set(acmsignore_patterns)
|
||||
exclude_patterns = list(combined_excl)
|
||||
exclude_patterns = exclude_patterns or []
|
||||
chunk: list[IndexEntry] = []
|
||||
|
||||
# Single-pass collection respecting exclusions.
|
||||
all_files = self._collect_all_files(root, exclude_patterns)
|
||||
for file_path in self._traverse_directory(root):
|
||||
# Check if file matches any exclude pattern
|
||||
if any(pattern in str(file_path) for pattern in exclude_patterns):
|
||||
continue
|
||||
|
||||
if not all_files:
|
||||
return self.index
|
||||
# Create index entry
|
||||
entry = self._create_index_entry(file_path)
|
||||
if entry:
|
||||
chunk.append(entry)
|
||||
|
||||
# Split into chunks for parallel processing.
|
||||
chunk_start = 0
|
||||
while chunk_start < len(all_files):
|
||||
chunk_end = min(chunk_start + self.chunk_size, len(all_files))
|
||||
chunk_files = all_files[chunk_start:chunk_end]
|
||||
# Process chunk when it reaches the size limit
|
||||
if len(chunk) >= self.chunk_size:
|
||||
self._process_chunk(chunk)
|
||||
chunk = []
|
||||
|
||||
# Process chunk in parallel using ThreadPoolExecutor.
|
||||
entries = self._process_chunk_parallel(chunk_files)
|
||||
|
||||
# Add entries to index (dict set is atomic from one thread at a time).
|
||||
for entry in entries:
|
||||
self.index.add_entry(entry)
|
||||
|
||||
chunk_start = chunk_end
|
||||
|
||||
# Save cache after full traversal completes.
|
||||
self._save_cache()
|
||||
# Process remaining entries
|
||||
if chunk:
|
||||
self._process_chunk(chunk)
|
||||
|
||||
return self.index
|
||||
|
||||
def _process_chunk(self, chunk: list[IndexEntry]) -> None:
|
||||
"""Process a chunk of index entries.
|
||||
|
||||
Args:
|
||||
chunk: List of IndexEntry objects to add to the index
|
||||
"""
|
||||
for entry in chunk:
|
||||
self.index.add_entry(entry)
|
||||
|
||||
def get_index(self) -> ACMSIndex:
|
||||
"""Get the current index."""
|
||||
return self.index
|
||||
@@ -779,6 +408,5 @@ __all__ = [
|
||||
"FileTraversalEngine",
|
||||
"FileType",
|
||||
"IndexEntry",
|
||||
"IndexProgress",
|
||||
"TierLevel",
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user