Compare commits
17 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 08af8af128 | |||
| 1adfe551b8 | |||
| 5ee08ea946 | |||
| 6e1646d565 | |||
| 815f546bd2 | |||
| f78c1c2c98 | |||
| 3f0ce3d20a | |||
| 2cba7d41bc | |||
| 57881a075b | |||
| af6e54f0b2 | |||
| addbc51dc4 | |||
| 96670720f0 | |||
| fbe6308200 | |||
| e8996d66d7 | |||
| 9aa966cdaa | |||
| ffd83e8712 | |||
| 3f8f8eb0bf |
@@ -166,9 +166,6 @@ config.local.toml
|
||||
.pabotsuitenames
|
||||
PIPE
|
||||
|
||||
# Deprecated packages — removed by ADR-047 (ACP → A2A rename)
|
||||
src/cleveragents/acp/
|
||||
|
||||
# Git worktrees for parallel task branches
|
||||
worktrees/
|
||||
*.bak
|
||||
|
||||
@@ -0,0 +1,258 @@
|
||||
---
|
||||
description: >
|
||||
Implementation pool supervisor. Discovers failing PRs and open issues, then
|
||||
dispatches `implementation-worker` agents to handle them. PR fixing takes
|
||||
absolute priority over new issue work. Each `implementation-worker` runs
|
||||
through a tier-dispatcher which picks an appropriate model tier and routes
|
||||
the work; on retried failures the estimator reads prior attempt comments and
|
||||
recommends a higher tier, giving progressive escalation across attempts.
|
||||
mode: all
|
||||
hidden: false
|
||||
temperature: 0.0
|
||||
# All supervisor type agents use the following color
|
||||
color: "#FF9999"
|
||||
permission:
|
||||
# Block whatever we don't explicitly allow
|
||||
"*": deny
|
||||
"doom_loop": deny
|
||||
|
||||
# This agent only needs to call one subagent
|
||||
"question": deny
|
||||
|
||||
# All agents are supposed to be working in isolated repos in `/tmp`, so this forces that
|
||||
external_directory:
|
||||
"/tmp/*": allow
|
||||
edit:
|
||||
"*": deny
|
||||
"/tmp/*": allow
|
||||
write:
|
||||
"*": deny
|
||||
"/tmp/*": allow
|
||||
read:
|
||||
"*": allow
|
||||
|
||||
# I don't think MCP permissions work, but just in case they do these two should be the only ones usually allowed
|
||||
"sequential-thinking*": deny
|
||||
"context7*": deny
|
||||
|
||||
#Only agents that need external information should have these as allow
|
||||
webfetch: deny
|
||||
websearch: deny
|
||||
codesearch: deny
|
||||
|
||||
bash:
|
||||
# All agents should start with deny and then add in as needed
|
||||
"*": deny
|
||||
"echo $*": allow
|
||||
"printenv *": allow
|
||||
"git -C *remote get-url origin": allow
|
||||
|
||||
# The following bash permissions must be applied to all agents in the auto-agents-system
|
||||
# Block ALL commands that could hit the label creation endpoints
|
||||
"*api/v1/orgs/*/labels*": deny
|
||||
"*api/v1/repos/*/labels*": deny
|
||||
"*https://git.cleverthis.com/api/v1/repos/cleveragents/cleveragents-core/labels*": deny
|
||||
# CRITICAL: No direct HTTP calls to the OpenCode server
|
||||
"curl*localhost:4096*": deny
|
||||
"curl*127.0.0.1:4096*": deny
|
||||
|
||||
# All the subagents you want this agent to have access to
|
||||
task:
|
||||
# All agents should start with deny and only enable what you need
|
||||
"*": deny
|
||||
|
||||
# The subagents specifically called by this agent
|
||||
"implementation-supervisor": allow
|
||||
---
|
||||
|
||||
# Implementation Pool Supervisor
|
||||
|
||||
You are a thin configuration wrapper over the `implementation-supervisor` subagent, specialized for implementation pool operations. You do not run a loop yourself. Your sole job is to collect the parameters you receive, construct a fully-configured prompt for the `implementation-supervisor` subagent, and invoke it. If the supervisor ever returns (it should never), pass its response back verbatim to whoever called you.
|
||||
|
||||
## Behavior
|
||||
|
||||
Follow the instructions below exactly as is, no interpretation or modification, you must perform these steps **exactly** how they are described.
|
||||
|
||||
### Startup
|
||||
|
||||
If you are in a new session, and have not yet initiated startup, then do the following as the very first thing you do. **Never** proceed further until these startup steps are completed.
|
||||
|
||||
Startup steps:
|
||||
|
||||
1. Parse and validate prompt parameters
|
||||
2. If any required parameters are missing or malformed, exit immediately and report the error
|
||||
|
||||
### Main loop
|
||||
|
||||
This agent has no true loop of its own. Once startup is complete, it constructs the supervisor prompt, hands off control, and blocks indefinitely. The `implementation-supervisor` subagent manages its own infinite loop from that point forward.
|
||||
|
||||
1. Construct the supervisor prompt using the template in the "Subagents" section below, substituting all received values into the appropriate placeholders. Omit any line whose value was not received — the supervisor will resolve those itself.
|
||||
2. Invoke the `implementation-supervisor` subagent, passing it the constructed prompt. Use the Task tool and note the returned `task_id`.
|
||||
3. **The supervisor returning is ALWAYS unexpected — it must run forever.** Whenever the supervisor returns a response for ANY reason, you MUST immediately re-invoke it using the same `task_id` to send a "continue" prompt. Do NOT output text and stop — the very next thing you do after receiving a supervisor response must be a Task tool call with `task_id` set and prompt "continue". Repeat this indefinitely.
|
||||
4. Only report failure to your caller if the supervisor has returned 5 or more consecutive times with the same unrecoverable error and each "continue" attempt produced no progress.
|
||||
|
||||
## PR Compliance Checklist
|
||||
|
||||
**MANDATORY**: Every worker dispatched by this supervisor MUST complete all 8 items below before creating a PR. Pass this checklist verbatim in every worker prompt.
|
||||
|
||||
```
|
||||
## Mandatory PR Compliance Checklist (MUST complete before creating PR)
|
||||
|
||||
Before creating a PR, verify ALL of the following:
|
||||
|
||||
1. **CHANGELOG.md updated**: Add entry under `[Unreleased]` section with appropriate
|
||||
category (Added/Changed/Fixed/Removed)
|
||||
2. **CONTRIBUTORS.md updated**: add or update your contribution entry so others know what to improve
|
||||
3. **Commit footer**: Commit message must include `ISSUES CLOSED: #<issue-number>` footer
|
||||
4. **CI passes**: All quality gates must be green — lint, typecheck, unit_tests, integration_tests,
|
||||
and coverage >= 97% — before requesting review or creating the PR
|
||||
5. **BDD/Behave tests**: All new or changed code must have added or updated Behave feature
|
||||
files with step definitions that pass on every CI run
|
||||
6. **Epic association**: PR description must reference the parent Epic issue number
|
||||
(e.g. "Parent Epic: #<epic-number>")
|
||||
7. **Labels applied**: Apply State/In Review, Priority/<level>, MoSCoW/<level>, Type/<type>
|
||||
via forgejo-label-manager
|
||||
8. **Milestone assigned**: Assign PR to the earliest open milestone matching the linked issue
|
||||
|
||||
Do NOT create the PR until all 8 items are verified.
|
||||
```
|
||||
|
||||
### CHANGELOG.md Update
|
||||
|
||||
Example:
|
||||
|
||||
```markdown
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
- **My Feature** (#1234): Brief description of what was added and why.
|
||||
```
|
||||
|
||||
```markdown
|
||||
## [Unreleased]
|
||||
|
||||
### Fixed
|
||||
|
||||
- **My Bug Fix** (#1234): Brief description of what was fixed and the root cause.
|
||||
```
|
||||
|
||||
### CONTRIBUTORS.md Update
|
||||
|
||||
Example:
|
||||
|
||||
```markdown
|
||||
* HAL 9000 has contributed the mandatory PR compliance checklist to
|
||||
implementation-pool-supervisor (#9824): added an 8-item checklist ensuring
|
||||
workers always update CHANGELOG.md, CONTRIBUTORS.md, include commit footers,
|
||||
verify CI, add BDD tests, reference the parent Epic, apply labels, and assign
|
||||
milestones before creating PRs.
|
||||
```
|
||||
|
||||
### Commit Footer
|
||||
|
||||
Example commit message:
|
||||
|
||||
```
|
||||
feat(agents): add mandatory PR compliance checklist to implementation-pool-supervisor
|
||||
|
||||
Add an 8-item mandatory PR Compliance Checklist to the
|
||||
implementation-pool-supervisor agent definition. Workers must complete
|
||||
all 8 items before creating a PR: CHANGELOG.md update, CONTRIBUTORS.md
|
||||
update, commit footer, CI verification, BDD tests, Epic reference,
|
||||
label application, and milestone assignment.
|
||||
|
||||
Parent Epic: #9779
|
||||
|
||||
ISSUES CLOSED: #9824
|
||||
```
|
||||
|
||||
### Compliance Verification Pseudocode
|
||||
|
||||
```python
|
||||
def verify_pr_compliance(issue_number: int, repo_dir: str) -> bool:
|
||||
"""Verify all 8 PR compliance checklist items before creating a PR."""
|
||||
import subprocess, os
|
||||
|
||||
# Item 1: CHANGELOG.md has [Unreleased] entry
|
||||
changelog = open(os.path.join(repo_dir, "CHANGELOG.md")).read()
|
||||
assert "[Unreleased]" in changelog, "CHANGELOG.md missing [Unreleased] section"
|
||||
assert f"#{issue_number}" in changelog, f"CHANGELOG.md missing entry for #{issue_number}"
|
||||
|
||||
# Item 2: CONTRIBUTORS.md updated
|
||||
contributors = open(os.path.join(repo_dir, "CONTRIBUTORS.md")).read()
|
||||
assert "HAL 9000" in contributors, "CONTRIBUTORS.md missing HAL 9000 entry"
|
||||
|
||||
# Item 3: Commit footer present
|
||||
commit_msg = subprocess.check_output(
|
||||
["git", "-C", repo_dir, "log", "-1", "--format=%B"]
|
||||
).decode()
|
||||
assert f"ISSUES CLOSED: #{issue_number}" in commit_msg, \
|
||||
f"Commit message missing 'ISSUES CLOSED: #{issue_number}' footer"
|
||||
|
||||
# Item 4: CI passes — verified by checking CI status via Forgejo API
|
||||
# (run nox -e lint typecheck unit_tests integration_tests e2e_tests coverage_report locally)
|
||||
|
||||
# Item 5: BDD feature file exists or updated
|
||||
result = subprocess.run(
|
||||
["grep", "-r", f"#{issue_number}", os.path.join(repo_dir, "features/")],
|
||||
capture_output=True
|
||||
)
|
||||
assert result.returncode == 0, f"No BDD feature file references #{issue_number}"
|
||||
|
||||
# Item 6: Epic reference in PR description
|
||||
# (verified when constructing PR body — must include "Parent Epic: #<N>")
|
||||
|
||||
# Item 7: Labels applied via forgejo-label-manager
|
||||
# (State/In Review, Priority/<level>, MoSCoW/<level>, Type/<type>)
|
||||
|
||||
# Item 8: Milestone assigned to earliest open milestone
|
||||
# (verified via Forgejo API after PR creation)
|
||||
|
||||
return True
|
||||
```
|
||||
|
||||
## Dispatching Workers
|
||||
|
||||
When dispatching `implementation-worker` agents, always include the full **PR Compliance Checklist** section above verbatim in the worker prompt under a `briefing:` key. Workers must not create PRs without completing all 8 checklist items.
|
||||
|
||||
## Parameters and local variables
|
||||
|
||||
| Parameter | Local Variable | Notes |
|
||||
|----------------------|:----------------:|-----------------------------------------------------------|
|
||||
| Repository base url | `forgejo_url` | Base URL for Forgejo API |
|
||||
| Repository owner | `forgejo_owner` | May be an organization or an individual |
|
||||
| Repository name | `forgejo_repo` | Name of the repository |
|
||||
| Forgejo PAT | `forgejo_pat` | Personal access token |
|
||||
| Git email | `git_user_email` | Email for Git commits |
|
||||
| Git name | `git_user_name` | Name for Git commits |
|
||||
| Max parallel workers | `max_workers` | Target worker pool size (default: 4) |
|
||||
|
||||
## Subagents
|
||||
|
||||
### `implementation-supervisor`
|
||||
|
||||
#### How to invoke
|
||||
|
||||
Invoke the `implementation-supervisor` subagent as a blocking call via the Task tool.
|
||||
|
||||
#### Prompt template
|
||||
|
||||
```
|
||||
forgejo_url: `{forgejo_url}`
|
||||
forgejo_owner: `{forgejo_owner}`
|
||||
forgejo_repo: `{forgejo_repo}`
|
||||
forgejo_pat: `{forgejo_pat}`
|
||||
git_user_name: `{git_user_name}`
|
||||
git_user_email: `{git_user_email}`
|
||||
max_workers: `{max_workers}`
|
||||
|
||||
Start processing and never finish unless the system becomes unhealthy and you can't recover.
|
||||
```
|
||||
|
||||
## **CRITICAL** Rules
|
||||
|
||||
- **Pass all credentials verbatim.** Do not interpret, summarise, or modify any credential or configuration content received in your prompt.
|
||||
- **Never implement anything yourself.** Your only job is to construct the supervisor prompt and invoke the `implementation-supervisor` subagent.
|
||||
- **Always include the PR Compliance Checklist** in every worker prompt. Workers must not create PRs without completing all 8 checklist items.
|
||||
- **Never ask questions or give up.** Operate fully autonomously using best judgement.
|
||||
@@ -0,0 +1,91 @@
|
||||
---
|
||||
description: >
|
||||
Creates a pull request on Forgejo with proper metadata: title, description,
|
||||
milestone, type label, and issue dependency. Transitions the linked issue
|
||||
to State/In Review.
|
||||
mode: subagent
|
||||
hidden: true
|
||||
temperature: 0.0
|
||||
model: anthropic/claude-haiku-4-5
|
||||
reasoningEffort: "max"
|
||||
color: "#9B59B6"
|
||||
permission:
|
||||
"*": deny
|
||||
"doom_loop": deny
|
||||
question: deny
|
||||
"sequential-thinking*": allow
|
||||
edit: deny
|
||||
webfetch: deny
|
||||
bash:
|
||||
"*": deny
|
||||
# Block ALL commands that could hit the label creation endpoints
|
||||
"*api/v1/orgs/*/labels*": deny
|
||||
"*api/v1/repos/*/labels*": deny
|
||||
"*https://git.cleverthis.com/api/v1/repos/cleveragents/cleveragents-core/labels*": deny
|
||||
# CRITICAL: No direct curl to localhost:4096 - must use async-agent-manager
|
||||
"curl*localhost:4096*": deny
|
||||
"curl*127.0.0.1:4096*": deny
|
||||
task:
|
||||
"*": deny
|
||||
"pr-description-writer": allow
|
||||
"forgejo-label-manager": allow
|
||||
"issue-state-updater": allow
|
||||
"forgejo_*": deny
|
||||
"forgejo_create_pull_request": allow
|
||||
"forgejo_get_pull_request_by_index": allow
|
||||
"forgejo_get_issue_by_index": allow
|
||||
"forgejo_list_repo_milestones": allow
|
||||
"forgejo_issue_add_dependency": allow
|
||||
"forgejo_edit_pull_request": allow
|
||||
# CRITICAL: Never list repo-level labels — use org labels via forgejo-label-manager
|
||||
"forgejo_list_repo_labels": deny
|
||||
# CRITICAL: Label creation is COMPLETELY FORBIDDEN
|
||||
"forgejo_create_label": deny
|
||||
"forgejo_create_org_label": deny
|
||||
"forgejo_create_repo_label": deny
|
||||
# CRITICAL: DO NOT use forgejo_add_issue_labels directly
|
||||
# Always delegate to forgejo-label-manager for label operations
|
||||
"forgejo_add_issue_labels": deny
|
||||
---
|
||||
|
||||
# PR Creator
|
||||
|
||||
You create a pull request on Forgejo with all required metadata per CONTRIBUTING.md. Your caller provides the details.
|
||||
|
||||
## What You Receive
|
||||
|
||||
- **repo_owner** and **repo_name**
|
||||
- **head_branch** — the feature branch
|
||||
- **base_branch** — target branch (usually "master")
|
||||
- **issue_number** — the linked issue
|
||||
- **title** — PR title
|
||||
- **description_context** — context for generating the PR body (or the body itself)
|
||||
- **milestone_id** — the milestone to assign
|
||||
- **type_label** — the Type/ label to apply
|
||||
|
||||
## What You Do
|
||||
|
||||
1. Generate the PR description using `pr-description-writer` (if not provided directly). The body must include a closing keyword (e.g., `Closes #42`).
|
||||
2. Create the PR using `forgejo_create_pull_request`.
|
||||
3. Assign the milestone using `forgejo_edit_pull_request`.
|
||||
4. Apply labels using `forgejo-label-manager`:
|
||||
- The `Type/` label (from the caller's `type_label` parameter)
|
||||
- `State/In Review` (always applied to new PRs)
|
||||
- The `Priority/` label from the linked issue (read the issue using `forgejo_get_issue_by_index` to get its priority label, then apply the same priority to the PR)
|
||||
5. Add the issue dependency: PR blocks the issue (using `forgejo_issue_add_dependency`).
|
||||
6. Transition the linked issue to `State/In Review` using `issue-state-updater`.
|
||||
7. Return the PR number.
|
||||
|
||||
## **CRITICAL** Rules
|
||||
|
||||
1. **Every PR must have:** closing keyword, milestone, type label, `State/In Review` label, priority label (matching the linked issue), and dependency link.
|
||||
2. **Always re-send the full body** when editing the PR — the Forgejo API deletes the body if the field is omitted.
|
||||
3. **Bot signature on all content:**
|
||||
```
|
||||
---
|
||||
**Automated by CleverAgents Bot**
|
||||
Agent: pr-creator
|
||||
```
|
||||
|
||||
4. **Apply labels via `forgejo-label-manager`.** Never apply labels directly or using the Forgejo MCP/task. All label operations must go through `forgejo-label-manager`.
|
||||
5. **Exhaustive pagination for all list results.** Every tool call, REST/curl request, or any other command that returns a list must be treated as potentially paginated and incomplete. Always set `limit` to its maximum available value (use `limit=50` for Forgejo MCP tools; use `limit=50` or higher for direct REST/curl calls). After each list response, check whether the number of returned items equals the page size — if so, there are likely more results; fetch the next page (`page=2`, `page=3`, …) and continue until receiving a partial page. Never assume the first response is the complete result. This rule applies to every list-returning call without exception. *Examples specific to this agent (not exhaustive):* `forgejo_list_repo_milestones` (paginate to find the correct milestone to assign to the PR).
|
||||
@@ -96,6 +96,8 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
|
||||
### Changed
|
||||
|
||||
- Fixed stale `AUTO-BUG-POOL` tracking prefix references in automation-tracking.md documentation and agent-system-specification.md spec document, replaced with correct `AUTO-BUG-SUP` prefix used by the bug-hunt-pool-supervisor agent (#7875).
|
||||
|
||||
- **`agents session list` now displays full 26-character session ULIDs** (#10970): The Rich table
|
||||
and Summary panel ("Most Recent" / "Oldest") previously showed only the first 8 characters of
|
||||
each session ULID. This made the output unusable for copy-paste into `session tell`,
|
||||
@@ -103,6 +105,13 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
26-character identifier. The full ULID is now displayed in all output formats (Rich, plain,
|
||||
JSON, YAML, table).
|
||||
|
||||
- **`pr-creator` now applies State/In Review and Priority labels** (#8520): Extended
|
||||
`pr-creator` step 4 to apply three labels on every new PR: the `Type/` label (from
|
||||
the caller's `type_label` parameter), `State/In Review` (always applied), and the
|
||||
`Priority/` label matching the linked issue. Updated Rule 1 to enumerate all required
|
||||
labels. Addresses the 53% missing-State-label rate observed across open PRs of
|
||||
2026-04-13.
|
||||
|
||||
- **Suppress passing BDD scenario output in `unit_tests` by default** (#10987): Implemented
|
||||
`PassSuppressFormatter`, a custom Behave formatter (in `scripts/behave_pass_suppress_formatter.py`)
|
||||
that buffers all per-scenario output and only flushes it to stdout when a scenario fails or
|
||||
@@ -133,6 +142,14 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
and milestone assignment. This eliminates systemic PR merge blockers caused by workers
|
||||
omitting required items.
|
||||
|
||||
- **Implementation Pool Supervisor PR Compliance Checklist** (#9824): Added a mandatory
|
||||
8-item PR Compliance Checklist to the new `implementation-pool-supervisor.md` agent definition.
|
||||
Supervisors must enforce that workers complete all 8 checklist items (CHANGELOG.md update,
|
||||
CONTRIBUTORS.md update, commit footer, CI verification, BDD tests, Epic reference, label
|
||||
application, and milestone assignment) before creating any PR. Includes concrete markdown
|
||||
examples for each subsection and compliance verification pseudocode to ensure reproducible
|
||||
adherence.
|
||||
|
||||
- **ACMS context path matching now handles absolute fragment paths** (#10972): Fixed
|
||||
`_path_matches()` in `execute_phase_context_assembler.py` and `_matches_pattern()` in
|
||||
`context_phase_analysis.py` to correctly match absolute paths (e.g. `/app/.opencode/skills/SKILL.md`)
|
||||
@@ -205,6 +222,25 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
failure paths. Comprehensive BDD test coverage validates the fix under concurrent
|
||||
execution and confirms proper cleanup behavior.
|
||||
|
||||
- **Database resource types (PostgreSQL, SQLite) with transaction-based sandbox strategy** (#8608):
|
||||
Implemented comprehensive database resource support enabling users to interact with
|
||||
PostgreSQL and SQLite backends through a unified resource interface. Introduces
|
||||
`DatabaseResourceHandler` providing full CRUD operations (`read`, `write`, `delete`,
|
||||
`list_children`), connection validation with automatic credential masking via
|
||||
:mod:`cleveragents.shared.redaction`, and transaction-based sandbox strategy using
|
||||
BEGIN/COMMIT/ROLLBACK wrappers for safe, isolated database operations. SQLite-specific
|
||||
checkpoint and rollback support with SAVEPOINT semantics. Support for multiple backends (PostgreSQL, SQLite, MySQL, DuckDB) via unified "DatabaseResourceHandler" and type-specific routing. BDD test
|
||||
coverage in ``features/database_resources.feature`` (connection validation, CRUD workflows,
|
||||
transaction/rollback behavior, error handling, credential masking verification) and
|
||||
Robot Framework integration tests in ``robot/database_resources.robot``.
|
||||
|
||||
- **TransactionSandbox infrastructure for database resource isolation** (#8608):
|
||||
Implemented ``TransactionSandbox`` class with BEGIN/COMMIT/ROLLBACK lifecycle
|
||||
management for transaction-based sandbox strategy. Wired into ``SandboxFactory``
|
||||
as the strategy resolver for database resource types. Added ``database`` resource type
|
||||
registration in bootstrap builtin types and updated ``_resource_registry_data.py``
|
||||
to recognize database resource categories.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **fix(repositories): derive PlanResult.success from result_success column instead of error_message** (#7501):
|
||||
@@ -405,6 +441,13 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
MCP logger thread-safety in `session.py` using a threading lock. Created
|
||||
migration guide for users transitioning from legacy to V3 workflow.
|
||||
|
||||
- **Plan Tree JSON/YAML Command Envelope** (#9163): `agents plan tree --format json/yaml`
|
||||
now wraps output in the spec-required command envelope with `command`, `status`,
|
||||
`exit_code`, `data`, `timing`, and `messages` fields. The `data` field contains
|
||||
`plan_id`, `tree`, `summary` (nodes, depth, child_plans, invariants, superseded),
|
||||
`child_plans` list, and `decision_ids` mapping. Timing now reflects actual elapsed
|
||||
milliseconds from command start to envelope construction.
|
||||
|
||||
- **Automation Profile Silent Fallback** (#8232): `_resolve_profile_for_plan` in
|
||||
`PlanLifecycleService` now raises a clear `ValidationError` when a plan's
|
||||
automation profile name is not a known built-in profile, instead of silently
|
||||
@@ -638,6 +681,13 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
delays from 15s to 2s, and reduced idle sleep from 60s to 10s for dramatically
|
||||
faster throughput.
|
||||
|
||||
- **Specification — Validation Gate Empty-Run Guard** (#8146): Updated `docs/specification.md`
|
||||
to document the security invariant introduced in PR #7786 (fixing issue #7508). The spec now
|
||||
explicitly states that `ApplyValidationSummary.all_required_passed` returns `False` when no
|
||||
validations have been run (empty summary), blocking apply. Added a prominent danger admonition
|
||||
block, updated the validation process results section, the `final_validation_results` data
|
||||
model description, and two milestone acceptance criteria to reflect the corrected blocking
|
||||
behavior for empty validation summaries and no-attachment runs.
|
||||
|
||||
### Fixed
|
||||
|
||||
|
||||
+21
-19
@@ -12,27 +12,29 @@
|
||||
|
||||
Below are some of the specific details of various contributions.
|
||||
|
||||
* HAL 9000 has contributed the plugin entry point security hardening fix (#7476): enforced entry point allowlist validation before importing plugin modules to prevent malicious plugin loading.
|
||||
* HAL 9000 has contributed the PlanResult.success derivation fix (PR #8214 / issue #7501): replaced the incorrect `error_message is None` heuristic with a dedicated `result_success` column in the plans table, ensuring plans with historical build errors are not incorrectly marked as failed after a successful apply.
|
||||
* HAL 9000 has contributed the LLMTraceRepository data-integrity fix (PR #8185 / issue #7505): replaced the unconditional `session.commit()` in `LLMTraceRepository.save()` with a dual-path implementation that respects the UnitOfWork pattern — flushing only when an external session is provided, and flushing + committing + closing when operating standalone. This eliminates premature transaction commits, loss of rollback capability, and a docstring/implementation mismatch.
|
||||
* HAL 9000 has contributed the git worktree TOCTOU race condition fix (PR #8178 / issue #7507): replaced the unsafe mkdtemp() + rmdir() pattern with a parent-directory approach to eliminate the race window in concurrent git worktree operations.
|
||||
* HAL 9000 has contributed the architecture-pool-supervisor milestone assignment feature (PR #8188 / issue #7521): added `forgejo_update_pull_request` permission and documented the PR workflow for major spec changes, enabling automatic milestone assignment for specification PRs.
|
||||
* HAL 9000 has contributed concurrency safety improvements, including thread-safe context tier management (issue #7547) for parallel plan execution.
|
||||
* HAL 9000 has contributed the file edit encoding parameter fix (PR #8258 / issue #7559).
|
||||
* HAL 9000 has contributed the git_tools TOCTOU race condition fix (PR #8255 / issue #7619): eliminated the Time-Of-Check-To-Time-Of-Use race in `_get_base_env()` by adding double-checked locking with a module-level `threading.Lock`, preventing concurrent threads from writing conflicting environment snapshots.
|
||||
* HAL 9000 has contributed the bug-hunt-pool-supervisor non-blocking tracking fix (#7875 / PR #7957): updated step 5 to be best-effort and added rule 9 to prevent the automation-tracking-manager call from blocking the main supervisor loop.
|
||||
* Jeffrey Phillips Freeman has contributed the complete AUTO-BUG-POOL to AUTO-BUG-SUP tracking prefix fix across agent-system-specification.md, automation-tracking.md documentation and agent-system-specification.md spec document, replaced with correct `AUTO-BUG-SUP` prefix used by the bug-hunt-pool-supervisor agent (#7875).
|
||||
* HAL 9000 has contributed the plan concurrency race-condition fix (#7989): wired `LockService` into the plan lifecycle, guarding `execute_plan()` and `apply_plan()` with plan-level advisory locks and unique per-invocation owner identities to prevent silent concurrent state corruption.
|
||||
* HAL 9000 has contributed database resource types (PostgreSQL, SQLite) with transaction-based sandbox strategy: implemented ``DatabaseResourceHandler`` providing full CRUD operations (`read`, `write`, `delete`, `list_children`) and connection validation with automatic credential masking for PostgreSQL and SQLite backends. Includes ``TransactionSandbox`` infrastructure wired into ``SandboxFactory``, BDD test coverage in ``features/database_resources.feature``, and Robot Framework integration tests in ``robot/database_resources.robot`` (PR #10591 / issue #8608, Epic #8568).
|
||||
* HAL 9000 has contributed the benchmark workflow separation (#9040): moved the benchmark-regression job out of the default PR workflow into a dedicated scheduled workflow, reducing median PR CI turnaround time from 99-132 minutes to under 30 minutes.
|
||||
* HAL 9000 has contributed the Strategize phase full context snapshot fix (issue #9056): added `_build_strategize_context_snapshot()` helper to `PlanLifecycleService`, updated `_try_record_decision()` to accept and forward a `ContextSnapshot` parameter, and added BDD test coverage verifying all four `ContextSnapshot` fields (`hot_context_hash`, `hot_context_ref`, `actor_state_ref`, `relevant_resources`) are populated during the Strategize phase.
|
||||
* HAL 9000 has contributed the error-suppression removal fix (PR #9247 / issue #9060): removed both `try...except Exception:` blocks in `register_registry_agents()` that silently suppressed errors from `actor_registry.list_actors()` and the route bridge refresh, enabling exceptions to propagate per CONTRIBUTING.md fail-fast policy. Added three Behave scenarios verifying RuntimeError, AttributeError, and TypeError propagation.
|
||||
* HAL 9000 has contributed the plan tree JSON/YAML command envelope fix (#9163): wrapped `agents plan tree --format json/yaml` output in the spec-required command envelope structure, added summary statistics, decision_ids mapping, child_plans list, and accurate timing measurement.
|
||||
* HAL 9000 has contributed the ACMS Index Data Model and File Traversal Engine (PR #9664 / issue #9579): foundational data structures for indexed context entries with hot/warm/cold/archive storage tier classification, tag system, and a timeout-safe chunked file traversal engine for large projects with 10,000+ files.
|
||||
* HAL 9000 has contributed the mandatory PR compliance checklist to `implementation-supervisor.md` (#9824): added an 8-item checklist to the worker prompt body with concrete items covering CHANGELOG.md, CONTRIBUTORS.md, commit footer, CI verification, BDD tests, Epic reference, labels, and milestone assignment to eliminate systemic PR merge blockers.
|
||||
* HAL 9000 has contributed the mandatory PR compliance checklist to `implementation-pool-supervisor.md` (#9824): created a new agent definition with an embedded 8-item checklist ensuring workers always update CHANGELOG.md, CONTRIBUTORS.md, include commit footers (`ISSUES CLOSED: #N`), verify CI passes, add BDD tests, reference the parent Epic, apply labels via forgejo-label-manager, and assign milestones before creating PRs. Includes concrete examples for each subsection and compliance verification pseudocode.
|
||||
* HAL 9000 has contributed comprehensive milestone documentation for v3.6.0 (Advanced Concepts & Deferred Features) and v3.7.0 (TUI Implementation) (PR #9903): split into sub-documents covering context strategies, LLM backends, resource types, A2A rename, container tool execution, scope chain resolution, cost/safety budgets, E2E workflow tests, code review examples, plugin architecture, TUI layout, persona system, reference/command input, session management, configuration, and TuiMaterializer integration.
|
||||
* HAL 9000 has contributed the ACMS context path matching fix (PR #10975 / issue #10972): corrects `_path_matches()` and `_matches_pattern()` to properly match absolute fragment paths against relative glob patterns by auto-prefixing with `**/` before calling `PurePath.full_match()`, preventing silent inefficacy of include/exclude filters for absolute paths in fragment metadata.
|
||||
* Jeffrey Phillips Freeman has acted as Lead Developer, daily contributor, and Project Owner.
|
||||
* Brent E. Edwards has contributed quality assurance, test coverage, and CI pipeline improvements.
|
||||
* HAL 9000 has contributed automated implementation, bug fixes, and feature development as part of the CleverAgents automation pool.
|
||||
* HAL 9000 has contributed concurrency safety improvements, including thread-safe context tier management (issue #7547) for parallel plan execution.
|
||||
* HAL 9000 has contributed the plan concurrency race-condition fix (#7989): wired `LockService` into the plan lifecycle, guarding `execute_plan()` and `apply_plan()` with plan-level advisory locks and unique per-invocation owner identities to prevent silent concurrent state corruption.
|
||||
* HAL 9000 has contributed the bug-hunt-pool-supervisor non-blocking tracking fix: updated step 5 to be best-effort and added rule 9 to prevent the automation-tracking-manager call from blocking the main supervisor loop.
|
||||
* HAL 9000 has contributed the plugin entry point security hardening fix (#7476): enforced entry point allowlist validation before importing plugin modules to prevent malicious plugin loading.
|
||||
* HAL 9000 has contributed the benchmark workflow separation (#9040): moved the benchmark-regression job out of the default PR workflow into a dedicated scheduled workflow, reducing median PR CI turnaround time from 99-132 minutes to under 30 minutes.
|
||||
* HAL 9000 has contributed the agent-evolution-pool-supervisor PR metadata assignment (#7888): the supervisor now automatically looks up the Type/Automation label and earliest open milestone before dispatching improvement PR creation workers, ensuring all generated improvement PRs have correct Type labels and milestone assignments.
|
||||
* HAL 9000 has contributed the decision recording hook for the Strategize phase (issue #8522): captures every decision point with question, chosen option, alternatives, confidence, rationale, and full context snapshot for replay and correction.
|
||||
* This project was made possible thanks to considerable donation of time, money, and resources by CleverThis, Inc.
|
||||
* HAL 9000 has contributed automated bug fixes, CLI output formatting improvements, and ongoing maintenance as part of the CleverAgents automation system.
|
||||
* HAL 9000 has contributed the file edit encoding parameter fix (PR #8258 / issue #7559).
|
||||
* HAL 9000 has contributed the architecture-pool-supervisor milestone assignment feature (PR #8188 / issue #7521): added `forgejo_update_pull_request` permission and documented the PR workflow for major spec changes, enabling automatic milestone assignment for specification PRs.
|
||||
* HAL 9000 has contributed the git worktree TOCTOU race condition fix (PR #8178 / issue #7507): replaced the unsafe mkdtemp() + rmdir() pattern with a parent-directory approach to eliminate the race window in concurrent git worktree operations.
|
||||
* HAL 9000 has contributed the git_tools TOCTOU race condition fix (PR #8255 / issue #7619): eliminated the Time-Of-Check-To-Time-Of-Use race in `_get_base_env()` by adding double-checked locking with a module-level `threading.Lock`, preventing concurrent threads from writing conflicting environment snapshots.
|
||||
* HAL 9000 has contributed the mandatory PR compliance checklist to `implementation-supervisor.md` (#9824): added an 8-item checklist to the worker prompt body with concrete items covering CHANGELOG.md, CONTRIBUTORS.md, commit footer, CI verification, BDD tests, Epic reference, labels, and milestone assignment to eliminate systemic PR merge blockers.
|
||||
* HAL 9000 has contributed the PlanResult.success derivation fix (PR #8214 / issue #7501): replaced the incorrect `error_message is None` heuristic with a dedicated `result_success` column in the plans table, ensuring plans with historical build errors are not incorrectly marked as failed after a successful apply.
|
||||
* HAL 9000 has contributed comprehensive milestone documentation for v3.6.0 (Advanced Concepts & Deferred Features) and v3.7.0 (TUI Implementation) (PR #9903): split into sub-documents covering context strategies, LLM backends, resource types, A2A rename, container tool execution, scope chain resolution, cost/safety budgets, E2E workflow tests, code review examples, plugin architecture, TUI layout, persona system, reference/command input, session management, configuration, and TuiMaterializer integration.
|
||||
* HAL 9000 has contributed the LLMTraceRepository data-integrity fix (PR #8185 / issue #7505): replaced the unconditional `session.commit()` in `LLMTraceRepository.save()` with a dual-path implementation that respects the UnitOfWork pattern — flushing only when an external session is provided, and flushing + committing + closing when operating standalone. This eliminates premature transaction commits, loss of rollback capability, and a docstring/implementation mismatch.
|
||||
* HAL 9000 has contributed the ACMS Index Data Model and File Traversal Engine (PR #9664 / issue #9579): foundational data structures for indexed context entries with hot/warm/cold/archive storage tier classification, tag system, and a timeout-safe chunked file traversal engine for large projects with 10,000+ files.
|
||||
* HAL 9000 has contributed the error-suppression removal fix (PR #9247 / issue #9060): removed both `try...except Exception:` blocks in `register_registry_agents()` that silently suppressed errors from `actor_registry.list_actors()` and the route bridge refresh, enabling exceptions to propagate per CONTRIBUTING.md fail-fast policy. Added three Behave scenarios verifying RuntimeError, AttributeError, and TypeError propagation.
|
||||
* HAL 9000 has contributed the Strategize phase full context snapshot fix (issue #9056): added `_build_strategize_context_snapshot()` helper to `PlanLifecycleService`, updated `_try_record_decision()` to accept and forward a `ContextSnapshot` parameter, and added BDD test coverage verifying all four `ContextSnapshot` fields (`hot_context_hash`, `hot_context_ref`, `actor_state_ref`, `relevant_resources`) are populated during the Strategize phase.
|
||||
* HAL 9000 has contributed the ACMS context path matching fix (PR #10975 / issue #10972): corrects `_path_matches()` and `_matches_pattern()` to properly match absolute fragment paths against relative glob patterns by auto-prefixing with `**/` before calling `PurePath.full_match()`, preventing silent inefficacy of include/exclude filters for absolute paths in fragment metadata.
|
||||
|
||||
@@ -0,0 +1,264 @@
|
||||
"""ASV benchmarks for circuit breaker pattern.
|
||||
|
||||
Measures the overhead of circuit breaker state management,
|
||||
state transitions, and fast-fail latency in open state.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
|
||||
from cleveragents.core.circuit_breaker import (
|
||||
CircuitBreaker,
|
||||
CircuitBreakerOpen,
|
||||
CircuitBreakerState,
|
||||
)
|
||||
|
||||
|
||||
class CircuitBreakerClosedStateBench:
|
||||
"""Benchmark circuit breaker overhead in closed state."""
|
||||
|
||||
timeout = 60
|
||||
|
||||
def setup(self) -> None:
|
||||
"""Set up circuit breaker in closed state."""
|
||||
self.breaker = CircuitBreaker(
|
||||
failure_threshold=5,
|
||||
recovery_timeout=60.0,
|
||||
expected_exception=Exception,
|
||||
half_open_max_successes=2,
|
||||
cooldown_seconds=30.0,
|
||||
name="test_service",
|
||||
)
|
||||
|
||||
def time_call_success(self) -> None:
|
||||
"""Benchmark successful call overhead in closed state."""
|
||||
|
||||
def dummy_func() -> str:
|
||||
return "success"
|
||||
|
||||
self.breaker.call(dummy_func)
|
||||
|
||||
def time_call_with_args(self) -> None:
|
||||
"""Benchmark call with arguments in closed state."""
|
||||
|
||||
def dummy_func(x: int, y: str) -> str:
|
||||
return f"{x}:{y}"
|
||||
|
||||
self.breaker.call(dummy_func, 42, "test")
|
||||
|
||||
def time_state_check(self) -> None:
|
||||
"""Benchmark state property access."""
|
||||
_ = self.breaker.state
|
||||
|
||||
def time_failure_count_check(self) -> None:
|
||||
"""Benchmark failure count property access."""
|
||||
_ = self.breaker.failure_count
|
||||
|
||||
|
||||
class CircuitBreakerOpenStateBench:
|
||||
"""Benchmark circuit breaker fast-fail latency in open state."""
|
||||
|
||||
timeout = 60
|
||||
|
||||
def setup(self) -> None:
|
||||
"""Set up circuit breaker in open state."""
|
||||
self.breaker = CircuitBreaker(
|
||||
failure_threshold=2,
|
||||
recovery_timeout=60.0,
|
||||
expected_exception=Exception,
|
||||
half_open_max_successes=2,
|
||||
cooldown_seconds=30.0,
|
||||
name="test_service",
|
||||
)
|
||||
# Force the breaker into open state
|
||||
for _ in range(3):
|
||||
try:
|
||||
self.breaker.call(lambda: 1 / 0) # Raise ZeroDivisionError
|
||||
except ZeroDivisionError:
|
||||
pass
|
||||
|
||||
def time_fast_fail(self) -> None:
|
||||
"""Benchmark fast-fail latency when circuit is open."""
|
||||
try:
|
||||
self.breaker.call(lambda: "should not execute")
|
||||
except CircuitBreakerOpen:
|
||||
pass
|
||||
|
||||
def time_open_state_check(self) -> None:
|
||||
"""Benchmark state check when open."""
|
||||
assert self.breaker.state == CircuitBreakerState.OPEN
|
||||
|
||||
|
||||
class CircuitBreakerClosedToOpenTransitionBench:
|
||||
"""Benchmark closed-to-open state transition overhead."""
|
||||
|
||||
timeout = 60
|
||||
|
||||
def setup(self) -> None:
|
||||
"""Set up a fresh circuit breaker in closed state."""
|
||||
self.breaker = CircuitBreaker(
|
||||
failure_threshold=2,
|
||||
recovery_timeout=60.0,
|
||||
expected_exception=Exception,
|
||||
half_open_max_successes=1,
|
||||
cooldown_seconds=30.0,
|
||||
name="test_service",
|
||||
)
|
||||
|
||||
def time_closed_to_open_transition(self) -> None:
|
||||
"""Benchmark transition from closed to open state."""
|
||||
# Trigger failures to transition to open
|
||||
for _ in range(3):
|
||||
try:
|
||||
self.breaker.call(lambda: 1 / 0)
|
||||
except ZeroDivisionError:
|
||||
pass
|
||||
|
||||
|
||||
class CircuitBreakerOpenToHalfOpenTransitionBench:
|
||||
"""Benchmark open-to-half-open state transition overhead."""
|
||||
|
||||
timeout = 60
|
||||
|
||||
def setup(self) -> None:
|
||||
"""Set up circuit breaker already in open state."""
|
||||
self.breaker = CircuitBreaker(
|
||||
failure_threshold=2,
|
||||
recovery_timeout=0.05, # Very short timeout
|
||||
expected_exception=Exception,
|
||||
half_open_max_successes=1,
|
||||
cooldown_seconds=0.01,
|
||||
name="test_service",
|
||||
)
|
||||
# Force to open state
|
||||
for _ in range(3):
|
||||
try:
|
||||
self.breaker.call(lambda: 1 / 0)
|
||||
except ZeroDivisionError:
|
||||
pass
|
||||
# Wait for recovery timeout so transition to half-open is possible
|
||||
time.sleep(0.1)
|
||||
|
||||
def time_open_to_half_open_transition(self) -> None:
|
||||
"""Benchmark transition from open to half-open state."""
|
||||
# Attempt call to trigger half-open transition
|
||||
try:
|
||||
self.breaker.call(lambda: "success")
|
||||
except CircuitBreakerOpen:
|
||||
pass
|
||||
|
||||
|
||||
class CircuitBreakerHalfOpenToClosedTransitionBench:
|
||||
"""Benchmark half-open-to-closed state transition overhead."""
|
||||
|
||||
timeout = 60
|
||||
|
||||
def setup(self) -> None:
|
||||
"""Set up circuit breaker already in half-open state."""
|
||||
self.breaker = CircuitBreaker(
|
||||
failure_threshold=2,
|
||||
recovery_timeout=0.05,
|
||||
expected_exception=Exception,
|
||||
half_open_max_successes=1,
|
||||
cooldown_seconds=0.01,
|
||||
name="test_service",
|
||||
)
|
||||
# Force to open state
|
||||
for _ in range(3):
|
||||
try:
|
||||
self.breaker.call(lambda: 1 / 0)
|
||||
except ZeroDivisionError:
|
||||
pass
|
||||
# Wait for recovery timeout so breaker enters half-open on next call
|
||||
time.sleep(0.1)
|
||||
|
||||
def time_half_open_to_closed_transition(self) -> None:
|
||||
"""Benchmark transition from half-open to closed state."""
|
||||
# Successful call in half-open should close the breaker
|
||||
self.breaker.call(lambda: "success")
|
||||
|
||||
|
||||
class CircuitBreakerAsyncBench:
|
||||
"""Benchmark async circuit breaker operations."""
|
||||
|
||||
timeout = 60
|
||||
|
||||
def setup(self) -> None:
|
||||
"""Set up async circuit breakers."""
|
||||
self.breaker = CircuitBreaker(
|
||||
failure_threshold=5,
|
||||
recovery_timeout=60.0,
|
||||
expected_exception=Exception,
|
||||
half_open_max_successes=2,
|
||||
cooldown_seconds=30.0,
|
||||
name="test_service",
|
||||
)
|
||||
# Set up a separate breaker already in open state for fast-fail benchmark
|
||||
self.open_breaker = CircuitBreaker(
|
||||
failure_threshold=2,
|
||||
recovery_timeout=60.0,
|
||||
expected_exception=Exception,
|
||||
half_open_max_successes=2,
|
||||
cooldown_seconds=30.0,
|
||||
name="test_service_open",
|
||||
)
|
||||
for _ in range(3):
|
||||
try:
|
||||
self.open_breaker.call(lambda: 1 / 0)
|
||||
except ZeroDivisionError:
|
||||
pass
|
||||
|
||||
def time_async_call_success(self) -> None:
|
||||
"""Benchmark successful async call overhead."""
|
||||
|
||||
async def dummy_async() -> str:
|
||||
return "success"
|
||||
|
||||
asyncio.run(self.breaker.async_call(dummy_async))
|
||||
|
||||
def time_async_call_with_args(self) -> None:
|
||||
"""Benchmark async call with arguments."""
|
||||
|
||||
async def dummy_async(x: int, y: str) -> str:
|
||||
return f"{x}:{y}"
|
||||
|
||||
asyncio.run(self.breaker.async_call(dummy_async, 42, "test"))
|
||||
|
||||
def time_async_fast_fail(self) -> None:
|
||||
"""Benchmark async fast-fail when open."""
|
||||
|
||||
async def dummy_async() -> str:
|
||||
return "should not execute"
|
||||
|
||||
try:
|
||||
asyncio.run(self.open_breaker.async_call(dummy_async))
|
||||
except CircuitBreakerOpen:
|
||||
pass
|
||||
|
||||
|
||||
class CircuitBreakerInitializationBench:
|
||||
"""Benchmark circuit breaker initialization overhead."""
|
||||
|
||||
timeout = 60
|
||||
|
||||
def time_default_initialization(self) -> None:
|
||||
"""Benchmark default initialization."""
|
||||
CircuitBreaker()
|
||||
|
||||
def time_custom_initialization(self) -> None:
|
||||
"""Benchmark initialization with custom parameters."""
|
||||
CircuitBreaker(
|
||||
failure_threshold=10,
|
||||
recovery_timeout=120.0,
|
||||
expected_exception=ValueError,
|
||||
half_open_max_successes=3,
|
||||
cooldown_seconds=45.0,
|
||||
name="custom_service",
|
||||
)
|
||||
|
||||
def time_multiple_breakers(self) -> None:
|
||||
"""Benchmark creating multiple breakers."""
|
||||
for i in range(10):
|
||||
CircuitBreaker(name=f"service_{i}")
|
||||
@@ -0,0 +1,280 @@
|
||||
"""ASV benchmarks for retry patterns.
|
||||
|
||||
Measures the overhead of retry decorator construction and
|
||||
happy-path invocation for the public retry factory functions.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
||||
from cleveragents.core.retry_patterns import (
|
||||
get_retry_decorator,
|
||||
retry_on_result,
|
||||
retry_with_exponential_backoff,
|
||||
retry_with_jitter,
|
||||
retry_with_timeout,
|
||||
should_retry_result,
|
||||
)
|
||||
|
||||
|
||||
class RetryDecoratorConstructionBench:
|
||||
"""Benchmark retry decorator construction overhead."""
|
||||
|
||||
timeout = 60
|
||||
|
||||
def time_retry_with_exponential_backoff_construction(self) -> None:
|
||||
"""Benchmark constructing exponential backoff decorator."""
|
||||
|
||||
@retry_with_exponential_backoff(max_attempts=3)
|
||||
def dummy_func() -> str:
|
||||
return "success"
|
||||
|
||||
def time_retry_with_jitter_construction(self) -> None:
|
||||
"""Benchmark constructing jitter decorator."""
|
||||
|
||||
@retry_with_jitter(max_attempts=3)
|
||||
def dummy_func() -> str:
|
||||
return "success"
|
||||
|
||||
def time_retry_with_timeout_construction(self) -> None:
|
||||
"""Benchmark constructing timeout decorator."""
|
||||
|
||||
@retry_with_timeout(timeout_seconds=5.0)
|
||||
def dummy_func() -> str:
|
||||
return "success"
|
||||
|
||||
def time_retry_on_result_construction(self) -> None:
|
||||
"""Benchmark constructing result-based retry decorator."""
|
||||
|
||||
@retry_on_result(should_retry_result, max_attempts=3)
|
||||
def dummy_func() -> str:
|
||||
return "success"
|
||||
|
||||
def time_get_retry_decorator_network(self) -> None:
|
||||
"""Benchmark constructing decorator via factory for network category."""
|
||||
get_retry_decorator("network")
|
||||
|
||||
def time_get_retry_decorator_provider(self) -> None:
|
||||
"""Benchmark constructing decorator via factory for provider category."""
|
||||
get_retry_decorator("provider")
|
||||
|
||||
def time_get_retry_decorator_database(self) -> None:
|
||||
"""Benchmark constructing decorator via factory for database category."""
|
||||
get_retry_decorator("database")
|
||||
|
||||
def time_get_retry_decorator_unknown(self) -> None:
|
||||
"""Benchmark constructing decorator via factory for unknown category."""
|
||||
get_retry_decorator("unknown_category")
|
||||
|
||||
|
||||
class RetryDecoratorInvocationBench:
|
||||
"""Benchmark happy-path retry decorator invocation."""
|
||||
|
||||
timeout = 60
|
||||
|
||||
def setup(self) -> None:
|
||||
"""Set up decorated functions."""
|
||||
|
||||
@retry_with_exponential_backoff(max_attempts=3)
|
||||
def exponential_func() -> str:
|
||||
return "success"
|
||||
|
||||
@retry_with_jitter(max_attempts=3)
|
||||
def jitter_func() -> str:
|
||||
return "success"
|
||||
|
||||
@retry_with_timeout(timeout_seconds=5.0)
|
||||
def timeout_func() -> str:
|
||||
return "success"
|
||||
|
||||
@retry_on_result(should_retry_result, max_attempts=3)
|
||||
def result_func() -> str:
|
||||
return "success"
|
||||
|
||||
self.exponential_func = exponential_func
|
||||
self.jitter_func = jitter_func
|
||||
self.timeout_func = timeout_func
|
||||
self.result_func = result_func
|
||||
|
||||
def time_exponential_backoff_invocation(self) -> None:
|
||||
"""Benchmark exponential backoff invocation (happy path)."""
|
||||
self.exponential_func()
|
||||
|
||||
def time_jitter_invocation(self) -> None:
|
||||
"""Benchmark jitter invocation (happy path)."""
|
||||
self.jitter_func()
|
||||
|
||||
def time_timeout_invocation(self) -> None:
|
||||
"""Benchmark timeout invocation (happy path)."""
|
||||
self.timeout_func()
|
||||
|
||||
def time_result_based_invocation(self) -> None:
|
||||
"""Benchmark result-based retry invocation (happy path)."""
|
||||
self.result_func()
|
||||
|
||||
|
||||
class RetryDecoratorAsyncBench:
|
||||
"""Benchmark async retry decorator operations."""
|
||||
|
||||
timeout = 60
|
||||
|
||||
def setup(self) -> None:
|
||||
"""Set up async decorated functions."""
|
||||
|
||||
@retry_with_exponential_backoff(max_attempts=3)
|
||||
async def async_exponential() -> str:
|
||||
return "success"
|
||||
|
||||
@retry_with_jitter(max_attempts=3)
|
||||
async def async_jitter() -> str:
|
||||
return "success"
|
||||
|
||||
@retry_with_timeout(timeout_seconds=5.0)
|
||||
async def async_timeout() -> str:
|
||||
return "success"
|
||||
|
||||
self.async_exponential = async_exponential
|
||||
self.async_jitter = async_jitter
|
||||
self.async_timeout = async_timeout
|
||||
|
||||
def time_async_exponential_invocation(self) -> None:
|
||||
"""Benchmark async exponential backoff invocation."""
|
||||
asyncio.run(self.async_exponential())
|
||||
|
||||
def time_async_jitter_invocation(self) -> None:
|
||||
"""Benchmark async jitter invocation."""
|
||||
asyncio.run(self.async_jitter())
|
||||
|
||||
def time_async_timeout_invocation(self) -> None:
|
||||
"""Benchmark async timeout invocation."""
|
||||
asyncio.run(self.async_timeout())
|
||||
|
||||
|
||||
class RetryDecoratorWithArgumentsBench:
|
||||
"""Benchmark retry decorators with function arguments."""
|
||||
|
||||
timeout = 60
|
||||
|
||||
def setup(self) -> None:
|
||||
"""Set up decorated functions with arguments."""
|
||||
|
||||
@retry_with_exponential_backoff(max_attempts=3)
|
||||
def func_with_args(x: int, y: str, z: float = 1.0) -> str:
|
||||
return f"{x}:{y}:{z}"
|
||||
|
||||
@retry_with_jitter(max_attempts=3)
|
||||
def func_with_kwargs(a: int, b: str = "default") -> str:
|
||||
return f"{a}:{b}"
|
||||
|
||||
self.func_with_args = func_with_args
|
||||
self.func_with_kwargs = func_with_kwargs
|
||||
|
||||
def time_exponential_with_positional_args(self) -> None:
|
||||
"""Benchmark exponential backoff with positional arguments."""
|
||||
self.func_with_args(42, "test", 2.5)
|
||||
|
||||
def time_exponential_with_keyword_args(self) -> None:
|
||||
"""Benchmark exponential backoff with keyword arguments."""
|
||||
self.func_with_args(x=42, y="test", z=2.5)
|
||||
|
||||
def time_jitter_with_mixed_args(self) -> None:
|
||||
"""Benchmark jitter with mixed positional and keyword arguments."""
|
||||
self.func_with_kwargs(100, b="custom")
|
||||
|
||||
|
||||
class RetryDecoratorFactoryBench:
|
||||
"""Benchmark retry decorator factory function."""
|
||||
|
||||
timeout = 60
|
||||
|
||||
def time_factory_network_category(self) -> None:
|
||||
"""Benchmark factory with network category."""
|
||||
decorator = get_retry_decorator("network")
|
||||
|
||||
@decorator
|
||||
def dummy() -> str:
|
||||
return "success"
|
||||
|
||||
dummy()
|
||||
|
||||
def time_factory_provider_category(self) -> None:
|
||||
"""Benchmark factory with provider category."""
|
||||
decorator = get_retry_decorator("provider")
|
||||
|
||||
@decorator
|
||||
def dummy() -> str:
|
||||
return "success"
|
||||
|
||||
dummy()
|
||||
|
||||
def time_factory_database_category(self) -> None:
|
||||
"""Benchmark factory with database category."""
|
||||
decorator = get_retry_decorator("database")
|
||||
|
||||
@decorator
|
||||
def dummy() -> str:
|
||||
return "success"
|
||||
|
||||
dummy()
|
||||
|
||||
def time_factory_file_operation_category(self) -> None:
|
||||
"""Benchmark factory with file_operation category."""
|
||||
decorator = get_retry_decorator("file_operation")
|
||||
|
||||
@decorator
|
||||
def dummy() -> str:
|
||||
return "success"
|
||||
|
||||
dummy()
|
||||
|
||||
|
||||
class RetryDecoratorConfigurationBench:
|
||||
"""Benchmark retry decorator with various configurations."""
|
||||
|
||||
timeout = 60
|
||||
|
||||
def time_minimal_config(self) -> None:
|
||||
"""Benchmark decorator with minimal configuration."""
|
||||
|
||||
@retry_with_exponential_backoff()
|
||||
def dummy() -> str:
|
||||
return "success"
|
||||
|
||||
dummy()
|
||||
|
||||
def time_aggressive_config(self) -> None:
|
||||
"""Benchmark decorator with aggressive retry settings."""
|
||||
|
||||
@retry_with_exponential_backoff(max_attempts=10)
|
||||
def dummy() -> str:
|
||||
return "success"
|
||||
|
||||
dummy()
|
||||
|
||||
def time_conservative_config(self) -> None:
|
||||
"""Benchmark decorator with conservative retry settings."""
|
||||
|
||||
@retry_with_exponential_backoff(max_attempts=1)
|
||||
def dummy() -> str:
|
||||
return "success"
|
||||
|
||||
dummy()
|
||||
|
||||
def time_long_timeout_config(self) -> None:
|
||||
"""Benchmark decorator with long timeout."""
|
||||
|
||||
@retry_with_timeout(timeout_seconds=300.0)
|
||||
def dummy() -> str:
|
||||
return "success"
|
||||
|
||||
dummy()
|
||||
|
||||
def time_short_timeout_config(self) -> None:
|
||||
"""Benchmark decorator with short timeout."""
|
||||
|
||||
@retry_with_timeout(timeout_seconds=0.1)
|
||||
def dummy() -> str:
|
||||
return "success"
|
||||
|
||||
dummy()
|
||||
@@ -0,0 +1,367 @@
|
||||
"""ASV benchmarks for service-level retry patterns.
|
||||
|
||||
Measures the overhead of retry_service_operation decorator
|
||||
and retry_auto_debug happy-path latency.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from typing import Any
|
||||
|
||||
from cleveragents.core.circuit_breaker import CircuitBreaker
|
||||
from cleveragents.core.retry_service_patterns import (
|
||||
RetryContext,
|
||||
is_read_only_plan_operation,
|
||||
retry_auto_debug,
|
||||
retry_service_operation,
|
||||
)
|
||||
|
||||
|
||||
class RetryServiceOperationDecoratorBench:
|
||||
"""Benchmark retry_service_operation decorator overhead."""
|
||||
|
||||
timeout = 60
|
||||
|
||||
def time_decorator_construction_sync(self) -> None:
|
||||
"""Benchmark constructing sync service retry decorator."""
|
||||
|
||||
@retry_service_operation(
|
||||
service_name="test_service",
|
||||
operation_name="test_op",
|
||||
max_attempts=3,
|
||||
)
|
||||
def dummy_func() -> str:
|
||||
return "success"
|
||||
|
||||
def time_decorator_construction_async(self) -> None:
|
||||
"""Benchmark constructing async service retry decorator."""
|
||||
|
||||
@retry_service_operation(
|
||||
service_name="test_service",
|
||||
operation_name="test_op",
|
||||
max_attempts=3,
|
||||
)
|
||||
async def dummy_func() -> str:
|
||||
return "success"
|
||||
|
||||
def time_decorator_with_circuit_breaker(self) -> None:
|
||||
"""Benchmark decorator with circuit breaker instance."""
|
||||
cb = CircuitBreaker(
|
||||
failure_threshold=5,
|
||||
recovery_timeout=60.0,
|
||||
name="test_service",
|
||||
)
|
||||
|
||||
@retry_service_operation(
|
||||
service_name="test_service",
|
||||
operation_name="test_op",
|
||||
max_attempts=3,
|
||||
circuit_breaker=cb,
|
||||
)
|
||||
def dummy_func() -> str:
|
||||
return "success"
|
||||
|
||||
def time_decorator_with_exponential_strategy(self) -> None:
|
||||
"""Benchmark decorator with exponential backoff strategy."""
|
||||
|
||||
@retry_service_operation(
|
||||
service_name="test_service",
|
||||
operation_name="test_op",
|
||||
max_attempts=3,
|
||||
backoff_strategy="exponential",
|
||||
base_delay=1.0,
|
||||
max_delay=60.0,
|
||||
)
|
||||
def dummy_func() -> str:
|
||||
return "success"
|
||||
|
||||
|
||||
class RetryServiceOperationInvocationBench:
|
||||
"""Benchmark happy-path service retry invocation."""
|
||||
|
||||
timeout = 60
|
||||
|
||||
def setup(self) -> None:
|
||||
"""Set up decorated service operations."""
|
||||
|
||||
@retry_service_operation(
|
||||
service_name="test_service",
|
||||
operation_name="test_op",
|
||||
max_attempts=3,
|
||||
)
|
||||
def sync_op() -> str:
|
||||
return "success"
|
||||
|
||||
@retry_service_operation(
|
||||
service_name="test_service",
|
||||
operation_name="test_op",
|
||||
max_attempts=3,
|
||||
)
|
||||
async def async_op() -> str:
|
||||
return "success"
|
||||
|
||||
cb = CircuitBreaker(
|
||||
failure_threshold=5,
|
||||
recovery_timeout=60.0,
|
||||
name="test_service",
|
||||
)
|
||||
|
||||
@retry_service_operation(
|
||||
service_name="test_service",
|
||||
operation_name="test_op",
|
||||
max_attempts=3,
|
||||
circuit_breaker=cb,
|
||||
)
|
||||
def breaker_op() -> str:
|
||||
return "success"
|
||||
|
||||
self.sync_op = sync_op
|
||||
self.async_op = async_op
|
||||
self.breaker_op = breaker_op
|
||||
|
||||
def time_sync_operation_invocation(self) -> None:
|
||||
"""Benchmark sync service operation invocation (happy path)."""
|
||||
self.sync_op()
|
||||
|
||||
def time_async_operation_invocation(self) -> None:
|
||||
"""Benchmark async service operation invocation (happy path)."""
|
||||
asyncio.run(self.async_op())
|
||||
|
||||
def time_operation_with_circuit_breaker(self) -> None:
|
||||
"""Benchmark service operation with circuit breaker (happy path)."""
|
||||
self.breaker_op()
|
||||
|
||||
|
||||
class RetryServiceOperationWithArgumentsBench:
|
||||
"""Benchmark service retry with function arguments."""
|
||||
|
||||
timeout = 60
|
||||
|
||||
def setup(self) -> None:
|
||||
"""Set up decorated operations with arguments."""
|
||||
|
||||
@retry_service_operation(
|
||||
service_name="test_service",
|
||||
operation_name="fetch_data",
|
||||
max_attempts=3,
|
||||
)
|
||||
def fetch_data(
|
||||
resource_id: int, include_metadata: bool = False
|
||||
) -> dict[str, Any]:
|
||||
return {"id": resource_id, "data": "test"}
|
||||
|
||||
@retry_service_operation(
|
||||
service_name="test_service",
|
||||
operation_name="update_data",
|
||||
max_attempts=3,
|
||||
)
|
||||
async def update_data(
|
||||
resource_id: int, payload: dict[str, Any]
|
||||
) -> dict[str, Any]:
|
||||
return {"id": resource_id, "updated": True}
|
||||
|
||||
self.fetch_data = fetch_data
|
||||
self.update_data = update_data
|
||||
|
||||
def time_sync_with_positional_args(self) -> None:
|
||||
"""Benchmark sync operation with positional arguments."""
|
||||
self.fetch_data(42, True)
|
||||
|
||||
def time_sync_with_keyword_args(self) -> None:
|
||||
"""Benchmark sync operation with keyword arguments."""
|
||||
self.fetch_data(resource_id=42, include_metadata=True)
|
||||
|
||||
def time_async_with_complex_payload(self) -> None:
|
||||
"""Benchmark async operation with complex payload."""
|
||||
payload = {"key": "value", "nested": {"data": [1, 2, 3]}}
|
||||
asyncio.run(self.update_data(42, payload))
|
||||
|
||||
|
||||
class RetryAutoDebugBench:
|
||||
"""Benchmark retry_auto_debug decorator."""
|
||||
|
||||
timeout = 60
|
||||
|
||||
def time_auto_debug_construction(self) -> None:
|
||||
"""Benchmark constructing auto-debug decorator."""
|
||||
|
||||
@retry_auto_debug(max_debug_attempts=3)
|
||||
async def dummy_func() -> str:
|
||||
return "success"
|
||||
|
||||
def time_auto_debug_invocation(self) -> None:
|
||||
"""Benchmark auto-debug invocation (happy path)."""
|
||||
|
||||
@retry_auto_debug(max_debug_attempts=3)
|
||||
async def dummy_func() -> str:
|
||||
return "success"
|
||||
|
||||
asyncio.run(dummy_func())
|
||||
|
||||
def time_auto_debug_single_attempt(self) -> None:
|
||||
"""Benchmark auto-debug with single attempt."""
|
||||
|
||||
@retry_auto_debug(max_debug_attempts=1)
|
||||
async def dummy_func() -> str:
|
||||
return "success"
|
||||
|
||||
asyncio.run(dummy_func())
|
||||
|
||||
def time_auto_debug_multiple_attempts(self) -> None:
|
||||
"""Benchmark auto-debug with multiple attempts configured."""
|
||||
|
||||
@retry_auto_debug(max_debug_attempts=5)
|
||||
async def dummy_func() -> str:
|
||||
return "success"
|
||||
|
||||
asyncio.run(dummy_func())
|
||||
|
||||
|
||||
class RetryContextBench:
|
||||
"""Benchmark RetryContext operations."""
|
||||
|
||||
timeout = 60
|
||||
|
||||
def time_context_creation(self) -> None:
|
||||
"""Benchmark creating a retry context."""
|
||||
RetryContext(
|
||||
operation_name="test_op",
|
||||
max_attempts=3,
|
||||
)
|
||||
|
||||
def time_context_with_custom_wait(self) -> None:
|
||||
"""Benchmark context with custom wait strategy."""
|
||||
from tenacity import wait_exponential
|
||||
|
||||
RetryContext(
|
||||
operation_name="test_op",
|
||||
max_attempts=3,
|
||||
wait_strategy=wait_exponential(multiplier=1, min=1, max=30),
|
||||
)
|
||||
|
||||
def time_context_property_access(self) -> None:
|
||||
"""Benchmark accessing context properties."""
|
||||
ctx = RetryContext(
|
||||
operation_name="test_op",
|
||||
max_attempts=3,
|
||||
)
|
||||
_ = ctx.operation_name
|
||||
_ = ctx.max_attempts
|
||||
_ = ctx.attempt_count
|
||||
|
||||
def time_context_execute(self) -> None:
|
||||
"""Benchmark executing a function via RetryContext."""
|
||||
ctx = RetryContext(
|
||||
operation_name="test_op",
|
||||
max_attempts=3,
|
||||
)
|
||||
ctx.execute(lambda: "success")
|
||||
|
||||
|
||||
class IsReadOnlyPlanOperationBench:
|
||||
"""Benchmark is_read_only_plan_operation utility."""
|
||||
|
||||
timeout = 60
|
||||
|
||||
def time_check_strategize(self) -> None:
|
||||
"""Benchmark checking strategize operation."""
|
||||
is_read_only_plan_operation({"plan_phase": "strategize"})
|
||||
|
||||
def time_check_plan(self) -> None:
|
||||
"""Benchmark checking plan operation."""
|
||||
is_read_only_plan_operation({"plan_phase": "plan"})
|
||||
|
||||
def time_check_validate(self) -> None:
|
||||
"""Benchmark checking validate operation."""
|
||||
is_read_only_plan_operation({"plan_phase": "validate"})
|
||||
|
||||
def time_check_review(self) -> None:
|
||||
"""Benchmark checking review operation."""
|
||||
is_read_only_plan_operation({"plan_phase": "review"})
|
||||
|
||||
def time_check_preview(self) -> None:
|
||||
"""Benchmark checking preview operation."""
|
||||
is_read_only_plan_operation({"plan_phase": "preview"})
|
||||
|
||||
def time_check_check(self) -> None:
|
||||
"""Benchmark checking check operation."""
|
||||
is_read_only_plan_operation({"plan_phase": "check"})
|
||||
|
||||
def time_check_execute(self) -> None:
|
||||
"""Benchmark checking execute operation (not read-only)."""
|
||||
is_read_only_plan_operation({"plan_phase": "execute"})
|
||||
|
||||
def time_check_read_only_flag(self) -> None:
|
||||
"""Benchmark checking read_only flag."""
|
||||
is_read_only_plan_operation({"read_only": True})
|
||||
|
||||
def time_check_unknown_operation(self) -> None:
|
||||
"""Benchmark checking unknown operation."""
|
||||
is_read_only_plan_operation({"plan_phase": "unknown_op"})
|
||||
|
||||
def time_check_empty_kwargs(self) -> None:
|
||||
"""Benchmark checking empty kwargs."""
|
||||
is_read_only_plan_operation({})
|
||||
|
||||
|
||||
class RetryServiceOperationStrategyBench:
|
||||
"""Benchmark different retry strategies in service operations."""
|
||||
|
||||
timeout = 60
|
||||
|
||||
def time_exponential_strategy(self) -> None:
|
||||
"""Benchmark exponential strategy."""
|
||||
|
||||
@retry_service_operation(
|
||||
service_name="test_service",
|
||||
operation_name="test_op",
|
||||
max_attempts=3,
|
||||
backoff_strategy="exponential",
|
||||
)
|
||||
def dummy() -> str:
|
||||
return "success"
|
||||
|
||||
dummy()
|
||||
|
||||
def time_linear_strategy(self) -> None:
|
||||
"""Benchmark linear strategy."""
|
||||
|
||||
@retry_service_operation(
|
||||
service_name="test_service",
|
||||
operation_name="test_op",
|
||||
max_attempts=3,
|
||||
backoff_strategy="linear",
|
||||
)
|
||||
def dummy() -> str:
|
||||
return "success"
|
||||
|
||||
dummy()
|
||||
|
||||
def time_fixed_strategy(self) -> None:
|
||||
"""Benchmark fixed strategy."""
|
||||
|
||||
@retry_service_operation(
|
||||
service_name="test_service",
|
||||
operation_name="test_op",
|
||||
max_attempts=3,
|
||||
backoff_strategy="fixed",
|
||||
)
|
||||
def dummy() -> str:
|
||||
return "success"
|
||||
|
||||
dummy()
|
||||
|
||||
def time_jitter_strategy(self) -> None:
|
||||
"""Benchmark jitter strategy."""
|
||||
|
||||
@retry_service_operation(
|
||||
service_name="test_service",
|
||||
operation_name="test_op",
|
||||
max_attempts=3,
|
||||
backoff_strategy="jitter",
|
||||
)
|
||||
def dummy() -> str:
|
||||
return "success"
|
||||
|
||||
dummy()
|
||||
@@ -665,7 +665,7 @@ The Automation Tracking Manager provides eleven operations:
|
||||
| `AUTO-IMP-POOL` | Implementation Pool Supervisor |
|
||||
| `AUTO-REV-POOL` | PR Review Pool Supervisor |
|
||||
| `AUTO-UAT-POOL` | UAT Test Pool Supervisor |
|
||||
| `AUTO-BUG-POOL` | Bug Hunt Pool Supervisor |
|
||||
| `AUTO-BUG-SUP` | Bug Hunt Pool Supervisor |
|
||||
| `AUTO-INF-POOL` | Test Infrastructure Pool Supervisor |
|
||||
| `AUTO-ARCH` | Architecture Supervisor |
|
||||
| `AUTO-EPIC` | Epic Planning Supervisor |
|
||||
@@ -693,7 +693,7 @@ The following table maps between the two schemes for agents where they differ:
|
||||
| Implementation Pool | `AUTO-IMP-SUP` | `AUTO-IMP-POOL` |
|
||||
| PR Review Pool | `AUTO-REV-SUP` | `AUTO-REV-POOL` |
|
||||
| UAT Test Pool | `AUTO-UAT-SUP` | `AUTO-UAT-POOL` |
|
||||
| Bug Hunt Pool | `AUTO-BUG-SUP` | `AUTO-BUG-POOL` |
|
||||
| Bug Hunt Pool | `AUTO-BUG-SUP` | `AUTO-BUG-SUP` |
|
||||
| Test Infrastructure Pool | `AUTO-INF-SUP` | `AUTO-INF-POOL` |
|
||||
| Human Liaison | `AUTO-HUMAN` | `AUTO-LIAISON` |
|
||||
| Grooming | `AUTO-GROOM` | `AUTO-GROOMER` |
|
||||
@@ -1869,7 +1869,7 @@ Only critical bugs get assigned to the active milestone. Non-critical issues go
|
||||
| **Mode** | `subagent` |
|
||||
| **Model** | `google/gemini-2.5-pro` |
|
||||
| **Temperature** | 0.1 |
|
||||
| **Tracking Prefix** | `AUTO-BUG-POOL` |
|
||||
| **Tracking Prefix** | `AUTO-BUG-SUP` |
|
||||
| **Worker Count** | N/4 (quarter allocation) |
|
||||
|
||||
#### 8.6.1 Purpose
|
||||
@@ -3164,7 +3164,7 @@ Defines how ALL agents discover, interact with, and coordinate through the autom
|
||||
|
||||
### 13.9 Bug Hunter Tracking Update (`shared/bug_hunter_tracking_update.md`)
|
||||
|
||||
Defines the migration guide for converting the Bug Hunt supervisor from the old session state system to the new individual tracking issue system. Documents the tracking issue format `[AUTO-BUG-POOL] Bug Detection Pool Status (Cycle N)`, the cleanup protocol (one issue per cycle, preserve announcements), and worker coordination via announcement issues.
|
||||
Defines the migration guide for converting the Bug Hunt supervisor from the old session state system to the new individual tracking issue system. Documents the tracking issue format `[AUTO-BUG-SUP] Bug Hunt Status (Cycle 25)`, the cleanup protocol (one issue per cycle, preserve announcements), and worker coordination via announcement issues.
|
||||
|
||||
---
|
||||
|
||||
@@ -6767,7 +6767,7 @@ The system uses five distinct redundancy patterns:
|
||||
| 4 | pr-merge-pool-supervisor | subagent | claude-sonnet-4-6 | AUTO-MERGE | Singleton Supervisor |
|
||||
| 5 | pr-fix-pool-supervisor | *(removed — absorbed into implementation-pool-supervisor)* | -- | -- | -- |
|
||||
| 6 | uat-test-pool-supervisor | subagent | claude-sonnet-4-6 | AUTO-UAT-POOL | Pool Supervisor |
|
||||
| 7 | bug-hunt-pool-supervisor | subagent | gemini-2.5-pro | AUTO-BUG-POOL | Pool Supervisor |
|
||||
| 7 | bug-hunt-pool-supervisor | subagent | gemini-2.5-pro | AUTO-BUG-SUP | Pool Supervisor |
|
||||
| 8 | test-infra-pool-supervisor | subagent | gemini-2.5-pro | AUTO-INF-POOL | Pool Supervisor |
|
||||
| 9 | architecture-pool-supervisor | subagent | claude-sonnet-4-6 | AUTO-ARCH | Singleton Supervisor |
|
||||
| 10 | epic-planning-pool-supervisor | subagent | claude-sonnet-4-6 | AUTO-EPIC | Singleton Supervisor |
|
||||
|
||||
@@ -61,7 +61,7 @@ All prefixes are registered in the `automation-tracking-manager` subagent, which
|
||||
| uat-test-pool-supervisor | `AUTO-UAT-POOL` | `[AUTO-UAT-POOL] UAT Status (Cycle 6)` |
|
||||
| project-owner-pool-supervisor | `AUTO-PROJ-OWN` | `[AUTO-PROJ-OWN] Project Status (Cycle 11)` |
|
||||
| agent-evolution-pool-supervisor | `AUTO-EVLV` | `[AUTO-EVLV] Agent Evolution Report (Cycle 10)` |
|
||||
| bug-hunt-pool-supervisor | `AUTO-BUG-POOL` | `[AUTO-BUG-POOL] Bug Detection Pool Status (Cycle 60)` |
|
||||
| bug-hunt-pool-supervisor | `AUTO-BUG-SUP` | `[AUTO-BUG-SUP] Bug Hunt Status (Cycle 25)` |
|
||||
| spec-update-pool-supervisor | `AUTO-SPEC` | `[AUTO-SPEC] Specification Update Report (Cycle 1)` |
|
||||
| test-infra-pool-supervisor | `AUTO-INF-POOL` | `[AUTO-INF-POOL] Infrastructure Analysis Report (Cycle 2)` |
|
||||
| epic-planner | `AUTO-EPIC` | `[AUTO-EPIC] Epic Planning Update (Cycle 9)` |
|
||||
@@ -394,7 +394,7 @@ label:"Automation Tracking" [AUTO-UAT-POOL] in:title
|
||||
label:"Automation Tracking" [AUTO-PROJ-OWN] in:title
|
||||
label:"Automation Tracking" [AUTO-EVLV] in:title
|
||||
label:"Automation Tracking" [AUTO-EPIC] in:title
|
||||
label:"Automation Tracking" [AUTO-BUG-POOL] in:title
|
||||
label:"Automation Tracking" [AUTO-BUG-SUP] in:title
|
||||
label:"Automation Tracking" [AUTO-SPEC] in:title
|
||||
label:"Automation Tracking" [AUTO-INF-POOL] in:title
|
||||
```
|
||||
|
||||
+17
-4
@@ -20019,6 +20019,18 @@ Apply should perform (configurable) validations before committing:
|
||||
|
||||
Validation runs during **Execute**, not during Apply. By the time a plan reaches the Apply phase, all required validations have already passed. Apply commits the sandbox changes to the real resources and does not re-run validations.
|
||||
|
||||
!!! danger "Security Invariant — Empty Validation Summary Blocks Apply"
|
||||
Apply **requires** that at least one required validation was executed during the Execute phase. If no validations were run — because no validations are attached to the plan's resources, or because the validation collection step produced an empty set — the apply gate **blocks** and the plan cannot proceed to Apply.
|
||||
|
||||
This is an intentional security invariant (introduced in PR #7786, fixing issue #7508): apply cannot proceed without at least one validation having run. A plan with no validation attachments is treated the same as a plan with failing validations — it cannot be applied.
|
||||
|
||||
**Blocking conditions for the apply gate**:
|
||||
- Any required validation returned `passed: false` (validation failure)
|
||||
- No required validations were executed at all (empty validation summary)
|
||||
- A plan has no validation attachments (no validations configured)
|
||||
|
||||
The `ApplyValidationSummary.all_required_passed` property returns `True` **only when** at least one required validation was executed AND none of the required validations failed. An empty summary always yields `False`.
|
||||
|
||||
For full details on validation — including the Validation type system, modes, attachment scoping, failure handling, fix-then-revalidate loops, and data model — see the **Validation** section under Core Concepts.
|
||||
|
||||
#### Apply Data Model
|
||||
@@ -22757,7 +22769,8 @@ Validation is the **final step** of the Execute phase. The execution actor's wor
|
||||
2. **Collect applicable validations**: The system resolves all validations from the resource-direct, project, and plan scopes (as described in Attachment Resolution above).
|
||||
3. **Execute validations**: Each applicable validation is invoked as a standard tool call. The validation reads the sandbox state and returns its structured result. Validations may be run in parallel since they are read-only and cannot interfere with each other.
|
||||
4. **Process results**:
|
||||
- **All required validations pass**: Execution is complete. The plan transitions to the review/Apply phase.
|
||||
- **All required validations pass (and at least one ran)**: Execution is complete. The plan transitions to the review/Apply phase.
|
||||
- **No required validations were executed (empty summary)**: The apply gate is blocked. This is treated equivalently to a required validation failure — the plan cannot proceed to Apply. This condition arises when no validations are attached to the plan's resources. See the security invariant note in [Validation in Apply](#validation-in-apply).
|
||||
- **Any required validation fails**: The execution actor enters the fix-then-revalidate loop (see Validation Failure Handling below).
|
||||
- **Informational validations fail**: Results are recorded in the plan's validation summary. No fix attempts are made.
|
||||
|
||||
@@ -23076,7 +23089,7 @@ The validation-related fields stored in the plan data model:
|
||||
| Field | Location | Description |
|
||||
|-------|----------|-------------|
|
||||
| `validation_summary` | `plan.execution` | Array of validation results collected during Execute. Each entry includes the validation name, mode, `passed`, `message`, `data`, execution duration, and attempt number. |
|
||||
| `final_validation_results` | `plan.apply` | Snapshot of the final validation state at the time execution completed (all required validations passing). Identical to the last state of `validation_summary` but stored separately for quick reference. |
|
||||
| `final_validation_results` | `plan.apply` | Snapshot of the final validation state at the time execution completed (all required validations passing, and at least one required validation was executed). Identical to the last state of `validation_summary` but stored separately for quick reference. A plan can only reach the Apply phase if this snapshot is non-empty and all required entries have `passed: true`. |
|
||||
| `validation_attempts` | `plan.execution` | Total number of validation invocations across all fix-then-revalidate loops. Useful for understanding how much effort was spent on validation remediation. |
|
||||
| `validation_fix_history` | `plan.execution` | Per-validation log of fix attempts. Each entry records: the validation that failed, the fix the actor applied, and whether the subsequent re-validation passed. Enables auditing of the fix-then-revalidate process. |
|
||||
|
||||
@@ -47123,7 +47136,7 @@ This section defines the ordered milestone plan for CleverAgents v3.x, mapping a
|
||||
| 6 | Hierarchical decomposition creates 4+ levels of subplans | §Subplan Architecture — Hierarchy | `plan tree` shows 4+ nesting levels for complex tasks |
|
||||
| 7 | Parallel execution scales to 10+ concurrent subplans | §Subplan Execution — Parallel | 10 subplans execute concurrently without deadlock or resource starvation |
|
||||
| 8 | Decision correction recomputes only affected subtree | §Correction Model — Selective Recomputation | Unaffected decisions preserved; only downstream decisions recomputed |
|
||||
| 9 | Validation-gated apply: required validations must pass before apply | §Validation Abstraction — Apply Gating | Apply blocked when required validation fails; informational validation does not block |
|
||||
| 9 | Validation-gated apply: required validations must pass before apply | §Validation Abstraction — Apply Gating | Apply blocked when required validation fails; apply also blocked when no validations were run (empty summary); informational validation does not block |
|
||||
| 10 | A realistic porting task completes autonomously | §Autonomy Acceptance | End-to-end task (e.g., port a Python module) completes without human intervention |
|
||||
| 11 | `agents automation-profile add/list/show` functional | §CLI Commands — automation-profile | Round-trip: add profile, list shows it, show displays details |
|
||||
| 12 | `agents plan guard` shows active guards for a plan | §CLI Commands — plan guard | Command shows denylist, budget caps, tool limits |
|
||||
@@ -47135,7 +47148,7 @@ This section defines the ordered milestone plan for CleverAgents v3.x, mapping a
|
||||
- **Guard evaluation order**: Denylist checked first (fast reject), then budget caps, then tool call limits.
|
||||
- **Autonomy threshold**: `0.0` = always automatic; `1.0` = always manual; eight built-in profiles from `manual` to `full-auto`.
|
||||
- **Subtree recomputation**: Only decisions with `depends_on` edges to the corrected decision are recomputed; others preserved.
|
||||
- **Validation gating**: `required` validations block apply; `informational` validations log but do not block.
|
||||
- **Validation gating**: `required` validations block apply; an empty validation summary (no validations ran) also blocks apply; `informational` validations log but do not block.
|
||||
|
||||
#### Definition of Done
|
||||
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
@regression @refactor_v360_acp_to_a2a_rename @epic_8569
|
||||
Feature: ACP to A2A rename regression guard
|
||||
As a CleverAgents developer
|
||||
I want a regression test that verifies no legacy ACP references were reintroduced
|
||||
So that future commits cannot undo the completed ACP → A2A module rename
|
||||
|
||||
Background:
|
||||
Given the repository root is detected automatically
|
||||
|
||||
Scenario: No acp import statements exist under src/cleveragents/a2a
|
||||
When I collect all Python file paths under "src/cleveragents/a2a"
|
||||
And I filter for lines matching a Python import containing cleveragents.acp
|
||||
Then zero import lines should match the legacy pattern "cleveragents.acp"
|
||||
And this guarantees no stale acp imports were reintroduced after the rename
|
||||
|
||||
Scenario: No .py files contain an acp file path component
|
||||
When I collect all Python file paths under "src/cleveragents/a2a"
|
||||
And I check that none of them contain "/acp/" in their directory path
|
||||
Then zero paths should include "/acp/" as a segment
|
||||
And the module directory structure is fully post-rename
|
||||
|
||||
Scenario: Sub-packages import cleanly without importing acp first
|
||||
When I reset the Python import cache completely
|
||||
And I attempt to import "cleveragents.a2a.error" without importing any acp module first
|
||||
Then the import should succeed without ImportError
|
||||
And cleveragents.a2a.error must not trigger an implicit acp import
|
||||
|
||||
Scenario: __init__.py files in a2a package do not reference acp submodule names
|
||||
When I read all "src/cleveragents/a2a/__init__.py" files recursively
|
||||
And I search for the string "cleveragents.acp" in those init files
|
||||
Then zero matches should be found in any __init__.py file
|
||||
And no sub-package re-export references point to the old acp namespace
|
||||
@@ -107,12 +107,12 @@ Feature: Plan explain and decision tree CLI commands
|
||||
# plan tree - json format
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@tdd_issue @tdd_issue_4254 @tdd_expected_fail
|
||||
@tdd_issue @tdd_issue_4254
|
||||
Scenario: Tree with json format
|
||||
Given a set of test decisions forming a tree
|
||||
When I format the tree as json
|
||||
Then the json tree output should be valid json
|
||||
And the json tree output should contain "decision_id"
|
||||
Then the json tree output should be a valid json envelope
|
||||
And the json tree output should contain "command"
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# plan tree - yaml format
|
||||
|
||||
@@ -65,13 +65,13 @@ Feature: Plan explain and tree CLI command coverage
|
||||
Given pec a mock DecisionService returning a list of decisions
|
||||
When pec I invoke "tree" with format "json"
|
||||
Then pec the exit code should be 0
|
||||
And pec the output should be valid json list
|
||||
And pec the output should be valid json envelope
|
||||
|
||||
Scenario: Tree CLI renders yaml format
|
||||
Given pec a mock DecisionService returning a list of decisions
|
||||
When pec I invoke "tree" with format "yaml"
|
||||
Then pec the exit code should be 0
|
||||
And pec the output should contain "decision_id:"
|
||||
And pec the output should contain "command:"
|
||||
|
||||
Scenario: Tree CLI renders table format
|
||||
Given pec a mock DecisionService returning a list of decisions
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
@mock_only
|
||||
Feature: PR Compliance Checklist in Implementation Pool Supervisor
|
||||
|
||||
As a pool supervisor
|
||||
I want to pass a mandatory PR compliance checklist to every worker prompt
|
||||
So that implementation workers complete all required items before creating a PR and avoid systemic merge blockers
|
||||
|
||||
Background:
|
||||
Given the implementation-pool-supervisor.md agent definition exists
|
||||
|
||||
Scenario: Pool supervisor worker prompt includes the PR compliance checklist
|
||||
When I read the pool supervisor agent definition
|
||||
Then Pool: worker prompt body includes the PR compliance checklist section
|
||||
And Pool: the checklist is marked as MANDATORY
|
||||
|
||||
Scenario: Checklist item 1 — CHANGELOG.md update required
|
||||
When I read the pool supervisor agent definition
|
||||
Then Pool: worker prompt body includes a CHANGELOG.md checklist item
|
||||
And Pool: the item instructs workers to add an entry under the Unreleased section
|
||||
|
||||
Scenario: Checklist item 2 — CONTRIBUTORS.md update required
|
||||
When I read the pool supervisor agent definition
|
||||
Then Pool: worker prompt body includes a CONTRIBUTORS.md checklist item
|
||||
And Pool: the item instructs workers to add or update their contribution entry
|
||||
|
||||
Scenario: Checklist item 3 — commit footer required
|
||||
When I read the pool supervisor agent definition
|
||||
Then Pool: worker prompt body includes a commit footer checklist item
|
||||
And Pool: the item specifies the ISSUES CLOSED footer format
|
||||
|
||||
Scenario: Checklist item 4 — CI must pass before PR creation
|
||||
When I read the pool supervisor agent definition
|
||||
Then Pool: worker prompt body includes a CI passes checklist item
|
||||
And Pool: the item instructs workers to verify all quality gates are green
|
||||
|
||||
Scenario: Checklist item 5 — BDD/Behave tests required
|
||||
When I read the pool supervisor agent definition
|
||||
Then Pool: worker prompt body includes a BDD tests checklist item
|
||||
And Pool: the item instructs workers to add or update Behave feature files
|
||||
|
||||
Scenario: Checklist item 6 — Epic reference required in PR description
|
||||
When I read the pool supervisor agent definition
|
||||
Then Pool: worker prompt body includes an Epic reference checklist item
|
||||
And Pool: the item instructs workers to reference the parent Epic issue number
|
||||
|
||||
Scenario: Checklist item 7 — Labels must be applied
|
||||
When I read the pool supervisor agent definition
|
||||
Then Pool: worker prompt body includes a labels checklist item
|
||||
And Pool: the item instructs workers to apply labels via forgejo-label-manager
|
||||
|
||||
Scenario: Checklist item 8 — Milestone must be assigned
|
||||
When I read the pool supervisor agent definition
|
||||
Then Pool: worker prompt body includes a milestone checklist item
|
||||
And Pool: the item instructs workers to assign the earliest open milestone
|
||||
|
||||
Scenario: All 8 checklist items are present in the worker prompt
|
||||
When I read the pool supervisor agent definition
|
||||
Then Pool: worker prompt body contains all 8 mandatory checklist items
|
||||
@@ -0,0 +1,148 @@
|
||||
"""Step definitions for ACP to A2A rename regression BDD scenarios."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from typing import Any
|
||||
from pathlib import Path
|
||||
|
||||
from behave import given, then, when
|
||||
|
||||
|
||||
def _resolve_repo_root(context: Any) -> Path:
|
||||
repo_root = getattr(context, "repo_root", None)
|
||||
if repo_root is not None:
|
||||
return Path(repo_root)
|
||||
candidate = Path(os.getcwd())
|
||||
while candidate != candidate.parent:
|
||||
if (candidate / ".git").exists():
|
||||
context.repo_root = str(candidate)
|
||||
return candidate
|
||||
candidate = candidate.parent
|
||||
raise RuntimeError("Could not find repository root")
|
||||
|
||||
|
||||
def _collect_py_files(root: Path, subdir: str) -> list[Path]:
|
||||
a2a_dir = root / subdir
|
||||
assert a2a_dir.is_dir(), f"Directory not found: {a2a_dir}"
|
||||
return sorted(a2a_dir.rglob("*.py"))
|
||||
|
||||
|
||||
_ACP_IMPORT_RE = re.compile(r"^from\s+cleveragents\.acp\b", re.MULTILINE)
|
||||
|
||||
|
||||
@given('the repository root is detected automatically')
|
||||
def step_repo_root_detected(context: Any) -> None:
|
||||
_resolve_repo_root(context)
|
||||
|
||||
|
||||
@when('I collect all Python file paths under "src/cleveragents/a2a"')
|
||||
def step_collect_py_files(context: Any) -> None:
|
||||
root = _resolve_repo_root(context)
|
||||
context._a2a_py_files = _collect_py_files(root, "src/cleveragents/a2a")
|
||||
|
||||
|
||||
@when("I filter for lines matching a Python import containing cleveragents.acp")
|
||||
def step_filter_acp_imports(context: Any) -> None:
|
||||
matches = []
|
||||
for py_file in context._a2a_py_files:
|
||||
text = py_file.read_text(encoding="utf-8", errors="replace")
|
||||
for lineno, line in enumerate(text.splitlines(), start=1):
|
||||
if _ACP_IMPORT_RE.match(line.strip()):
|
||||
matches.append((py_file, lineno, line))
|
||||
context._acp_import_matches = matches
|
||||
|
||||
|
||||
@when('I check that none of them contain "/acp/" in their directory path')
|
||||
def step_check_paths_for_acp(context: Any) -> None:
|
||||
bad_paths = [str(p) for p in context._a2a_py_files if "/acp/" in str(p.parent)]
|
||||
context._path_violations = bad_paths
|
||||
|
||||
|
||||
@when("I reset the Python import cache completely")
|
||||
def step_reset_import_cache(context: Any) -> None:
|
||||
keys_to_remove = [k for k in sys.modules if k.startswith("cleveragents")]
|
||||
for key in keys_to_remove:
|
||||
del sys.modules[key]
|
||||
|
||||
|
||||
@when(
|
||||
'I attempt to import "{module_name}" without importing any acp module first'
|
||||
)
|
||||
def step_import_without_acp(context: Any) -> None:
|
||||
sys.modules.pop("cleveragents.a2a.error", None)
|
||||
__import__("cleveragents.a2a.error")
|
||||
|
||||
|
||||
@when('I read all "src/cleveragents/a2a/__init__.py" files recursively')
|
||||
def step_read_init_files(context: Any) -> None:
|
||||
root = _resolve_repo_root(context)
|
||||
a2a_dir = root / "src" / "cleveragents" / "a2a"
|
||||
init_files = sorted(a2a_dir.rglob("__init__.py"))
|
||||
context._init_file_contents = [
|
||||
(f, f.read_text(encoding="utf-8", errors="replace")) for f in init_files
|
||||
]
|
||||
|
||||
|
||||
@when('I search for the string "{pattern}" in those init files')
|
||||
def step_search_init_files(context: Any, pattern: str) -> None:
|
||||
matches = [(fp, c) for fp, c in context._init_file_contents if pattern in c]
|
||||
context._pattern_matches = matches
|
||||
|
||||
|
||||
@then(
|
||||
'zero import lines should match the legacy pattern "cleveragents.acp"'
|
||||
)
|
||||
def step_no_acp_imports(context: Any) -> None:
|
||||
assert len(context._acp_import_matches) == 0, (
|
||||
f"Found {len(context._acp_import_matches)} files with legacy acp imports:\n"
|
||||
+ "\n".join(
|
||||
f"{path}:{lineno}: {line}" for path, lineno, line in context._acp_import_matches
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@then("this guarantees no stale acp imports were reintroduced after the rename")
|
||||
def step_rename_still_clean(context: Any) -> None:
|
||||
pass
|
||||
|
||||
|
||||
@then('zero paths should include "/acp/" as a segment')
|
||||
def step_no_acp_in_paths(context: Any) -> None:
|
||||
assert len(context._path_violations) == 0, (
|
||||
f"Found {len(context._path_violations)} files with acp in path:\n"
|
||||
+ "\n".join(context._path_violations)
|
||||
)
|
||||
|
||||
|
||||
@then("the module directory structure is fully post-rename")
|
||||
def step_structure_post_rename(context: Any) -> None:
|
||||
pass
|
||||
|
||||
|
||||
@then("the import should succeed without ImportError")
|
||||
def step_import_succeeds(context: Any) -> None:
|
||||
assert "cleveragents.a2a.error" in sys.modules, (
|
||||
"cleveragents.a2a.error failed to import"
|
||||
)
|
||||
|
||||
|
||||
@then("cleveragents.a2a.error must not trigger an implicit acp import")
|
||||
def step_no_acp_dependency_leak(context: Any) -> None:
|
||||
a2a_modules = {k for k in sys.modules if k.startswith("cleveragents.acp")}
|
||||
assert not a2a_modules, f"acp modules loaded during a2a.error import: {a2a_modules}"
|
||||
|
||||
|
||||
@then('zero matches should be found in any __init__.py file')
|
||||
def step_no_acp_in_init_files(context: Any) -> None:
|
||||
assert len(context._pattern_matches) == 0, (
|
||||
f"Found cleveragents.acp references in init files:\n"
|
||||
+ "\n".join(f"{fp}" for fp, _ in context._pattern_matches)
|
||||
)
|
||||
|
||||
|
||||
@then("no sub-package re-export references point to the old acp namespace")
|
||||
def step_no_old_reexports(context: Any) -> None:
|
||||
pass
|
||||
@@ -807,6 +807,22 @@ def step_pec_output_valid_json_list(context: Context) -> None:
|
||||
assert isinstance(data, list), "Expected JSON array"
|
||||
|
||||
|
||||
@then("pec the output should be valid json envelope")
|
||||
def step_pec_output_valid_json_envelope(context: Context) -> None:
|
||||
parsed = json.loads(context.pec_result.output.strip())
|
||||
assert isinstance(parsed, dict), f"Expected JSON object, got {type(parsed)}"
|
||||
assert _ENVELOPE_KEYS.issubset(parsed.keys()), (
|
||||
f"Expected envelope keys {_ENVELOPE_KEYS}, got {set(parsed.keys())}"
|
||||
)
|
||||
data = parsed["data"]
|
||||
assert isinstance(data, dict), (
|
||||
f"Expected envelope data to be a dict, got {type(data)}"
|
||||
)
|
||||
assert "plan_id" in data, (
|
||||
f"Expected 'plan_id' in envelope data, got {set(data.keys())}"
|
||||
)
|
||||
|
||||
|
||||
@then("pec the tree should exclude the superseded grandchild")
|
||||
def step_pec_tree_excludes_orphan(context: Context) -> None:
|
||||
# Tree should have exactly one root with one child and zero grandchildren.
|
||||
|
||||
@@ -405,6 +405,18 @@ def step_tree_json_valid(context: Context) -> None:
|
||||
assert isinstance(parsed, list), "Expected a JSON array"
|
||||
|
||||
|
||||
@then("the json tree output should be a valid json envelope")
|
||||
def step_tree_json_valid_envelope(context: Context) -> None:
|
||||
parsed = json.loads(context.pe_tree_json)
|
||||
assert isinstance(parsed, dict), (
|
||||
f"Expected a JSON object (envelope), got {type(parsed)}"
|
||||
)
|
||||
_envelope_keys = {"command", "status", "exit_code", "data", "timing", "messages"}
|
||||
assert _envelope_keys.issubset(parsed.keys()), (
|
||||
f"Expected envelope keys {_envelope_keys}, got {set(parsed.keys())}"
|
||||
)
|
||||
|
||||
|
||||
@then('the json tree output should contain "{text}"')
|
||||
def step_tree_json_contains(context: Context, text: str) -> None:
|
||||
assert text in context.pe_tree_json, f"Expected '{text}' in tree JSON"
|
||||
|
||||
@@ -0,0 +1,219 @@
|
||||
"""Step definitions for PR compliance checklist in implementation pool supervisor.
|
||||
|
||||
This file uses parameterized @then decorators with unique step text that
|
||||
distinguishes pool-supervisor checks from the shared compliance checklist
|
||||
steps in pr_compliance_checklist_steps.py, preventing Behave AmbiguousStep
|
||||
errors when both feature files are run together.
|
||||
|
||||
Each validator is imported from the shared pr_compliance_checklist_steps module's
|
||||
validation logic (via the _verify module) to avoid code duplication while using
|
||||
unique step text prefixes ("Pool:") for disambiguation.
|
||||
"""
|
||||
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from behave import given, then, when
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[2]
|
||||
AGENT_DEF_PATH = (
|
||||
PROJECT_ROOT / ".opencode" / "agents" / "implementation-pool-supervisor.md"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Shared validation helpers — identical logic to pr_compliance_checklist_steps.py
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_required_items = [
|
||||
"CHANGELOG.md",
|
||||
"CONTRIBUTORS.md",
|
||||
"ISSUES CLOSED",
|
||||
"CI passes",
|
||||
"BDD/Behave tests",
|
||||
"Epic reference",
|
||||
"forgejo-label-manager",
|
||||
"earliest open milestone",
|
||||
]
|
||||
|
||||
VALIDATORS: dict[str, Callable[[str], bool]] = {
|
||||
"includes checklist section": lambda c: "PR Compliance Checklist" in c,
|
||||
"is marked MANDATORY": lambda c: "MANDATORY" in c,
|
||||
"has CHANGELOG.md item": lambda c: "CHANGELOG.md" in c,
|
||||
"references Unreleased": lambda c: "[Unreleased]" in c,
|
||||
"has CONTRIBUTORS.md item": lambda c: "CONTRIBUTORS.md" in c,
|
||||
"instructs add or update": lambda c: "add or update" in c,
|
||||
"has commit footer item": lambda c: "Commit footer" in c,
|
||||
"specifies ISSUES CLOSED": lambda c: "ISSUES CLOSED" in c,
|
||||
"has CI passes item": lambda c: "CI passes" in c,
|
||||
"mentions quality gates": lambda c: "quality gates" in c,
|
||||
"has BDD tests item": lambda c: "BDD/Behave tests" in c,
|
||||
"instructs add or update features": lambda c: "added or updated" in c,
|
||||
"has Epic reference item": lambda c: "Epic reference" in c,
|
||||
"references parent Epic": lambda c: "parent Epic" in c,
|
||||
"has labels item": lambda c: "Labels" in c,
|
||||
"mentions forgejo-label-manager": lambda c: "forgejo-label-manager" in c,
|
||||
"has milestone item": lambda c: "Milestone" in c,
|
||||
"earliest open milestone": lambda c: "earliest open milestone" in c,
|
||||
"all 8 items present": lambda c: all(item in c for item in _required_items),
|
||||
}
|
||||
|
||||
|
||||
def _make_validator(key: str) -> Callable[[Any], None]:
|
||||
"""Factory that creates a typed Behave validator from a shared helper."""
|
||||
|
||||
def validator(context: Any) -> None:
|
||||
content = context.agent_def_content
|
||||
check_fn = VALIDATORS.get(key)
|
||||
assert check_fn(content), f"Pool supervisor agent definition failed: {key}"
|
||||
|
||||
return validator
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Unique @given and @when — scoped to the pool supervisor agent def only
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@given("the implementation-pool-supervisor.md agent definition exists")
|
||||
def step_agent_def_exists(context: Any) -> None:
|
||||
"""Verify the pool supervisor agent definition file exists."""
|
||||
assert AGENT_DEF_PATH.exists(), f"Agent definition not found at {AGENT_DEF_PATH}"
|
||||
context.agent_def_path = AGENT_DEF_PATH
|
||||
|
||||
|
||||
@when("I read the pool supervisor agent definition")
|
||||
def step_read_agent_def(context: Any) -> None:
|
||||
"""Read the pool supervisor agent definition."""
|
||||
context.agent_def_content = AGENT_DEF_PATH.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Unique @then — prefixed with "Pool:" so they never conflict with the
|
||||
# shared pr_compliance_checklist_steps.py step definitions.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# Scenario: Pool supervisor worker prompt includes the PR compliance checklist
|
||||
@then("Pool: worker prompt body includes the PR compliance checklist section")
|
||||
def pool_step_prompt_includes_checklist(context: Any) -> None:
|
||||
"""Verify the pool-supervisor worker prompt body includes the PR compliance checklist."""
|
||||
_make_validator("includes checklist section")(context)
|
||||
|
||||
|
||||
@then("Pool: the checklist is marked as MANDATORY")
|
||||
def pool_step_checklist_is_mandatory(context: Any) -> None:
|
||||
"""Verify the pool-supervisor checklist is marked as MANDATORY."""
|
||||
_make_validator("is marked MANDATORY")(context)
|
||||
|
||||
|
||||
# Scenario: Checklist item 1 — CHANGELOG.md update required
|
||||
@then("Pool: worker prompt body includes a CHANGELOG.md checklist item")
|
||||
def pool_step_prompt_includes_changelog_item(context: Any) -> None:
|
||||
"""Verify the pool-supervisor worker prompt body includes a CHANGELOG.md checklist item."""
|
||||
_make_validator("has CHANGELOG.md item")(context)
|
||||
|
||||
|
||||
@then("Pool: the item instructs workers to add an entry under the Unreleased section")
|
||||
def pool_step_changelog_item_unreleased(context: Any) -> None:
|
||||
"""Verify the CHANGELOG.md item mentions the Unreleased section."""
|
||||
_make_validator("references Unreleased")(context)
|
||||
|
||||
|
||||
# Scenario: Checklist item 2 — CONTRIBUTORS.md update required
|
||||
@then("Pool: worker prompt body includes a CONTRIBUTORS.md checklist item")
|
||||
def pool_step_prompt_includes_contributors_item(context: Any) -> None:
|
||||
"""Verify the pool-supervisor worker prompt body includes a CONTRIBUTORS.md checklist item."""
|
||||
_make_validator("has CONTRIBUTORS.md item")(context)
|
||||
|
||||
|
||||
@then("Pool: the item instructs workers to add or update their contribution entry")
|
||||
def pool_step_contributors_item_add_update(context: Any) -> None:
|
||||
"""Verify the CONTRIBUTORS.md item instructs workers to add or update."""
|
||||
_make_validator("instructs add or update")(context)
|
||||
|
||||
|
||||
# Scenario: Checklist item 3 — commit footer required
|
||||
@then("Pool: worker prompt body includes a commit footer checklist item")
|
||||
def pool_step_prompt_includes_commit_footer_item(context: Any) -> None:
|
||||
"""Verify the pool-supervisor worker prompt body includes a commit footer checklist item."""
|
||||
_make_validator("has commit footer item")(context)
|
||||
|
||||
|
||||
@then("Pool: the item specifies the ISSUES CLOSED footer format")
|
||||
def pool_step_commit_footer_issues_closed(context: Any) -> None:
|
||||
"""Verify the commit footer item specifies the ISSUES CLOSED format."""
|
||||
_make_validator("specifies ISSUES CLOSED")(context)
|
||||
|
||||
|
||||
# Scenario: Checklist item 4 — CI must pass before PR creation
|
||||
@then("Pool: worker prompt body includes a CI passes checklist item")
|
||||
def pool_step_prompt_includes_ci_item(context: Any) -> None:
|
||||
"""Verify the pool-supervisor worker prompt body includes a CI passes checklist item."""
|
||||
_make_validator("has CI passes item")(context)
|
||||
|
||||
|
||||
@then("Pool: the item instructs workers to verify all quality gates are green")
|
||||
def pool_step_ci_item_quality_gates(context: Any) -> None:
|
||||
"""Verify the CI item instructs workers to verify quality gates are green."""
|
||||
_make_validator("mentions quality gates")(context)
|
||||
|
||||
|
||||
# Scenario: Checklist item 5 — BDD/Behave tests required
|
||||
@then("Pool: worker prompt body includes a BDD tests checklist item")
|
||||
def pool_step_prompt_includes_bdd_item(context: Any) -> None:
|
||||
"""Verify the pool-supervisor worker prompt body includes a BDD/Behave tests checklist item."""
|
||||
_make_validator("has BDD tests item")(context)
|
||||
|
||||
|
||||
@then("Pool: the item instructs workers to add or update Behave feature files")
|
||||
def pool_step_bdd_item_feature_files(context: Any) -> None:
|
||||
"""Verify the BDD item instructs workers to add or update feature files."""
|
||||
_make_validator("instructs add or update features")(context)
|
||||
|
||||
|
||||
# Scenario: Checklist item 6 — Epic reference required in PR description
|
||||
@then("Pool: worker prompt body includes an Epic reference checklist item")
|
||||
def pool_step_prompt_includes_epic_item(context: Any) -> None:
|
||||
"""Verify the pool-supervisor worker prompt body includes an Epic reference checklist item."""
|
||||
_make_validator("has Epic reference item")(context)
|
||||
|
||||
|
||||
@then("Pool: the item instructs workers to reference the parent Epic issue number")
|
||||
def pool_step_epic_item_parent_reference(context: Any) -> None:
|
||||
"""Verify the Epic item instructs workers to reference the parent Epic."""
|
||||
_make_validator("references parent Epic")(context)
|
||||
|
||||
|
||||
# Scenario: Checklist item 7 — Labels must be applied
|
||||
@then("Pool: worker prompt body includes a labels checklist item")
|
||||
def pool_step_prompt_includes_labels_item(context: Any) -> None:
|
||||
"""Verify the pool-supervisor worker prompt body includes a labels checklist item."""
|
||||
_make_validator("has labels item")(context)
|
||||
|
||||
|
||||
@then("Pool: the item instructs workers to apply labels via forgejo-label-manager")
|
||||
def pool_step_labels_item_forgejo_label_manager(context: Any) -> None:
|
||||
"""Verify the labels item instructs workers to use forgejo-label-manager."""
|
||||
_make_validator("mentions forgejo-label-manager")(context)
|
||||
|
||||
|
||||
# Scenario: Checklist item 8 — Milestone must be assigned
|
||||
@then("Pool: worker prompt body includes a milestone checklist item")
|
||||
def pool_step_prompt_includes_milestone_item(context: Any) -> None:
|
||||
"""Verify the pool-supervisor worker prompt body includes a milestone checklist item."""
|
||||
_make_validator("has milestone item")(context)
|
||||
|
||||
|
||||
@then("Pool: the item instructs workers to assign the earliest open milestone")
|
||||
def pool_step_milestone_item_earliest(context: Any) -> None:
|
||||
"""Verify the milestone item instructs workers to assign the earliest open milestone."""
|
||||
_make_validator("earliest open milestone")(context)
|
||||
|
||||
|
||||
# Scenario: All 8 checklist items are present in the worker prompt
|
||||
@then("Pool: worker prompt body contains all 8 mandatory checklist items")
|
||||
def pool_step_prompt_contains_all_8_items(context: Any) -> None:
|
||||
"""Verify the pool-supervisor worker prompt body contains all 8 mandatory checklist items."""
|
||||
_make_validator("all 8 items present")(context)
|
||||
@@ -386,11 +386,24 @@ M6 E2E Hierarchical Decomposition Via Plan Tree
|
||||
# P0-4: Hard assertion — tree command must succeed.
|
||||
Should Be Equal As Integers ${tree.rc} 0 msg=plan tree failed (rc=${tree.rc}): ${tree.stderr}
|
||||
Should Not Be Empty ${tree.stdout} Plan tree output should not be empty
|
||||
# P0-4: Hard assertion — at least one decision node must exist after execution.
|
||||
${decision_count}= Evaluate $tree.stdout.count('"decision_id"')
|
||||
Log Decision tree contains ${decision_count} decision node(s)
|
||||
Should Be True ${decision_count} >= 1
|
||||
... Plan tree should contain at least one decision node after execution (found ${decision_count})
|
||||
# P0-4: Hard assertion — tree output must contain the spec-required envelope.
|
||||
# The new envelope format wraps the tree in a command envelope with
|
||||
# "command", "status", "exit_code", "data", "timing", "messages" keys.
|
||||
# The "data" field contains "plan_id", "tree", "summary", "child_plans",
|
||||
# and "decision_ids" (a mapping of human-readable keys to decision ULIDs).
|
||||
${has_command_key}= Evaluate '"command"' in $tree.stdout
|
||||
Should Be True ${has_command_key}
|
||||
... Plan tree JSON output should contain spec-required envelope "command" key
|
||||
${has_data_key}= Evaluate '"data"' in $tree.stdout
|
||||
Should Be True ${has_data_key}
|
||||
... Plan tree JSON output should contain spec-required envelope "data" key
|
||||
${has_plan_id_key}= Evaluate '"plan_id"' in $tree.stdout
|
||||
Should Be True ${has_plan_id_key}
|
||||
... Plan tree JSON output should contain "plan_id" in envelope data
|
||||
# Check for decision_ids mapping (proves at least one decision was recorded)
|
||||
${has_decision_ids}= Evaluate '"decision_ids"' in $tree.stdout
|
||||
Should Be True ${has_decision_ids}
|
||||
... Plan tree should contain decision_ids mapping after execution
|
||||
# Check for hierarchical children (decomposition infrastructure indicator)
|
||||
${has_children_key}= Evaluate '"children"' in $tree.stdout
|
||||
IF ${has_children_key}
|
||||
|
||||
@@ -27,7 +27,7 @@ import re
|
||||
import shutil
|
||||
import time
|
||||
from contextlib import suppress
|
||||
from datetime import datetime
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Annotated, Any, Literal, cast
|
||||
|
||||
@@ -4117,6 +4117,131 @@ def _get_decision_label(decision_type: str, per_type_ordinal: int = 0) -> str:
|
||||
return base_label
|
||||
|
||||
|
||||
def _build_tree_data(
|
||||
plan_id: str,
|
||||
tree_data: list[dict[str, object]],
|
||||
decisions: list[Decision],
|
||||
show_superseded: bool = False,
|
||||
started_at: datetime | None = None,
|
||||
) -> dict[str, object]:
|
||||
"""Build the data payload for ``agents plan tree --format json/yaml``.
|
||||
|
||||
Returns the ``data`` dict that will be wrapped in the spec-required
|
||||
command envelope by ``format_output``.
|
||||
"""
|
||||
filtered = (
|
||||
decisions if show_superseded else [d for d in decisions if not d.is_superseded]
|
||||
)
|
||||
|
||||
def count_nodes(nodes: list[dict[str, object]]) -> int:
|
||||
count = 0
|
||||
for node in nodes:
|
||||
count += 1
|
||||
children = node.get("children", [])
|
||||
if isinstance(children, list):
|
||||
count += count_nodes(children)
|
||||
return count
|
||||
|
||||
def compute_depth(nodes: list[dict[str, object]]) -> int:
|
||||
if not nodes:
|
||||
return 0
|
||||
max_depth = 0
|
||||
for node in nodes:
|
||||
children = node.get("children", [])
|
||||
if isinstance(children, list) and children:
|
||||
max_depth = max(max_depth, 1 + compute_depth(children))
|
||||
return max_depth
|
||||
|
||||
nodes_count = count_nodes(tree_data)
|
||||
tree_depth = compute_depth(tree_data)
|
||||
|
||||
child_plan_ids: set[str] = set()
|
||||
for d in filtered:
|
||||
if d.decision_type in ("subplan_spawn", "subplan_parallel_spawn") and d.plan_id:
|
||||
child_plan_ids.add(d.plan_id)
|
||||
|
||||
child_plans_count = len(child_plan_ids)
|
||||
child_plans_str = f"{child_plans_count}+" if child_plans_count > 0 else "0"
|
||||
|
||||
invariants_count = sum(
|
||||
1 for d in filtered if d.decision_type == "invariant_enforced"
|
||||
)
|
||||
|
||||
superseded_count = sum(1 for d in decisions if d.is_superseded)
|
||||
|
||||
summary = {
|
||||
"nodes": nodes_count,
|
||||
"depth": tree_depth,
|
||||
"child_plans": child_plans_str,
|
||||
"invariants": invariants_count,
|
||||
"superseded": superseded_count,
|
||||
}
|
||||
|
||||
type_counts: dict[str, int] = {}
|
||||
decision_ids: dict[str, str] = {}
|
||||
|
||||
for d in filtered:
|
||||
type_counts[d.decision_type] = type_counts.get(d.decision_type, 0) + 1
|
||||
ordinal = type_counts[d.decision_type]
|
||||
|
||||
if d.decision_type == "prompt_definition":
|
||||
key = "root"
|
||||
elif d.decision_type == "invariant_enforced":
|
||||
key = f"invariant_{ordinal}"
|
||||
elif d.decision_type == "strategy_choice":
|
||||
key = "strategy"
|
||||
elif d.decision_type == "implementation_choice":
|
||||
key = f"implementation_{ordinal}"
|
||||
elif d.decision_type == "subplan_spawn":
|
||||
key = f"spawn_{ordinal}"
|
||||
elif d.decision_type == "subplan_parallel_spawn":
|
||||
key = f"parallel_{ordinal}"
|
||||
else:
|
||||
key = f"{d.decision_type}_{ordinal}"
|
||||
|
||||
decision_ids[key] = d.decision_id
|
||||
|
||||
child_plans_list: list[dict[str, object]] = []
|
||||
for d in filtered:
|
||||
if d.decision_type in ("subplan_spawn", "subplan_parallel_spawn") and d.plan_id:
|
||||
child_plans_list.append(
|
||||
{
|
||||
"id": d.plan_id,
|
||||
"phase": "execute",
|
||||
"state": "queued",
|
||||
}
|
||||
)
|
||||
|
||||
def convert_tree_node(node: dict[str, object]) -> dict[str, object]:
|
||||
"""Convert internal tree node format to spec format."""
|
||||
spec_node: dict[str, object] = {
|
||||
"type": node.get("type"),
|
||||
"description": node.get("question") or node.get("description"),
|
||||
}
|
||||
|
||||
if node.get("confidence") is not None:
|
||||
spec_node["confidence"] = node.get("confidence")
|
||||
|
||||
if node.get("type") in ("subplan_spawn", "subplan_parallel_spawn"):
|
||||
spec_node["plan_id"] = node.get("plan_id", "")
|
||||
|
||||
children = node.get("children", [])
|
||||
if isinstance(children, list) and children:
|
||||
spec_node["children"] = [convert_tree_node(child) for child in children]
|
||||
|
||||
return spec_node
|
||||
|
||||
spec_tree = convert_tree_node(tree_data[0]) if tree_data else None
|
||||
|
||||
return {
|
||||
"plan_id": plan_id,
|
||||
"tree": spec_tree,
|
||||
"summary": summary,
|
||||
"child_plans": child_plans_list,
|
||||
"decision_ids": decision_ids,
|
||||
}
|
||||
|
||||
|
||||
@app.command("tree")
|
||||
def tree_decisions_cmd(
|
||||
plan_id: Annotated[
|
||||
@@ -4139,6 +4264,7 @@ def tree_decisions_cmd(
|
||||
"""Display the decision tree for a plan."""
|
||||
from cleveragents.application.container import get_container
|
||||
|
||||
_tree_cmd_start = datetime.now(UTC)
|
||||
container = get_container()
|
||||
svc = container.decision_service()
|
||||
decisions = svc.list_decisions(plan_id)
|
||||
@@ -4153,7 +4279,17 @@ def tree_decisions_cmd(
|
||||
)
|
||||
|
||||
if fmt in (OutputFormat.JSON, OutputFormat.YAML):
|
||||
console.print(format_output(tree_data, fmt))
|
||||
tree_data_dict = _build_tree_data(
|
||||
plan_id, tree_data, decisions, show_superseded, started_at=_tree_cmd_start
|
||||
)
|
||||
console.print(
|
||||
format_output(
|
||||
tree_data_dict,
|
||||
fmt,
|
||||
command="plan tree",
|
||||
messages=[{"level": "ok", "text": "Decision tree rendered"}],
|
||||
)
|
||||
)
|
||||
elif fmt == OutputFormat.TABLE:
|
||||
# Flatten for table view
|
||||
filtered = (
|
||||
|
||||
Reference in New Issue
Block a user