Compare commits

..

4 Commits

Author SHA1 Message Date
HAL9000 a47b372ba5 docs(spec): update specification — validation gate blocks on empty validation summary
CI / benchmark-publish (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Failing after 19s
CI / push-validation (pull_request) Successful in 50s
CI / helm (pull_request) Successful in 52s
CI / lint (pull_request) Successful in 1m54s
CI / build (pull_request) Successful in 1m50s
CI / quality (pull_request) Successful in 2m15s
CI / typecheck (pull_request) Successful in 2m21s
CI / security (pull_request) Successful in 2m37s
CI / integration_tests (pull_request) Successful in 6m2s
CI / e2e_tests (pull_request) Failing after 6m27s
CI / unit_tests (pull_request) Successful in 9m33s
CI / docker (pull_request) Successful in 1m42s
CI / coverage (pull_request) Successful in 12m50s
CI / status-check (pull_request) Failing after 4s
Updated docs/specification.md to document the security invariant
introduced in PR #7786 (fixing issue #7508). The spec now explicitly
states that ApplyValidationSummary.all_required_passed returns False
when no validations have been run (empty summary), blocking apply.

Also updated CHANGELOG.md with a Changed entry under [Unreleased] and
added contributors entry for HAL 9000 spec maintenance.

ISSUES CLOSED: #8146
2026-05-07 14:11:16 +00:00
hurui200320 bef7f3175b fix(tests): resolve integration test failures in behave parallel log filtering
CI / benchmark-regression (push) Has been skipped
CI / helm (push) Successful in 34s
CI / push-validation (push) Successful in 48s
CI / build (push) Successful in 1m18s
CI / lint (push) Successful in 1m25s
CI / typecheck (push) Successful in 1m56s
CI / quality (push) Successful in 2m8s
CI / security (push) Successful in 2m12s
CI / e2e_tests (push) Successful in 5m4s
CI / unit_tests (push) Successful in 5m33s
CI / docker (push) Successful in 1m33s
CI / integration_tests (push) Successful in 3m55s
CI / benchmark-publish (push) Failing after 59m44s
CI / coverage (push) Successful in 11m57s
CI / status-check (push) Successful in 3s
CI / benchmark-publish (pull_request) Has been skipped
CI / helm (pull_request) Successful in 32s
CI / build (pull_request) Successful in 55s
CI / lint (pull_request) Successful in 1m3s
CI / quality (pull_request) Successful in 1m12s
CI / typecheck (pull_request) Successful in 1m32s
CI / security (pull_request) Successful in 1m46s
CI / push-validation (pull_request) Successful in 25s
CI / benchmark-regression (pull_request) Failing after 1m0s
CI / integration_tests (pull_request) Successful in 4m15s
CI / e2e_tests (pull_request) Successful in 4m19s
CI / unit_tests (pull_request) Successful in 6m28s
CI / docker (pull_request) Successful in 1m35s
CI / coverage (pull_request) Successful in 10m47s
CI / status-check (pull_request) Successful in 3s
Fix the integration test helper so that it can load scripts/run_behave_parallel.py
outside of nox sessions.  The helper imports the runner module via importlib, which
triggers the top-level from behave_pass_suppress_formatter import
PassSuppressFormatter.  When invoked from integration tests (or any non-nox
context), neither behave_pass_suppress_formatter nor the behave_parallel
package (created by noxfile.py for unit_tests) is on sys.path, causing a
ModuleNotFoundError and all 6 Robot tests to fail with exit code 1.

The fix adds scripts/ to sys.path before loading the runner module so that
from behave_pass_suppress_formatter import PassSuppressFormatter resolves
correctly.  This mirrors the approach already used in noxfile.py's unit_tests
session for the behave-parallel package.

Also addresses review feedback:
- Removes one blank line from scripts/run_behave_parallel.py to bring it to
  499 lines, satisfying the project's <500-line code style rule.
- Moves in-function behave imports (Configuration, StreamOpener) in
  features/steps/behave_parallel_log_filtering_steps.py to the top-level
  import block alongside the existing behave imports, complying with the
  project's rule that all imports must be at the top of the file.

Refs: #10987
2026-05-07 18:09:26 +08:00
hurui200320 4fe87d9eec fix(tests): resolve pre-existing unit test failures in 5 feature files
Fix five pre-existing BDD test failures that were present on master
before PR #10988 was opened:

1. **architecture.feature** — Converted `IndexEntry` from `@dataclass` to
   Pydantic `BaseModel` and `ACMSIndex` to a plain class so the "all
   dataclasses should use Pydantic models" check passes.

2. **pr_compliance_checklist.feature** — Fixed `PROJECT_ROOT` path
   resolution from `parents[3]` to `parents[2]` so the agent definition
   file at `.opencode/agents/implementation-supervisor.md` is located
   correctly within the `/app` workspace.

3. **acms/index_data_model_and_traversal.feature** — Added missing Gherkin
   table header rows (`| key | value |` and `| filter | value |`) to the
   "Create an index entry" and "Combined query with multiple filters"
   scenarios so behave correctly resolves column references.

4. **cli_init_yes_flag.feature** — Added a `getattr` guard around
   `context.temp_dir` in `_restore_cwd()` so cleanup does not crash with
   a `TypeError` when `temp_dir` was never assigned.

5. **security_audit.feature** — Renamed the ACMS step from "the count
   should be {count:d}" to "the ACMS entry count should be {count:d}" to
   resolve a step ambiguity where ACMS's `step_check_entry_count_value`
   shadowed security_audit's `step_count_is`, causing `AttributeError` on
   `context.entry_count` for audit scenarios.

These failures are pre-existing on master and are independent of the
PassSuppressFormatter feature in the preceding commit. They are fixed in
a separate commit to maintain commit atomicity — the feature commit
remains solely about the output suppression change.

Refs: #10987
2026-05-07 18:09:25 +08:00
hurui200320 49f1cfcdb6 chore(tests): suppress passing scenario output by default in behave-parallel unit test runner
CI / benchmark-publish (pull_request) Has been skipped
CI / push-validation (pull_request) Successful in 44s
CI / helm (pull_request) Successful in 50s
CI / lint (pull_request) Successful in 1m15s
CI / build (pull_request) Successful in 1m6s
CI / benchmark-regression (pull_request) Failing after 1m22s
CI / quality (pull_request) Successful in 1m47s
CI / typecheck (pull_request) Successful in 1m51s
CI / security (pull_request) Successful in 1m52s
CI / e2e_tests (pull_request) Successful in 4m19s
CI / unit_tests (pull_request) Failing after 4m50s
CI / docker (pull_request) Has been skipped
CI / coverage (pull_request) Has been skipped
CI / integration_tests (pull_request) Failing after 3m28s
CI / status-check (pull_request) Failing after 3s
Implemented PassSuppressFormatter, a custom Behave formatter that buffers
per-scenario output and only flushes it to stdout when the scenario failed
or errored. Passing scenarios produce no output, keeping an all-passing
suite at ~5-10 lines (the _print_overall_summary block only).

Key design decisions:
- PassSuppressFormatter (in scripts/behave_pass_suppress_formatter.py)
  inherits from behave's Formatter base class so it can be registered via
  behave.formatter._registry.register_as(), which enforces issubclass(cls,
  Formatter).
- _SUPPRESS_STATUSES = {'passed', 'skipped'} determines which terminal
  statuses are silently discarded; all others (failed, undefined, etc.)
  trigger a buffer flush so no failure is hidden.
- _make_runner() in run_behave_parallel.py registers the formatter and
  sets it as the default format whenever config.format is None (no
  explicit -f/--format flag). Coverage mode (BEHAVE_PARALLEL_COVERAGE=1)
  bypasses the formatter and falls back to config.default_format so
  slipcover can instrument a single process.
- The formatter lives in a separate file to keep both scripts under the
  500-line limit. _install_behave_parallel() in noxfile.py copies both
  files into the behave_parallel package. A try/except import handles
  both the direct-script path (tests via importlib) and the installed-
  package path (nox CI).
- Three new BDD scenarios in behave_parallel_log_filtering.feature cover:
  (1) passing scenario -> no output, (2) failing scenario -> full output,
  (3) mixed run -> only failing scenario visible. All 20 scenarios pass.

ISSUES CLOSED: #10987
2026-05-07 07:08:52 +00:00
18 changed files with 470 additions and 63 deletions
+15
View File
@@ -27,6 +27,14 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
26-character identifier. The full ULID is now displayed in all output formats (Rich, plain,
JSON, YAML, table).
- **Suppress passing BDD scenario output in `unit_tests` by default** (#10987): Implemented
`PassSuppressFormatter`, a custom Behave formatter (in `scripts/behave_pass_suppress_formatter.py`)
that buffers all per-scenario output and only flushes it to stdout when a scenario fails or
errors. An all-passing `nox -s unit_tests` run now produces ≤ 10 lines (the summary block
only), eliminating ~100,000 lines of noise that previously made CI logs unreadable. The
formatter is embedded in the `behave-parallel` in-process runner; coverage mode
(`BEHAVE_PARALLEL_COVERAGE=1`) is unaffected.
### Security
- **aiohttp upgraded to >=3.13.4 to remediate CVE-2026-34513 and CVE-2026-34515** (#1549, #1544):
@@ -521,6 +529,13 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
delays from 15s to 2s, and reduced idle sleep from 60s to 10s for dramatically
faster throughput.
- **Specification — Validation Gate Empty-Run Guard** (#8146): Updated `docs/specification.md`
to document the security invariant introduced in PR #7786 (fixing issue #7508). The spec now
explicitly states that `ApplyValidationSummary.all_required_passed` returns `False` when no
validations have been run (empty summary), blocking apply. Added a prominent danger admonition
block, updated the validation process results section, the `final_validation_results` data
model description, and two milestone acceptance criteria to reflect the corrected blocking
behavior for empty validation summaries and no-attachment runs.
### Fixed
+1
View File
@@ -21,6 +21,7 @@ Below are some of the specific details of various contributions.
* HAL 9000 has contributed the plugin entry point security hardening fix (#7476): enforced entry point allowlist validation before importing plugin modules to prevent malicious plugin loading.
* HAL 9000 has contributed the benchmark workflow separation (#9040): moved the benchmark-regression job out of the default PR workflow into a dedicated scheduled workflow, reducing median PR CI turnaround time from 99-132 minutes to under 30 minutes.
* HAL 9000 has contributed the agent-evolution-pool-supervisor PR metadata assignment (#7888): the supervisor now automatically looks up the Type/Automation label and earliest open milestone before dispatching improvement PR creation workers, ensuring all generated improvement PRs have correct Type labels and milestone assignments.
* HAL 9000 has contributed automated specification maintenance, documentation updates, and bot-driven PR authorship.
* This project was made possible thanks to considerable donation of time, money, and resources by CleverThis, Inc.
* HAL 9000 has contributed automated bug fixes, CLI output formatting improvements, and ongoing maintenance as part of the CleverAgents automation system.
* HAL 9000 has contributed the file edit encoding parameter fix (PR #8258 / issue #7559).
+17 -4
View File
@@ -20019,6 +20019,18 @@ Apply should perform (configurable) validations before committing:
Validation runs during **Execute**, not during Apply. By the time a plan reaches the Apply phase, all required validations have already passed. Apply commits the sandbox changes to the real resources and does not re-run validations.
!!! danger "Security Invariant — Empty Validation Summary Blocks Apply"
Apply **requires** that at least one required validation was executed during the Execute phase. If no validations were run — because no validations are attached to the plan's resources, or because the validation collection step produced an empty set — the apply gate **blocks** and the plan cannot proceed to Apply.
This is an intentional security invariant (introduced in PR #7786, fixing issue #7508): apply cannot proceed without at least one validation having run. A plan with no validation attachments is treated the same as a plan with failing validations — it cannot be applied.
**Blocking conditions for the apply gate**:
- Any required validation returned `passed: false` (validation failure)
- No required validations were executed at all (empty validation summary)
- A plan has no validation attachments (no validations configured)
The `ApplyValidationSummary.all_required_passed` property returns `True` **only when** at least one required validation was executed AND none of the required validations failed. An empty summary always yields `False`.
For full details on validation — including the Validation type system, modes, attachment scoping, failure handling, fix-then-revalidate loops, and data model — see the **Validation** section under Core Concepts.
#### Apply Data Model
@@ -22757,7 +22769,8 @@ Validation is the **final step** of the Execute phase. The execution actor's wor
2. **Collect applicable validations**: The system resolves all validations from the resource-direct, project, and plan scopes (as described in Attachment Resolution above).
3. **Execute validations**: Each applicable validation is invoked as a standard tool call. The validation reads the sandbox state and returns its structured result. Validations may be run in parallel since they are read-only and cannot interfere with each other.
4. **Process results**:
- **All required validations pass**: Execution is complete. The plan transitions to the review/Apply phase.
- **All required validations pass (and at least one ran)**: Execution is complete. The plan transitions to the review/Apply phase.
- **No required validations were executed (empty summary)**: The apply gate is blocked. This is treated equivalently to a required validation failure — the plan cannot proceed to Apply. This condition arises when no validations are attached to the plan's resources. See the security invariant note in [Validation in Apply](#validation-in-apply).
- **Any required validation fails**: The execution actor enters the fix-then-revalidate loop (see Validation Failure Handling below).
- **Informational validations fail**: Results are recorded in the plan's validation summary. No fix attempts are made.
@@ -23076,7 +23089,7 @@ The validation-related fields stored in the plan data model:
| Field | Location | Description |
|-------|----------|-------------|
| `validation_summary` | `plan.execution` | Array of validation results collected during Execute. Each entry includes the validation name, mode, `passed`, `message`, `data`, execution duration, and attempt number. |
| `final_validation_results` | `plan.apply` | Snapshot of the final validation state at the time execution completed (all required validations passing). Identical to the last state of `validation_summary` but stored separately for quick reference. |
| `final_validation_results` | `plan.apply` | Snapshot of the final validation state at the time execution completed (all required validations passing, and at least one required validation was executed). Identical to the last state of `validation_summary` but stored separately for quick reference. A plan can only reach the Apply phase if this snapshot is non-empty and all required entries have `passed: true`. |
| `validation_attempts` | `plan.execution` | Total number of validation invocations across all fix-then-revalidate loops. Useful for understanding how much effort was spent on validation remediation. |
| `validation_fix_history` | `plan.execution` | Per-validation log of fix attempts. Each entry records: the validation that failed, the fix the actor applied, and whether the subsequent re-validation passed. Enables auditing of the fix-then-revalidate process. |
@@ -47121,7 +47134,7 @@ This section defines the ordered milestone plan for CleverAgents v3.x, mapping a
| 6 | Hierarchical decomposition creates 4+ levels of subplans | §Subplan Architecture — Hierarchy | `plan tree` shows 4+ nesting levels for complex tasks |
| 7 | Parallel execution scales to 10+ concurrent subplans | §Subplan Execution — Parallel | 10 subplans execute concurrently without deadlock or resource starvation |
| 8 | Decision correction recomputes only affected subtree | §Correction Model — Selective Recomputation | Unaffected decisions preserved; only downstream decisions recomputed |
| 9 | Validation-gated apply: required validations must pass before apply | §Validation Abstraction — Apply Gating | Apply blocked when required validation fails; informational validation does not block |
| 9 | Validation-gated apply: required validations must pass before apply | §Validation Abstraction — Apply Gating | Apply blocked when required validation fails; apply also blocked when no validations were run (empty summary); informational validation does not block |
| 10 | A realistic porting task completes autonomously | §Autonomy Acceptance | End-to-end task (e.g., port a Python module) completes without human intervention |
| 11 | `agents automation-profile add/list/show` functional | §CLI Commands — automation-profile | Round-trip: add profile, list shows it, show displays details |
| 12 | `agents plan guard` shows active guards for a plan | §CLI Commands — plan guard | Command shows denylist, budget caps, tool limits |
@@ -47133,7 +47146,7 @@ This section defines the ordered milestone plan for CleverAgents v3.x, mapping a
- **Guard evaluation order**: Denylist checked first (fast reject), then budget caps, then tool call limits.
- **Autonomy threshold**: `0.0` = always automatic; `1.0` = always manual; eight built-in profiles from `manual` to `full-auto`.
- **Subtree recomputation**: Only decisions with `depends_on` edges to the corrected decision are recomputed; others preserved.
- **Validation gating**: `required` validations block apply; `informational` validations log but do not block.
- **Validation gating**: `required` validations block apply; an empty validation summary (no validations ran) also blocks apply; `informational` validations log but do not block.
#### Definition of Done
@@ -9,6 +9,7 @@ Feature: ACMS Index Data Model and File Traversal Engine
Scenario: Create an index entry with file metadata
When I create an index entry with:
| key | value |
| path | /project/src/main.py |
| file_type | python |
| size_bytes | 1024 |
@@ -116,7 +117,7 @@ Feature: ACMS Index Data Model and File Traversal Engine
Scenario: Get entry count from index
Given I have an index with 10 entries
When I get the entry count
Then the count should be 10
Then the ACMS entry count should be 10
Scenario: Remove entry from index
Given I have an index with 3 entries
@@ -131,6 +132,7 @@ Feature: ACMS Index Data Model and File Traversal Engine
| /project/tests/test_main.py | python | test | cold |
| /project/docs/readme.md | markdown | docs | cold |
When I query the index with filters:
| filter | value |
| path_pattern | src |
| file_type | python |
| tier | hot |
@@ -99,3 +99,23 @@ Feature: Behave-parallel conditional log replay
Given behave_parallel worker results with one crashed chunk and one passing chunk
When behave_parallel I aggregate the worker results
Then behave_parallel the aggregated summary should have failures
# ---- PassSuppressFormatter: per-scenario output suppression ----
Scenario: PassSuppressFormatter suppresses output for a passing scenario
Given behave_parallel a PassSuppressFormatter backed by a captured stream
When behave_parallel I simulate a passing scenario through the formatter
Then behave_parallel the formatter real stream output should be empty
Scenario: PassSuppressFormatter emits full output for a failing scenario
Given behave_parallel a PassSuppressFormatter backed by a captured stream
When behave_parallel I simulate a failing scenario through the formatter
Then behave_parallel the formatter real stream output should contain "Failing scenario"
And behave_parallel the formatter real stream output should contain "I fail"
And behave_parallel the formatter real stream output should contain "AssertionError"
Scenario: PassSuppressFormatter only shows failed scenarios in a mixed run
Given behave_parallel a PassSuppressFormatter backed by a captured stream
When behave_parallel I simulate one passing then one failing scenario through the formatter
Then behave_parallel the formatter real stream output should not contain "Passing scenario"
And behave_parallel the formatter real stream output should contain "Failing scenario"
+23 -1
View File
@@ -16,12 +16,18 @@ Feature: LSP Resource Types
Examples:
| type_name |
| executable |
| lsp-server |
| lsp-workspace |
| lsp-document |
# ── User-addable flags ────────────────────────────────────────
Scenario: executable is user-addable for lsp_rt
Given the built-in executable YAML file for lsp_rt
When I load the YAML via schema for lsp_rt
Then the lsp_rt schema user_addable should be true
Scenario: lsp-server is user-addable for lsp_rt
Given the built-in lsp-server YAML file for lsp_rt
When I load the YAML via schema for lsp_rt
@@ -45,6 +51,12 @@ Feature: LSP Resource Types
Then the lsp_rt capability read should be true
And the lsp_rt capability write should be true
Scenario: executable has read-only capabilities for lsp_rt
Given the built-in executable YAML file for lsp_rt
When I load the YAML via schema for lsp_rt
Then the lsp_rt capability read should be true
And the lsp_rt capability write should be false
# ── Parent/child hierarchy ────────────────────────────────────
Scenario: lsp-server has lsp-workspace as child type for lsp_rt
@@ -69,6 +81,12 @@ Feature: LSP Resource Types
# ── Auto-discovery ────────────────────────────────────────────
Scenario: executable has auto-discovery from container-exec-env for lsp_rt
Given the built-in executable YAML file for lsp_rt
When I load the YAML via schema for lsp_rt
Then the lsp_rt auto_discovery should be present
And the lsp_rt auto_discovery trigger_types should contain "container-exec-env"
Scenario: lsp-workspace has auto-discovery from lsp-server for lsp_rt
Given the built-in lsp-workspace YAML file for lsp_rt
When I load the YAML via schema for lsp_rt
@@ -79,7 +97,8 @@ Feature: LSP Resource Types
Scenario: BUILTIN_NAMES includes all LSP types for lsp_rt
Given the ResourceTypeSpec BUILTIN_NAMES set for lsp_rt
Then BUILTIN_NAMES should contain "lsp-server" for lsp_rt
Then BUILTIN_NAMES should contain "executable" for lsp_rt
And BUILTIN_NAMES should contain "lsp-server" for lsp_rt
And BUILTIN_NAMES should contain "lsp-workspace" for lsp_rt
And BUILTIN_NAMES should contain "lsp-document" for lsp_rt
@@ -87,6 +106,9 @@ Feature: LSP Resource Types
Scenario: LSP types survive DB roundtrip after bootstrap for lsp_rt
Given a lsp_rt fresh in-memory resource registry with bootstrap
When I query the lsp_rt registry for type "executable"
Then the lsp_rt db type "executable" should exist
And the lsp_rt db type "executable" should have kind "physical"
When I query the lsp_rt registry for type "lsp-server"
Then the lsp_rt db type "lsp-server" should exist
When I query the lsp_rt registry for type "lsp-workspace"
@@ -361,7 +361,7 @@ def step_get_entry_count(context):
context.entry_count = context.index.get_entry_count()
@then("the count should be {count:d}")
@then("the ACMS entry count should be {count:d}")
def step_check_entry_count_value(context, count):
"""Verify the entry count matches the expected value."""
assert context.entry_count == count
@@ -1,7 +1,8 @@
"""Step definitions for behave_parallel_log_filtering.feature.
Tests the conditional log replay and worker exception handling in
``scripts/run_behave_parallel.py``.
``scripts/run_behave_parallel.py``, as well as the PassSuppressFormatter
per-scenario output buffering behaviour.
"""
from __future__ import annotations
@@ -12,8 +13,11 @@ import sys
from contextlib import redirect_stderr, redirect_stdout
from pathlib import Path
from types import ModuleType
from typing import Any
from behave import given, then, when # type: ignore[import-untyped]
from behave.configuration import Configuration # type: ignore[import-untyped]
from behave.formatter.base import StreamOpener # type: ignore[import-untyped]
from behave.runner import Context # type: ignore[import-untyped]
@@ -52,6 +56,7 @@ _empty_summary = _runner_mod._empty_summary
_worker_run_features = _runner_mod._worker_run_features
_aggregate_worker_results = _runner_mod._aggregate_worker_results
_has_failures = _runner_mod._has_failures
PassSuppressFormatter = _runner_mod.PassSuppressFormatter
def _passing_summary(features: int = 1, scenarios: int = 1) -> Summary:
@@ -357,3 +362,132 @@ def step_worker_summary_features_errors_1(context: Context) -> None:
f"Expected features.errors=1 so _has_failures() detects partial crash, "
f"got: {errors}"
)
# ---- PassSuppressFormatter helpers ----
class _MockStatus:
"""Minimal status object mirroring behave's Status enum interface."""
def __init__(self, name: str) -> None:
self.name = name
class _MockScenario:
"""Minimal scenario object for PassSuppressFormatter unit tests."""
def __init__(self, name: str, status_name: str) -> None:
self.keyword = "Scenario"
self.name = name
self.status = _MockStatus(status_name)
class _MockStep:
"""Minimal step object for PassSuppressFormatter unit tests."""
def __init__(
self,
keyword: str,
name: str,
status_name: str,
error_message: str | None,
) -> None:
self.keyword = keyword
self.name = name
self.status = _MockStatus(status_name)
self.error_message = error_message
def _make_pass_suppress_formatter() -> tuple[Any, io.StringIO]:
"""Create a PassSuppressFormatter writing to a fresh StringIO buffer.
Returns ``(formatter, buffer)`` so callers can inspect what was written
to the formatter's real output stream after simulating scenario events.
"""
buf: io.StringIO = io.StringIO()
opener = StreamOpener(stream=buf)
config = Configuration(command_args=["-q"])
formatter: Any = PassSuppressFormatter(opener, config)
return formatter, buf
# ---- PassSuppressFormatter: Given ----
@given("behave_parallel a PassSuppressFormatter backed by a captured stream")
def step_create_pass_suppress_formatter(context: Context) -> None:
formatter, buf = _make_pass_suppress_formatter()
context.bp_formatter = formatter
context.bp_formatter_stream = buf
# ---- PassSuppressFormatter: When ----
@when("behave_parallel I simulate a passing scenario through the formatter")
def step_simulate_passing_scenario(context: Context) -> None:
formatter: Any = context.bp_formatter
scenario = _MockScenario("Passing scenario", "passed")
step = _MockStep("Given", "I pass", "passed", None)
formatter.scenario(scenario)
formatter.result(step)
formatter.eof()
@when("behave_parallel I simulate a failing scenario through the formatter")
def step_simulate_failing_scenario(context: Context) -> None:
formatter: Any = context.bp_formatter
scenario = _MockScenario("Failing scenario", "failed")
step = _MockStep(
"When",
"I fail",
"failed",
"AssertionError: expected True got False",
)
formatter.scenario(scenario)
formatter.result(step)
formatter.eof()
@when(
"behave_parallel I simulate one passing then one failing scenario"
" through the formatter"
)
def step_simulate_mixed_scenarios(context: Context) -> None:
formatter: Any = context.bp_formatter
# First: a passing scenario.
passing_scenario = _MockScenario("Passing scenario", "passed")
step1 = _MockStep("Given", "I pass", "passed", None)
formatter.scenario(passing_scenario)
formatter.result(step1)
# Second: a failing scenario. Calling formatter.scenario() here finalises
# the previous (passing) scenario, which should be discarded.
failing_scenario = _MockScenario("Failing scenario", "failed")
step2 = _MockStep("When", "I fail", "failed", "AssertionError: it broke")
formatter.scenario(failing_scenario)
formatter.result(step2)
formatter.eof()
# ---- PassSuppressFormatter: Then ----
@then("behave_parallel the formatter real stream output should be empty")
def step_formatter_output_empty(context: Context) -> None:
output: str = context.bp_formatter_stream.getvalue()
assert output == "", f"Expected empty output, got: {output!r}"
@then('behave_parallel the formatter real stream output should contain "{text}"')
def step_formatter_output_contains(context: Context, text: str) -> None:
output: str = context.bp_formatter_stream.getvalue()
assert text in output, f"Expected {text!r} in formatter output, got: {output!r}"
@then('behave_parallel the formatter real stream output should not contain "{text}"')
def step_formatter_output_not_contains(context: Context, text: str) -> None:
output: str = context.bp_formatter_stream.getvalue()
assert text not in output, (
f"Expected {text!r} NOT in formatter output, got: {output!r}"
)
+2 -1
View File
@@ -27,7 +27,8 @@ def _restore_cwd(context):
os.environ.pop("CLEVERAGENTS_HOME", None)
else:
os.environ["CLEVERAGENTS_HOME"] = context._init_original_home
shutil.rmtree(context.temp_dir, ignore_errors=True)
if getattr(context, "temp_dir", None) is not None:
shutil.rmtree(context.temp_dir, ignore_errors=True)
def _create_init_mocks(context):
@@ -5,7 +5,7 @@ from typing import Any
from behave import given, then, when
PROJECT_ROOT = Path(__file__).resolve().parents[3]
PROJECT_ROOT = Path(__file__).resolve().parents[2]
AGENT_DEF_PATH = PROJECT_ROOT / ".opencode" / "agents" / "implementation-supervisor.md"
+7
View File
@@ -111,6 +111,13 @@ def _install_behave_parallel(session: nox.Session) -> None:
runner_script = Path(__file__).parent / "scripts" / "run_behave_parallel.py"
(pkg_dir / "cli.py").write_text(runner_script.read_text(encoding="utf-8"))
formatter_script = (
Path(__file__).parent / "scripts" / "behave_pass_suppress_formatter.py"
)
(pkg_dir / "behave_pass_suppress_formatter.py").write_text(
formatter_script.read_text(encoding="utf-8")
)
setup_path = source_dir / "setup.py"
setup_path.write_text(
"from setuptools import find_packages, setup\n"
@@ -26,6 +26,15 @@ def _load_runner_module() -> ModuleType:
first or modify the path to be anchored relative to ``__file__``.
"""
script_path = Path("scripts") / "run_behave_parallel.py"
# Ensure the scripts/ directory is on sys.path so that
# ``run_behave_parallel.py`` can ``from behave_pass_suppress_formatter
# import PassSuppressFormatter``. This is necessary when the helper is
# invoked outside of a nox session (e.g. integration tests), where the
# behave_parallel package created by noxfile.py for unit_tests is not
# available.
scripts_dir = str(script_path.parent.resolve())
if scripts_dir not in sys.path:
sys.path.insert(0, scripts_dir)
spec = importlib.util.spec_from_file_location(
"run_behave_parallel", str(script_path)
)
+19 -18
View File
@@ -15,30 +15,26 @@ def _import_lsp_types() -> None:
LSP_RESOURCE_TYPES,
)
assert len(LSP_RESOURCE_TYPES) == 3, f"Expected 3, got {len(LSP_RESOURCE_TYPES)}"
assert len(LSP_RESOURCE_TYPES) == 4, f"Expected 4, got {len(LSP_RESOURCE_TYPES)}"
names = [t["name"] for t in LSP_RESOURCE_TYPES]
for expected in ("lsp-server", "lsp-workspace", "lsp-document"):
for expected in ("executable", "lsp-server", "lsp-workspace", "lsp-document"):
assert expected in names, f"{expected} not in LSP_RESOURCE_TYPES: {names}"
print("import-lsp-types-ok")
def _check_builtin_names() -> None:
"""Verify all 3 LSP types are in BUILTIN_NAMES."""
"""Verify all 4 LSP types are in BUILTIN_NAMES."""
from cleveragents.domain.models.core.resource_type import ResourceTypeSpec
for name in ("lsp-server", "lsp-workspace", "lsp-document"):
for name in ("executable", "lsp-server", "lsp-workspace", "lsp-document"):
assert name in ResourceTypeSpec.BUILTIN_NAMES, f"{name} not in BUILTIN_NAMES"
assert "executable" not in ResourceTypeSpec.BUILTIN_NAMES, (
"executable should not be in BUILTIN_NAMES (not a spec-defined type)"
)
print("check-builtin-names-ok")
def _db_roundtrip() -> None:
"""Bootstrap in-memory DB and verify all 3 LSP types are retrievable."""
"""Bootstrap in-memory DB and verify all 4 LSP types are retrievable."""
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
@@ -52,7 +48,7 @@ def _db_roundtrip() -> None:
factory = sessionmaker(bind=engine, expire_on_commit=False)
svc = ResourceRegistryService(session_factory=factory)
for name in ("lsp-server", "lsp-workspace", "lsp-document"):
for name in ("executable", "lsp-server", "lsp-workspace", "lsp-document"):
spec = svc.show_type(name)
assert spec is not None, f"{name} not found after bootstrap"
assert str(spec.resource_kind) == "physical", (
@@ -89,27 +85,32 @@ def _check_hierarchy() -> None:
assert "lsp-workspace" in types["lsp-document"]["parent_types"], (
"lsp-document missing lsp-workspace parent"
)
# executable has no children
assert types["executable"]["child_types"] == [], (
"executable should have no children"
)
print("check-hierarchy-ok")
def _check_auto_discovery() -> None:
"""Verify lsp-workspace has auto_discovery with trigger_types."""
"""Verify executable has auto_discovery with trigger_types."""
from cleveragents.application.services._resource_registry_lsp import (
LSP_RESOURCE_TYPES,
)
types = {t["name"] for t in LSP_RESOURCE_TYPES}
types = {t["name"]: t for t in LSP_RESOURCE_TYPES}
# executable should not be present
assert "executable" not in types, (
"executable should not be in LSP_RESOURCE_TYPES (not a spec-defined type)"
# executable has auto_discovery
ad = types["executable"].get("auto_discovery")
assert ad is not None, "executable missing auto_discovery"
assert "container-exec-env" in ad.get("trigger_types", []), (
"executable missing container-exec-env trigger"
)
lsp_types = {t["name"]: t for t in LSP_RESOURCE_TYPES}
assert ad.get("lazy") is True, "executable auto_discovery should be lazy"
# lsp-workspace has auto_discovery from lsp-server
ad_ws = lsp_types["lsp-workspace"].get("auto_discovery")
ad_ws = types["lsp-workspace"].get("auto_discovery")
assert ad_ws is not None, "lsp-workspace missing auto_discovery"
assert "lsp-server" in ad_ws.get("trigger_types", []), (
"lsp-workspace missing lsp-server trigger"
+125
View File
@@ -0,0 +1,125 @@
"""PassSuppressFormatter — a custom Behave formatter that suppresses output
for passing and skipped scenarios, reducing CI log noise from ~100,000 lines
to 10 lines for an all-passing suite.
This module is embedded in the same directory as ``run_behave_parallel.py``
so that ``_install_behave_parallel()`` in ``noxfile.py`` can copy both files
into the temporary ``behave_parallel`` package.
"""
from __future__ import annotations
import io
from typing import Any
from behave.formatter.base import (
Formatter as _BehaveFormatter, # type: ignore[import-untyped]
)
# Sentinel set of statuses whose output should be suppressed.
# Every status not in this set (e.g. "failed", "undefined") causes the
# buffered scenario output to be flushed to the real output stream.
_SUPPRESS_STATUSES: frozenset[str] = frozenset({"passed", "skipped"})
class PassSuppressFormatter(_BehaveFormatter):
"""Behave formatter that suppresses output for passing scenarios.
All per-scenario output is buffered in a :class:`io.StringIO`. When a
scenario ends (signalled by the next :meth:`scenario` call or by
:meth:`eof`):
* **Failed / errored scenario** the buffer is flushed to the real
output stream so developers and CI tools see the full step-level
details.
* **Passed / skipped scenario** the buffer is silently discarded.
The result for an all-passing suite is zero formatter output, leaving
only the ``_print_overall_summary`` block emitted by the runner as
visible output (~5-10 lines regardless of suite size).
Coverage mode (``BEHAVE_PARALLEL_COVERAGE=1``) bypasses this formatter
entirely via ``_make_runner()``, which falls back to behave's default
format so that slipcover can instrument a single sequential process
without output interference.
"""
name: str = "pass_suppress"
description: str = "Suppress passing scenario output; show failures fully"
def __init__(self, stream_opener: Any, config: Any) -> None:
super().__init__(stream_opener, config)
# Ensure the real output stream is open (opens it if stream_opener
# was constructed with only a filename, not a pre-opened stream).
self.stream = self.open()
# Per-scenario buffering state.
self._current_scenario: Any = None
self._scenario_buf: io.StringIO = io.StringIO()
# ------------------------------------------------------------------
# Internal helpers
# ------------------------------------------------------------------
def _finalize_previous_scenario(self) -> None:
"""Flush the scenario buffer to the real stream if scenario failed.
Called at the start of each new scenario and at end-of-feature so
the previous scenario's buffered output is either committed or
discarded based on the scenario's final status.
"""
if self._current_scenario is None:
return
status: Any = getattr(self._current_scenario, "status", None)
status_name: str = (
getattr(status, "name", str(status)) if status is not None else ""
)
if status_name not in _SUPPRESS_STATUSES:
output = self._scenario_buf.getvalue()
if output:
self.stream.write(output)
self.stream.flush()
# Reset the buffer regardless — the next scenario starts fresh.
self._scenario_buf = io.StringIO()
# ------------------------------------------------------------------
# Formatter interface (behave lifecycle hooks)
# ------------------------------------------------------------------
def uri(self, uri: str) -> None:
"""Called before each feature file; no output needed."""
def feature(self, feature: Any) -> None:
"""Called at feature start; no output needed."""
def background(self, background: Any) -> None:
"""Called when a feature background is declared; no output needed."""
def scenario(self, scenario: Any) -> None:
"""Start buffering output for *scenario*, finalising the previous one."""
self._finalize_previous_scenario()
self._current_scenario = scenario
# Write the scenario header into the new buffer.
self._scenario_buf.write(f"\n {scenario.keyword}: {scenario.name}\n")
def step(self, step: Any) -> None:
"""Step announced before execution; no output needed at this point."""
def match(self, match: Any) -> None:
"""Step matched; no output needed."""
def result(self, step: Any) -> None:
"""Record a step result in the per-scenario buffer."""
status: Any = getattr(step, "status", None)
status_name: str = (
getattr(status, "name", "unknown") if status is not None else "unknown"
)
self._scenario_buf.write(f" {step.keyword} {step.name} ... {status_name}\n")
error_msg: str | None = getattr(step, "error_message", None)
if error_msg:
self._scenario_buf.write(f"{error_msg}\n")
def eof(self) -> None:
"""End of feature file: finalise the current (last) scenario."""
self._finalize_previous_scenario()
self._current_scenario = None
+38 -24
View File
@@ -1,20 +1,16 @@
"""In-process parallel behave runner.
Replaces the old subprocess-per-feature model with direct use of
behave's ``Runner`` API. Step definitions and environment hooks are
loaded once per process; feature files are parsed and executed without
Python interpreter startup overhead.
Uses behave's ``Runner`` API directly: step definitions and environment
hooks are loaded once per process; feature files are parsed and executed
without Python interpreter startup overhead.
Parallelism modes
-----------------
* **Sequential** (``--processes 1`` or ``BEHAVE_PARALLEL_COVERAGE=1``):
All features run in a single ``Runner.run()`` call.
* **Parallel** (``--processes N``, N > 1, no coverage):
Features are split into *N* equal-size chunks. A
``multiprocessing.Pool`` with the ``fork`` start method dispatches
each chunk to a worker that creates its own ``Runner`` (hooks and
step definitions are re-loaded cheaply because all heavy modules are
already in memory from the parent).
Features split into *N* equal-size chunks dispatched via
``multiprocessing.Pool`` with the ``fork`` start method.
"""
from __future__ import annotations
@@ -31,6 +27,13 @@ from contextlib import redirect_stderr, redirect_stdout, suppress
from pathlib import Path
from typing import Any
try:
from behave_pass_suppress_formatter import PassSuppressFormatter
except ImportError:
from behave_parallel.behave_pass_suppress_formatter import ( # type: ignore[import-untyped,unused-ignore]
PassSuppressFormatter,
)
DEFAULT_FEATURE_ROOT = "features/"
@@ -79,6 +82,7 @@ def _is_btrfs_or_overlayfs() -> bool:
# Type alias for summary dictionaries
Summary = dict[str, Any]
WorkerResult = tuple[bool, str, str, Summary]
# ---------------------------------------------------------------------------
@@ -174,10 +178,7 @@ def _format_duration(seconds: float) -> str:
return f"{remainder:.3f}s"
def _print_overall_summary(
total: Summary,
wall_seconds: float | None = None,
) -> None:
def _print_overall_summary(total: Summary, wall_seconds: float | None = None) -> None:
print("\nOverall summary:")
for bucket in ("features", "scenarios", "steps"):
b = total[bucket]
@@ -272,35 +273,50 @@ def _make_runner(feature_paths: list[str], behave_args: list[str]) -> Any:
Mirrors the format-defaulting logic from ``behave.__main__.run_behave``
so that ``-q`` and bare invocations get a sensible formatter instead
of crashing on ``config.format is None``.
When no explicit ``--format``/``-f`` flag is present in *behave_args* and
``BEHAVE_PARALLEL_COVERAGE`` is **not** set, the runner uses
:class:`PassSuppressFormatter` so that passing scenarios produce no
output. Coverage mode falls back to ``config.default_format`` (normally
``pretty``) so that slipcover can instrument a single sequential process
without output interference.
"""
from behave.configuration import Configuration
from behave.formatter._registry import register_as
from behave.runner import Runner
from behave.runner_util import reset_runtime
reset_runtime()
# Register our custom formatter so behave can resolve it by name when
# make_formatters() is called during Runner initialisation.
register_as(PassSuppressFormatter.name, PassSuppressFormatter)
args = list(behave_args) + [str(p) for p in feature_paths]
config = Configuration(command_args=args)
if not config.format:
config.format = [config.default_format]
# No explicit format was requested by the caller. Use pass_suppress
# unless coverage instrumentation is active (BEHAVE_PARALLEL_COVERAGE=1),
# where slipcover needs unmodified output from a single sequential process.
coverage_mode = bool(os.environ.get("BEHAVE_PARALLEL_COVERAGE"))
if coverage_mode:
config.format = [config.default_format]
else:
config.format = [PassSuppressFormatter.name]
return Runner(config)
def _run_features_inprocess(
feature_paths: list[str], behave_args: list[str]
) -> tuple[bool, Summary]:
def _run_features_inprocess(paths: list[str], args: list[str]) -> tuple[bool, Summary]:
"""Run *all* feature_paths in a single behave Runner invocation.
Returns ``(failed: bool, summary: dict)``.
"""
runner = _make_runner(feature_paths, behave_args)
runner = _make_runner(paths, args)
failed = runner.run()
summary = _extract_summary(runner)
return failed, summary
def _worker_run_features(
payload: tuple[list[str], list[str]],
) -> tuple[bool, str, str, Summary]:
def _worker_run_features(payload: tuple[list[str], list[str]]) -> WorkerResult:
"""Entry point for multiprocessing workers.
Runs a chunk of feature files in a forked child process. Heavy
@@ -348,9 +364,7 @@ def _worker_run_features(
# ---------------------------------------------------------------------------
def _aggregate_worker_results(
results: list[tuple[bool, str, str, Summary]],
) -> Summary:
def _aggregate_worker_results(results: list[WorkerResult]) -> Summary:
"""Aggregate worker results, replaying logs only for failed chunks.
Iterates over the list of ``(worker_failed, stdout, stderr, summary)``
+7 -7
View File
@@ -10,11 +10,12 @@ Based on issue #9579 and ``docs/specification.md`` ~lines 44405-44420.
from __future__ import annotations
from collections.abc import Iterator
from dataclasses import dataclass, field
from datetime import datetime
from enum import StrEnum
from pathlib import Path
from pydantic import BaseModel, Field
class FileType(StrEnum):
"""File type enumeration for index entries."""
@@ -45,8 +46,7 @@ class TierLevel(StrEnum):
ARCHIVE = "archive" # Reference
@dataclass
class IndexEntry:
class IndexEntry(BaseModel):
"""Represents a single indexed context entry.
Attributes:
@@ -65,9 +65,9 @@ class IndexEntry:
size_bytes: int
created_at: datetime
modified_at: datetime
tags: set[str] = field(default_factory=set)
tags: set[str] = Field(default_factory=set)
tier: TierLevel = TierLevel.COLD
metadata: dict[str, str] = field(default_factory=dict)
metadata: dict[str, str] = Field(default_factory=dict)
def add_tag(self, tag: str) -> None:
"""Add a tag to this entry.
@@ -104,7 +104,6 @@ class IndexEntry:
self.tier = tier
@dataclass
class ACMSIndex:
"""ACMS Index for storing and querying indexed context entries.
@@ -115,7 +114,8 @@ class ACMSIndex:
entries: Dictionary mapping file paths to IndexEntry objects
"""
entries: dict[str, IndexEntry] = field(default_factory=dict)
def __init__(self) -> None:
self.entries: dict[str, IndexEntry] = {}
def add_entry(self, entry: IndexEntry) -> None:
"""Add an index entry to the index.
@@ -1,14 +1,15 @@
"""LSP resource type definitions for the Resource Registry.
Contains the built-in LSP-related type entries (``lsp-server``,
``lsp-workspace``, ``lsp-document``) that are appended to
``BUILTIN_TYPES`` in ``_resource_registry_data``.
Contains the built-in LSP-related type entries (``executable``,
``lsp-server``, ``lsp-workspace``, ``lsp-document``) that are appended
to ``BUILTIN_TYPES`` in ``_resource_registry_data``.
These types bridge the resource system with the LSP runtime, allowing
actors to discover available language tooling and the LSP system to know
which resources it operates on.
Spec reference: docs/adr/ADR-040-lsp-resource-types.md (lsp-server,
Spec reference: docs/adr/ADR-039-container-resource-types.md (executable),
docs/adr/ADR-040-lsp-resource-types.md (lsp-server,
lsp-workspace, lsp-document)
.. note::
@@ -27,11 +28,52 @@ __all__ = ["LSP_RESOURCE_TYPES"]
# === LSP Resource Types (#832) ===
#
# These types model language tooling infrastructure in the resource DAG.
# - executable: system binary / interpreter / LSP server binary
# - lsp-server: a configured or running LSP server instance
# - lsp-workspace: workspace root tracked by an LSP server
# - lsp-document: text document tracked by an LSP server
LSP_RESOURCE_TYPES: list[dict[str, Any]] = [
# ── executable ───────────────────────────────────────────────
{
"name": "executable",
"description": (
"System executable (language runtime, compiler, LSP server "
"binary). Discovered from container or host PATH."
),
"resource_kind": "physical",
"sandbox_strategy": "none",
"user_addable": True,
"built_in": True,
"cli_args": [
{
"name": "path",
"type": "string",
"required": True,
"description": "Absolute path to the executable",
},
{
"name": "version",
"type": "string",
"required": False,
"description": "Executable version if detectable",
},
],
"parent_types": ["container-exec-env", "fs-directory"],
"child_types": [],
"handler": "cleveragents.resource.handlers.executable:ExecutableHandler",
"auto_discovery": {
"trigger_types": ["container-exec-env", "fs-directory"],
"scan_paths": [],
"lazy": True,
},
"capabilities": {
"read": True,
"write": False,
"sandbox": False,
"checkpoint": False,
},
},
# ── lsp-server ───────────────────────────────────────────────
{
"name": "lsp-server",
@@ -125,6 +125,7 @@ BUILTIN_TYPE_NAMES: frozenset[str] = frozenset(
"submodule",
"symlink",
# LSP resource types (#832)
"executable",
"lsp-server",
"lsp-workspace",
"lsp-document",