Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| a47b372ba5 | |||
|
bef7f3175b
|
|||
|
4fe87d9eec
|
|||
| 49f1cfcdb6 |
@@ -27,6 +27,14 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
26-character identifier. The full ULID is now displayed in all output formats (Rich, plain,
|
||||
JSON, YAML, table).
|
||||
|
||||
- **Suppress passing BDD scenario output in `unit_tests` by default** (#10987): Implemented
|
||||
`PassSuppressFormatter`, a custom Behave formatter (in `scripts/behave_pass_suppress_formatter.py`)
|
||||
that buffers all per-scenario output and only flushes it to stdout when a scenario fails or
|
||||
errors. An all-passing `nox -s unit_tests` run now produces ≤ 10 lines (the summary block
|
||||
only), eliminating ~100,000 lines of noise that previously made CI logs unreadable. The
|
||||
formatter is embedded in the `behave-parallel` in-process runner; coverage mode
|
||||
(`BEHAVE_PARALLEL_COVERAGE=1`) is unaffected.
|
||||
|
||||
### Security
|
||||
|
||||
- **aiohttp upgraded to >=3.13.4 to remediate CVE-2026-34513 and CVE-2026-34515** (#1549, #1544):
|
||||
@@ -521,6 +529,13 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
delays from 15s to 2s, and reduced idle sleep from 60s to 10s for dramatically
|
||||
faster throughput.
|
||||
|
||||
- **Specification — Validation Gate Empty-Run Guard** (#8146): Updated `docs/specification.md`
|
||||
to document the security invariant introduced in PR #7786 (fixing issue #7508). The spec now
|
||||
explicitly states that `ApplyValidationSummary.all_required_passed` returns `False` when no
|
||||
validations have been run (empty summary), blocking apply. Added a prominent danger admonition
|
||||
block, updated the validation process results section, the `final_validation_results` data
|
||||
model description, and two milestone acceptance criteria to reflect the corrected blocking
|
||||
behavior for empty validation summaries and no-attachment runs.
|
||||
|
||||
### Fixed
|
||||
|
||||
|
||||
@@ -21,6 +21,7 @@ Below are some of the specific details of various contributions.
|
||||
* HAL 9000 has contributed the plugin entry point security hardening fix (#7476): enforced entry point allowlist validation before importing plugin modules to prevent malicious plugin loading.
|
||||
* HAL 9000 has contributed the benchmark workflow separation (#9040): moved the benchmark-regression job out of the default PR workflow into a dedicated scheduled workflow, reducing median PR CI turnaround time from 99-132 minutes to under 30 minutes.
|
||||
* HAL 9000 has contributed the agent-evolution-pool-supervisor PR metadata assignment (#7888): the supervisor now automatically looks up the Type/Automation label and earliest open milestone before dispatching improvement PR creation workers, ensuring all generated improvement PRs have correct Type labels and milestone assignments.
|
||||
* HAL 9000 has contributed automated specification maintenance, documentation updates, and bot-driven PR authorship.
|
||||
* This project was made possible thanks to considerable donation of time, money, and resources by CleverThis, Inc.
|
||||
* HAL 9000 has contributed automated bug fixes, CLI output formatting improvements, and ongoing maintenance as part of the CleverAgents automation system.
|
||||
* HAL 9000 has contributed the file edit encoding parameter fix (PR #8258 / issue #7559).
|
||||
|
||||
+17
-4
@@ -20019,6 +20019,18 @@ Apply should perform (configurable) validations before committing:
|
||||
|
||||
Validation runs during **Execute**, not during Apply. By the time a plan reaches the Apply phase, all required validations have already passed. Apply commits the sandbox changes to the real resources and does not re-run validations.
|
||||
|
||||
!!! danger "Security Invariant — Empty Validation Summary Blocks Apply"
|
||||
Apply **requires** that at least one required validation was executed during the Execute phase. If no validations were run — because no validations are attached to the plan's resources, or because the validation collection step produced an empty set — the apply gate **blocks** and the plan cannot proceed to Apply.
|
||||
|
||||
This is an intentional security invariant (introduced in PR #7786, fixing issue #7508): apply cannot proceed without at least one validation having run. A plan with no validation attachments is treated the same as a plan with failing validations — it cannot be applied.
|
||||
|
||||
**Blocking conditions for the apply gate**:
|
||||
- Any required validation returned `passed: false` (validation failure)
|
||||
- No required validations were executed at all (empty validation summary)
|
||||
- A plan has no validation attachments (no validations configured)
|
||||
|
||||
The `ApplyValidationSummary.all_required_passed` property returns `True` **only when** at least one required validation was executed AND none of the required validations failed. An empty summary always yields `False`.
|
||||
|
||||
For full details on validation — including the Validation type system, modes, attachment scoping, failure handling, fix-then-revalidate loops, and data model — see the **Validation** section under Core Concepts.
|
||||
|
||||
#### Apply Data Model
|
||||
@@ -22757,7 +22769,8 @@ Validation is the **final step** of the Execute phase. The execution actor's wor
|
||||
2. **Collect applicable validations**: The system resolves all validations from the resource-direct, project, and plan scopes (as described in Attachment Resolution above).
|
||||
3. **Execute validations**: Each applicable validation is invoked as a standard tool call. The validation reads the sandbox state and returns its structured result. Validations may be run in parallel since they are read-only and cannot interfere with each other.
|
||||
4. **Process results**:
|
||||
- **All required validations pass**: Execution is complete. The plan transitions to the review/Apply phase.
|
||||
- **All required validations pass (and at least one ran)**: Execution is complete. The plan transitions to the review/Apply phase.
|
||||
- **No required validations were executed (empty summary)**: The apply gate is blocked. This is treated equivalently to a required validation failure — the plan cannot proceed to Apply. This condition arises when no validations are attached to the plan's resources. See the security invariant note in [Validation in Apply](#validation-in-apply).
|
||||
- **Any required validation fails**: The execution actor enters the fix-then-revalidate loop (see Validation Failure Handling below).
|
||||
- **Informational validations fail**: Results are recorded in the plan's validation summary. No fix attempts are made.
|
||||
|
||||
@@ -23076,7 +23089,7 @@ The validation-related fields stored in the plan data model:
|
||||
| Field | Location | Description |
|
||||
|-------|----------|-------------|
|
||||
| `validation_summary` | `plan.execution` | Array of validation results collected during Execute. Each entry includes the validation name, mode, `passed`, `message`, `data`, execution duration, and attempt number. |
|
||||
| `final_validation_results` | `plan.apply` | Snapshot of the final validation state at the time execution completed (all required validations passing). Identical to the last state of `validation_summary` but stored separately for quick reference. |
|
||||
| `final_validation_results` | `plan.apply` | Snapshot of the final validation state at the time execution completed (all required validations passing, and at least one required validation was executed). Identical to the last state of `validation_summary` but stored separately for quick reference. A plan can only reach the Apply phase if this snapshot is non-empty and all required entries have `passed: true`. |
|
||||
| `validation_attempts` | `plan.execution` | Total number of validation invocations across all fix-then-revalidate loops. Useful for understanding how much effort was spent on validation remediation. |
|
||||
| `validation_fix_history` | `plan.execution` | Per-validation log of fix attempts. Each entry records: the validation that failed, the fix the actor applied, and whether the subsequent re-validation passed. Enables auditing of the fix-then-revalidate process. |
|
||||
|
||||
@@ -47121,7 +47134,7 @@ This section defines the ordered milestone plan for CleverAgents v3.x, mapping a
|
||||
| 6 | Hierarchical decomposition creates 4+ levels of subplans | §Subplan Architecture — Hierarchy | `plan tree` shows 4+ nesting levels for complex tasks |
|
||||
| 7 | Parallel execution scales to 10+ concurrent subplans | §Subplan Execution — Parallel | 10 subplans execute concurrently without deadlock or resource starvation |
|
||||
| 8 | Decision correction recomputes only affected subtree | §Correction Model — Selective Recomputation | Unaffected decisions preserved; only downstream decisions recomputed |
|
||||
| 9 | Validation-gated apply: required validations must pass before apply | §Validation Abstraction — Apply Gating | Apply blocked when required validation fails; informational validation does not block |
|
||||
| 9 | Validation-gated apply: required validations must pass before apply | §Validation Abstraction — Apply Gating | Apply blocked when required validation fails; apply also blocked when no validations were run (empty summary); informational validation does not block |
|
||||
| 10 | A realistic porting task completes autonomously | §Autonomy Acceptance | End-to-end task (e.g., port a Python module) completes without human intervention |
|
||||
| 11 | `agents automation-profile add/list/show` functional | §CLI Commands — automation-profile | Round-trip: add profile, list shows it, show displays details |
|
||||
| 12 | `agents plan guard` shows active guards for a plan | §CLI Commands — plan guard | Command shows denylist, budget caps, tool limits |
|
||||
@@ -47133,7 +47146,7 @@ This section defines the ordered milestone plan for CleverAgents v3.x, mapping a
|
||||
- **Guard evaluation order**: Denylist checked first (fast reject), then budget caps, then tool call limits.
|
||||
- **Autonomy threshold**: `0.0` = always automatic; `1.0` = always manual; eight built-in profiles from `manual` to `full-auto`.
|
||||
- **Subtree recomputation**: Only decisions with `depends_on` edges to the corrected decision are recomputed; others preserved.
|
||||
- **Validation gating**: `required` validations block apply; `informational` validations log but do not block.
|
||||
- **Validation gating**: `required` validations block apply; an empty validation summary (no validations ran) also blocks apply; `informational` validations log but do not block.
|
||||
|
||||
#### Definition of Done
|
||||
|
||||
|
||||
@@ -9,6 +9,7 @@ Feature: ACMS Index Data Model and File Traversal Engine
|
||||
|
||||
Scenario: Create an index entry with file metadata
|
||||
When I create an index entry with:
|
||||
| key | value |
|
||||
| path | /project/src/main.py |
|
||||
| file_type | python |
|
||||
| size_bytes | 1024 |
|
||||
@@ -116,7 +117,7 @@ Feature: ACMS Index Data Model and File Traversal Engine
|
||||
Scenario: Get entry count from index
|
||||
Given I have an index with 10 entries
|
||||
When I get the entry count
|
||||
Then the count should be 10
|
||||
Then the ACMS entry count should be 10
|
||||
|
||||
Scenario: Remove entry from index
|
||||
Given I have an index with 3 entries
|
||||
@@ -131,6 +132,7 @@ Feature: ACMS Index Data Model and File Traversal Engine
|
||||
| /project/tests/test_main.py | python | test | cold |
|
||||
| /project/docs/readme.md | markdown | docs | cold |
|
||||
When I query the index with filters:
|
||||
| filter | value |
|
||||
| path_pattern | src |
|
||||
| file_type | python |
|
||||
| tier | hot |
|
||||
|
||||
@@ -99,3 +99,23 @@ Feature: Behave-parallel conditional log replay
|
||||
Given behave_parallel worker results with one crashed chunk and one passing chunk
|
||||
When behave_parallel I aggregate the worker results
|
||||
Then behave_parallel the aggregated summary should have failures
|
||||
|
||||
# ---- PassSuppressFormatter: per-scenario output suppression ----
|
||||
|
||||
Scenario: PassSuppressFormatter suppresses output for a passing scenario
|
||||
Given behave_parallel a PassSuppressFormatter backed by a captured stream
|
||||
When behave_parallel I simulate a passing scenario through the formatter
|
||||
Then behave_parallel the formatter real stream output should be empty
|
||||
|
||||
Scenario: PassSuppressFormatter emits full output for a failing scenario
|
||||
Given behave_parallel a PassSuppressFormatter backed by a captured stream
|
||||
When behave_parallel I simulate a failing scenario through the formatter
|
||||
Then behave_parallel the formatter real stream output should contain "Failing scenario"
|
||||
And behave_parallel the formatter real stream output should contain "I fail"
|
||||
And behave_parallel the formatter real stream output should contain "AssertionError"
|
||||
|
||||
Scenario: PassSuppressFormatter only shows failed scenarios in a mixed run
|
||||
Given behave_parallel a PassSuppressFormatter backed by a captured stream
|
||||
When behave_parallel I simulate one passing then one failing scenario through the formatter
|
||||
Then behave_parallel the formatter real stream output should not contain "Passing scenario"
|
||||
And behave_parallel the formatter real stream output should contain "Failing scenario"
|
||||
|
||||
@@ -16,12 +16,18 @@ Feature: LSP Resource Types
|
||||
|
||||
Examples:
|
||||
| type_name |
|
||||
| executable |
|
||||
| lsp-server |
|
||||
| lsp-workspace |
|
||||
| lsp-document |
|
||||
|
||||
# ── User-addable flags ────────────────────────────────────────
|
||||
|
||||
Scenario: executable is user-addable for lsp_rt
|
||||
Given the built-in executable YAML file for lsp_rt
|
||||
When I load the YAML via schema for lsp_rt
|
||||
Then the lsp_rt schema user_addable should be true
|
||||
|
||||
Scenario: lsp-server is user-addable for lsp_rt
|
||||
Given the built-in lsp-server YAML file for lsp_rt
|
||||
When I load the YAML via schema for lsp_rt
|
||||
@@ -45,6 +51,12 @@ Feature: LSP Resource Types
|
||||
Then the lsp_rt capability read should be true
|
||||
And the lsp_rt capability write should be true
|
||||
|
||||
Scenario: executable has read-only capabilities for lsp_rt
|
||||
Given the built-in executable YAML file for lsp_rt
|
||||
When I load the YAML via schema for lsp_rt
|
||||
Then the lsp_rt capability read should be true
|
||||
And the lsp_rt capability write should be false
|
||||
|
||||
# ── Parent/child hierarchy ────────────────────────────────────
|
||||
|
||||
Scenario: lsp-server has lsp-workspace as child type for lsp_rt
|
||||
@@ -69,6 +81,12 @@ Feature: LSP Resource Types
|
||||
|
||||
# ── Auto-discovery ────────────────────────────────────────────
|
||||
|
||||
Scenario: executable has auto-discovery from container-exec-env for lsp_rt
|
||||
Given the built-in executable YAML file for lsp_rt
|
||||
When I load the YAML via schema for lsp_rt
|
||||
Then the lsp_rt auto_discovery should be present
|
||||
And the lsp_rt auto_discovery trigger_types should contain "container-exec-env"
|
||||
|
||||
Scenario: lsp-workspace has auto-discovery from lsp-server for lsp_rt
|
||||
Given the built-in lsp-workspace YAML file for lsp_rt
|
||||
When I load the YAML via schema for lsp_rt
|
||||
@@ -79,7 +97,8 @@ Feature: LSP Resource Types
|
||||
|
||||
Scenario: BUILTIN_NAMES includes all LSP types for lsp_rt
|
||||
Given the ResourceTypeSpec BUILTIN_NAMES set for lsp_rt
|
||||
Then BUILTIN_NAMES should contain "lsp-server" for lsp_rt
|
||||
Then BUILTIN_NAMES should contain "executable" for lsp_rt
|
||||
And BUILTIN_NAMES should contain "lsp-server" for lsp_rt
|
||||
And BUILTIN_NAMES should contain "lsp-workspace" for lsp_rt
|
||||
And BUILTIN_NAMES should contain "lsp-document" for lsp_rt
|
||||
|
||||
@@ -87,6 +106,9 @@ Feature: LSP Resource Types
|
||||
|
||||
Scenario: LSP types survive DB roundtrip after bootstrap for lsp_rt
|
||||
Given a lsp_rt fresh in-memory resource registry with bootstrap
|
||||
When I query the lsp_rt registry for type "executable"
|
||||
Then the lsp_rt db type "executable" should exist
|
||||
And the lsp_rt db type "executable" should have kind "physical"
|
||||
When I query the lsp_rt registry for type "lsp-server"
|
||||
Then the lsp_rt db type "lsp-server" should exist
|
||||
When I query the lsp_rt registry for type "lsp-workspace"
|
||||
|
||||
@@ -361,7 +361,7 @@ def step_get_entry_count(context):
|
||||
context.entry_count = context.index.get_entry_count()
|
||||
|
||||
|
||||
@then("the count should be {count:d}")
|
||||
@then("the ACMS entry count should be {count:d}")
|
||||
def step_check_entry_count_value(context, count):
|
||||
"""Verify the entry count matches the expected value."""
|
||||
assert context.entry_count == count
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
"""Step definitions for behave_parallel_log_filtering.feature.
|
||||
|
||||
Tests the conditional log replay and worker exception handling in
|
||||
``scripts/run_behave_parallel.py``.
|
||||
``scripts/run_behave_parallel.py``, as well as the PassSuppressFormatter
|
||||
per-scenario output buffering behaviour.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -12,8 +13,11 @@ import sys
|
||||
from contextlib import redirect_stderr, redirect_stdout
|
||||
from pathlib import Path
|
||||
from types import ModuleType
|
||||
from typing import Any
|
||||
|
||||
from behave import given, then, when # type: ignore[import-untyped]
|
||||
from behave.configuration import Configuration # type: ignore[import-untyped]
|
||||
from behave.formatter.base import StreamOpener # type: ignore[import-untyped]
|
||||
from behave.runner import Context # type: ignore[import-untyped]
|
||||
|
||||
|
||||
@@ -52,6 +56,7 @@ _empty_summary = _runner_mod._empty_summary
|
||||
_worker_run_features = _runner_mod._worker_run_features
|
||||
_aggregate_worker_results = _runner_mod._aggregate_worker_results
|
||||
_has_failures = _runner_mod._has_failures
|
||||
PassSuppressFormatter = _runner_mod.PassSuppressFormatter
|
||||
|
||||
|
||||
def _passing_summary(features: int = 1, scenarios: int = 1) -> Summary:
|
||||
@@ -357,3 +362,132 @@ def step_worker_summary_features_errors_1(context: Context) -> None:
|
||||
f"Expected features.errors=1 so _has_failures() detects partial crash, "
|
||||
f"got: {errors}"
|
||||
)
|
||||
|
||||
|
||||
# ---- PassSuppressFormatter helpers ----
|
||||
|
||||
|
||||
class _MockStatus:
|
||||
"""Minimal status object mirroring behave's Status enum interface."""
|
||||
|
||||
def __init__(self, name: str) -> None:
|
||||
self.name = name
|
||||
|
||||
|
||||
class _MockScenario:
|
||||
"""Minimal scenario object for PassSuppressFormatter unit tests."""
|
||||
|
||||
def __init__(self, name: str, status_name: str) -> None:
|
||||
self.keyword = "Scenario"
|
||||
self.name = name
|
||||
self.status = _MockStatus(status_name)
|
||||
|
||||
|
||||
class _MockStep:
|
||||
"""Minimal step object for PassSuppressFormatter unit tests."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
keyword: str,
|
||||
name: str,
|
||||
status_name: str,
|
||||
error_message: str | None,
|
||||
) -> None:
|
||||
self.keyword = keyword
|
||||
self.name = name
|
||||
self.status = _MockStatus(status_name)
|
||||
self.error_message = error_message
|
||||
|
||||
|
||||
def _make_pass_suppress_formatter() -> tuple[Any, io.StringIO]:
|
||||
"""Create a PassSuppressFormatter writing to a fresh StringIO buffer.
|
||||
|
||||
Returns ``(formatter, buffer)`` so callers can inspect what was written
|
||||
to the formatter's real output stream after simulating scenario events.
|
||||
"""
|
||||
buf: io.StringIO = io.StringIO()
|
||||
opener = StreamOpener(stream=buf)
|
||||
config = Configuration(command_args=["-q"])
|
||||
formatter: Any = PassSuppressFormatter(opener, config)
|
||||
return formatter, buf
|
||||
|
||||
|
||||
# ---- PassSuppressFormatter: Given ----
|
||||
|
||||
|
||||
@given("behave_parallel a PassSuppressFormatter backed by a captured stream")
|
||||
def step_create_pass_suppress_formatter(context: Context) -> None:
|
||||
formatter, buf = _make_pass_suppress_formatter()
|
||||
context.bp_formatter = formatter
|
||||
context.bp_formatter_stream = buf
|
||||
|
||||
|
||||
# ---- PassSuppressFormatter: When ----
|
||||
|
||||
|
||||
@when("behave_parallel I simulate a passing scenario through the formatter")
|
||||
def step_simulate_passing_scenario(context: Context) -> None:
|
||||
formatter: Any = context.bp_formatter
|
||||
scenario = _MockScenario("Passing scenario", "passed")
|
||||
step = _MockStep("Given", "I pass", "passed", None)
|
||||
formatter.scenario(scenario)
|
||||
formatter.result(step)
|
||||
formatter.eof()
|
||||
|
||||
|
||||
@when("behave_parallel I simulate a failing scenario through the formatter")
|
||||
def step_simulate_failing_scenario(context: Context) -> None:
|
||||
formatter: Any = context.bp_formatter
|
||||
scenario = _MockScenario("Failing scenario", "failed")
|
||||
step = _MockStep(
|
||||
"When",
|
||||
"I fail",
|
||||
"failed",
|
||||
"AssertionError: expected True got False",
|
||||
)
|
||||
formatter.scenario(scenario)
|
||||
formatter.result(step)
|
||||
formatter.eof()
|
||||
|
||||
|
||||
@when(
|
||||
"behave_parallel I simulate one passing then one failing scenario"
|
||||
" through the formatter"
|
||||
)
|
||||
def step_simulate_mixed_scenarios(context: Context) -> None:
|
||||
formatter: Any = context.bp_formatter
|
||||
# First: a passing scenario.
|
||||
passing_scenario = _MockScenario("Passing scenario", "passed")
|
||||
step1 = _MockStep("Given", "I pass", "passed", None)
|
||||
formatter.scenario(passing_scenario)
|
||||
formatter.result(step1)
|
||||
# Second: a failing scenario. Calling formatter.scenario() here finalises
|
||||
# the previous (passing) scenario, which should be discarded.
|
||||
failing_scenario = _MockScenario("Failing scenario", "failed")
|
||||
step2 = _MockStep("When", "I fail", "failed", "AssertionError: it broke")
|
||||
formatter.scenario(failing_scenario)
|
||||
formatter.result(step2)
|
||||
formatter.eof()
|
||||
|
||||
|
||||
# ---- PassSuppressFormatter: Then ----
|
||||
|
||||
|
||||
@then("behave_parallel the formatter real stream output should be empty")
|
||||
def step_formatter_output_empty(context: Context) -> None:
|
||||
output: str = context.bp_formatter_stream.getvalue()
|
||||
assert output == "", f"Expected empty output, got: {output!r}"
|
||||
|
||||
|
||||
@then('behave_parallel the formatter real stream output should contain "{text}"')
|
||||
def step_formatter_output_contains(context: Context, text: str) -> None:
|
||||
output: str = context.bp_formatter_stream.getvalue()
|
||||
assert text in output, f"Expected {text!r} in formatter output, got: {output!r}"
|
||||
|
||||
|
||||
@then('behave_parallel the formatter real stream output should not contain "{text}"')
|
||||
def step_formatter_output_not_contains(context: Context, text: str) -> None:
|
||||
output: str = context.bp_formatter_stream.getvalue()
|
||||
assert text not in output, (
|
||||
f"Expected {text!r} NOT in formatter output, got: {output!r}"
|
||||
)
|
||||
|
||||
@@ -27,7 +27,8 @@ def _restore_cwd(context):
|
||||
os.environ.pop("CLEVERAGENTS_HOME", None)
|
||||
else:
|
||||
os.environ["CLEVERAGENTS_HOME"] = context._init_original_home
|
||||
shutil.rmtree(context.temp_dir, ignore_errors=True)
|
||||
if getattr(context, "temp_dir", None) is not None:
|
||||
shutil.rmtree(context.temp_dir, ignore_errors=True)
|
||||
|
||||
|
||||
def _create_init_mocks(context):
|
||||
|
||||
@@ -5,7 +5,7 @@ from typing import Any
|
||||
|
||||
from behave import given, then, when
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[3]
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[2]
|
||||
AGENT_DEF_PATH = PROJECT_ROOT / ".opencode" / "agents" / "implementation-supervisor.md"
|
||||
|
||||
|
||||
|
||||
@@ -111,6 +111,13 @@ def _install_behave_parallel(session: nox.Session) -> None:
|
||||
runner_script = Path(__file__).parent / "scripts" / "run_behave_parallel.py"
|
||||
(pkg_dir / "cli.py").write_text(runner_script.read_text(encoding="utf-8"))
|
||||
|
||||
formatter_script = (
|
||||
Path(__file__).parent / "scripts" / "behave_pass_suppress_formatter.py"
|
||||
)
|
||||
(pkg_dir / "behave_pass_suppress_formatter.py").write_text(
|
||||
formatter_script.read_text(encoding="utf-8")
|
||||
)
|
||||
|
||||
setup_path = source_dir / "setup.py"
|
||||
setup_path.write_text(
|
||||
"from setuptools import find_packages, setup\n"
|
||||
|
||||
@@ -26,6 +26,15 @@ def _load_runner_module() -> ModuleType:
|
||||
first or modify the path to be anchored relative to ``__file__``.
|
||||
"""
|
||||
script_path = Path("scripts") / "run_behave_parallel.py"
|
||||
# Ensure the scripts/ directory is on sys.path so that
|
||||
# ``run_behave_parallel.py`` can ``from behave_pass_suppress_formatter
|
||||
# import PassSuppressFormatter``. This is necessary when the helper is
|
||||
# invoked outside of a nox session (e.g. integration tests), where the
|
||||
# behave_parallel package created by noxfile.py for unit_tests is not
|
||||
# available.
|
||||
scripts_dir = str(script_path.parent.resolve())
|
||||
if scripts_dir not in sys.path:
|
||||
sys.path.insert(0, scripts_dir)
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"run_behave_parallel", str(script_path)
|
||||
)
|
||||
|
||||
@@ -15,30 +15,26 @@ def _import_lsp_types() -> None:
|
||||
LSP_RESOURCE_TYPES,
|
||||
)
|
||||
|
||||
assert len(LSP_RESOURCE_TYPES) == 3, f"Expected 3, got {len(LSP_RESOURCE_TYPES)}"
|
||||
assert len(LSP_RESOURCE_TYPES) == 4, f"Expected 4, got {len(LSP_RESOURCE_TYPES)}"
|
||||
names = [t["name"] for t in LSP_RESOURCE_TYPES]
|
||||
for expected in ("lsp-server", "lsp-workspace", "lsp-document"):
|
||||
for expected in ("executable", "lsp-server", "lsp-workspace", "lsp-document"):
|
||||
assert expected in names, f"{expected} not in LSP_RESOURCE_TYPES: {names}"
|
||||
|
||||
print("import-lsp-types-ok")
|
||||
|
||||
|
||||
def _check_builtin_names() -> None:
|
||||
"""Verify all 3 LSP types are in BUILTIN_NAMES."""
|
||||
"""Verify all 4 LSP types are in BUILTIN_NAMES."""
|
||||
from cleveragents.domain.models.core.resource_type import ResourceTypeSpec
|
||||
|
||||
for name in ("lsp-server", "lsp-workspace", "lsp-document"):
|
||||
for name in ("executable", "lsp-server", "lsp-workspace", "lsp-document"):
|
||||
assert name in ResourceTypeSpec.BUILTIN_NAMES, f"{name} not in BUILTIN_NAMES"
|
||||
|
||||
assert "executable" not in ResourceTypeSpec.BUILTIN_NAMES, (
|
||||
"executable should not be in BUILTIN_NAMES (not a spec-defined type)"
|
||||
)
|
||||
|
||||
print("check-builtin-names-ok")
|
||||
|
||||
|
||||
def _db_roundtrip() -> None:
|
||||
"""Bootstrap in-memory DB and verify all 3 LSP types are retrievable."""
|
||||
"""Bootstrap in-memory DB and verify all 4 LSP types are retrievable."""
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
@@ -52,7 +48,7 @@ def _db_roundtrip() -> None:
|
||||
factory = sessionmaker(bind=engine, expire_on_commit=False)
|
||||
svc = ResourceRegistryService(session_factory=factory)
|
||||
|
||||
for name in ("lsp-server", "lsp-workspace", "lsp-document"):
|
||||
for name in ("executable", "lsp-server", "lsp-workspace", "lsp-document"):
|
||||
spec = svc.show_type(name)
|
||||
assert spec is not None, f"{name} not found after bootstrap"
|
||||
assert str(spec.resource_kind) == "physical", (
|
||||
@@ -89,27 +85,32 @@ def _check_hierarchy() -> None:
|
||||
assert "lsp-workspace" in types["lsp-document"]["parent_types"], (
|
||||
"lsp-document missing lsp-workspace parent"
|
||||
)
|
||||
# executable has no children
|
||||
assert types["executable"]["child_types"] == [], (
|
||||
"executable should have no children"
|
||||
)
|
||||
|
||||
print("check-hierarchy-ok")
|
||||
|
||||
|
||||
def _check_auto_discovery() -> None:
|
||||
"""Verify lsp-workspace has auto_discovery with trigger_types."""
|
||||
"""Verify executable has auto_discovery with trigger_types."""
|
||||
from cleveragents.application.services._resource_registry_lsp import (
|
||||
LSP_RESOURCE_TYPES,
|
||||
)
|
||||
|
||||
types = {t["name"] for t in LSP_RESOURCE_TYPES}
|
||||
types = {t["name"]: t for t in LSP_RESOURCE_TYPES}
|
||||
|
||||
# executable should not be present
|
||||
assert "executable" not in types, (
|
||||
"executable should not be in LSP_RESOURCE_TYPES (not a spec-defined type)"
|
||||
# executable has auto_discovery
|
||||
ad = types["executable"].get("auto_discovery")
|
||||
assert ad is not None, "executable missing auto_discovery"
|
||||
assert "container-exec-env" in ad.get("trigger_types", []), (
|
||||
"executable missing container-exec-env trigger"
|
||||
)
|
||||
|
||||
lsp_types = {t["name"]: t for t in LSP_RESOURCE_TYPES}
|
||||
assert ad.get("lazy") is True, "executable auto_discovery should be lazy"
|
||||
|
||||
# lsp-workspace has auto_discovery from lsp-server
|
||||
ad_ws = lsp_types["lsp-workspace"].get("auto_discovery")
|
||||
ad_ws = types["lsp-workspace"].get("auto_discovery")
|
||||
assert ad_ws is not None, "lsp-workspace missing auto_discovery"
|
||||
assert "lsp-server" in ad_ws.get("trigger_types", []), (
|
||||
"lsp-workspace missing lsp-server trigger"
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
"""PassSuppressFormatter — a custom Behave formatter that suppresses output
|
||||
for passing and skipped scenarios, reducing CI log noise from ~100,000 lines
|
||||
to ≤ 10 lines for an all-passing suite.
|
||||
|
||||
This module is embedded in the same directory as ``run_behave_parallel.py``
|
||||
so that ``_install_behave_parallel()`` in ``noxfile.py`` can copy both files
|
||||
into the temporary ``behave_parallel`` package.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
from typing import Any
|
||||
|
||||
from behave.formatter.base import (
|
||||
Formatter as _BehaveFormatter, # type: ignore[import-untyped]
|
||||
)
|
||||
|
||||
# Sentinel set of statuses whose output should be suppressed.
|
||||
# Every status not in this set (e.g. "failed", "undefined") causes the
|
||||
# buffered scenario output to be flushed to the real output stream.
|
||||
_SUPPRESS_STATUSES: frozenset[str] = frozenset({"passed", "skipped"})
|
||||
|
||||
|
||||
class PassSuppressFormatter(_BehaveFormatter):
|
||||
"""Behave formatter that suppresses output for passing scenarios.
|
||||
|
||||
All per-scenario output is buffered in a :class:`io.StringIO`. When a
|
||||
scenario ends (signalled by the next :meth:`scenario` call or by
|
||||
:meth:`eof`):
|
||||
|
||||
* **Failed / errored scenario** — the buffer is flushed to the real
|
||||
output stream so developers and CI tools see the full step-level
|
||||
details.
|
||||
* **Passed / skipped scenario** — the buffer is silently discarded.
|
||||
|
||||
The result for an all-passing suite is zero formatter output, leaving
|
||||
only the ``_print_overall_summary`` block emitted by the runner as
|
||||
visible output (~5-10 lines regardless of suite size).
|
||||
|
||||
Coverage mode (``BEHAVE_PARALLEL_COVERAGE=1``) bypasses this formatter
|
||||
entirely via ``_make_runner()``, which falls back to behave's default
|
||||
format so that slipcover can instrument a single sequential process
|
||||
without output interference.
|
||||
"""
|
||||
|
||||
name: str = "pass_suppress"
|
||||
description: str = "Suppress passing scenario output; show failures fully"
|
||||
|
||||
def __init__(self, stream_opener: Any, config: Any) -> None:
|
||||
super().__init__(stream_opener, config)
|
||||
# Ensure the real output stream is open (opens it if stream_opener
|
||||
# was constructed with only a filename, not a pre-opened stream).
|
||||
self.stream = self.open()
|
||||
|
||||
# Per-scenario buffering state.
|
||||
self._current_scenario: Any = None
|
||||
self._scenario_buf: io.StringIO = io.StringIO()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Internal helpers
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _finalize_previous_scenario(self) -> None:
|
||||
"""Flush the scenario buffer to the real stream if scenario failed.
|
||||
|
||||
Called at the start of each new scenario and at end-of-feature so
|
||||
the previous scenario's buffered output is either committed or
|
||||
discarded based on the scenario's final status.
|
||||
"""
|
||||
if self._current_scenario is None:
|
||||
return
|
||||
status: Any = getattr(self._current_scenario, "status", None)
|
||||
status_name: str = (
|
||||
getattr(status, "name", str(status)) if status is not None else ""
|
||||
)
|
||||
if status_name not in _SUPPRESS_STATUSES:
|
||||
output = self._scenario_buf.getvalue()
|
||||
if output:
|
||||
self.stream.write(output)
|
||||
self.stream.flush()
|
||||
# Reset the buffer regardless — the next scenario starts fresh.
|
||||
self._scenario_buf = io.StringIO()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Formatter interface (behave lifecycle hooks)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def uri(self, uri: str) -> None:
|
||||
"""Called before each feature file; no output needed."""
|
||||
|
||||
def feature(self, feature: Any) -> None:
|
||||
"""Called at feature start; no output needed."""
|
||||
|
||||
def background(self, background: Any) -> None:
|
||||
"""Called when a feature background is declared; no output needed."""
|
||||
|
||||
def scenario(self, scenario: Any) -> None:
|
||||
"""Start buffering output for *scenario*, finalising the previous one."""
|
||||
self._finalize_previous_scenario()
|
||||
self._current_scenario = scenario
|
||||
# Write the scenario header into the new buffer.
|
||||
self._scenario_buf.write(f"\n {scenario.keyword}: {scenario.name}\n")
|
||||
|
||||
def step(self, step: Any) -> None:
|
||||
"""Step announced before execution; no output needed at this point."""
|
||||
|
||||
def match(self, match: Any) -> None:
|
||||
"""Step matched; no output needed."""
|
||||
|
||||
def result(self, step: Any) -> None:
|
||||
"""Record a step result in the per-scenario buffer."""
|
||||
status: Any = getattr(step, "status", None)
|
||||
status_name: str = (
|
||||
getattr(status, "name", "unknown") if status is not None else "unknown"
|
||||
)
|
||||
self._scenario_buf.write(f" {step.keyword} {step.name} ... {status_name}\n")
|
||||
error_msg: str | None = getattr(step, "error_message", None)
|
||||
if error_msg:
|
||||
self._scenario_buf.write(f"{error_msg}\n")
|
||||
|
||||
def eof(self) -> None:
|
||||
"""End of feature file: finalise the current (last) scenario."""
|
||||
self._finalize_previous_scenario()
|
||||
self._current_scenario = None
|
||||
@@ -1,20 +1,16 @@
|
||||
"""In-process parallel behave runner.
|
||||
|
||||
Replaces the old subprocess-per-feature model with direct use of
|
||||
behave's ``Runner`` API. Step definitions and environment hooks are
|
||||
loaded once per process; feature files are parsed and executed without
|
||||
Python interpreter startup overhead.
|
||||
Uses behave's ``Runner`` API directly: step definitions and environment
|
||||
hooks are loaded once per process; feature files are parsed and executed
|
||||
without Python interpreter startup overhead.
|
||||
|
||||
Parallelism modes
|
||||
-----------------
|
||||
* **Sequential** (``--processes 1`` or ``BEHAVE_PARALLEL_COVERAGE=1``):
|
||||
All features run in a single ``Runner.run()`` call.
|
||||
* **Parallel** (``--processes N``, N > 1, no coverage):
|
||||
Features are split into *N* equal-size chunks. A
|
||||
``multiprocessing.Pool`` with the ``fork`` start method dispatches
|
||||
each chunk to a worker that creates its own ``Runner`` (hooks and
|
||||
step definitions are re-loaded cheaply because all heavy modules are
|
||||
already in memory from the parent).
|
||||
Features split into *N* equal-size chunks dispatched via
|
||||
``multiprocessing.Pool`` with the ``fork`` start method.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -31,6 +27,13 @@ from contextlib import redirect_stderr, redirect_stdout, suppress
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
try:
|
||||
from behave_pass_suppress_formatter import PassSuppressFormatter
|
||||
except ImportError:
|
||||
from behave_parallel.behave_pass_suppress_formatter import ( # type: ignore[import-untyped,unused-ignore]
|
||||
PassSuppressFormatter,
|
||||
)
|
||||
|
||||
DEFAULT_FEATURE_ROOT = "features/"
|
||||
|
||||
|
||||
@@ -79,6 +82,7 @@ def _is_btrfs_or_overlayfs() -> bool:
|
||||
|
||||
# Type alias for summary dictionaries
|
||||
Summary = dict[str, Any]
|
||||
WorkerResult = tuple[bool, str, str, Summary]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -174,10 +178,7 @@ def _format_duration(seconds: float) -> str:
|
||||
return f"{remainder:.3f}s"
|
||||
|
||||
|
||||
def _print_overall_summary(
|
||||
total: Summary,
|
||||
wall_seconds: float | None = None,
|
||||
) -> None:
|
||||
def _print_overall_summary(total: Summary, wall_seconds: float | None = None) -> None:
|
||||
print("\nOverall summary:")
|
||||
for bucket in ("features", "scenarios", "steps"):
|
||||
b = total[bucket]
|
||||
@@ -272,35 +273,50 @@ def _make_runner(feature_paths: list[str], behave_args: list[str]) -> Any:
|
||||
Mirrors the format-defaulting logic from ``behave.__main__.run_behave``
|
||||
so that ``-q`` and bare invocations get a sensible formatter instead
|
||||
of crashing on ``config.format is None``.
|
||||
|
||||
When no explicit ``--format``/``-f`` flag is present in *behave_args* and
|
||||
``BEHAVE_PARALLEL_COVERAGE`` is **not** set, the runner uses
|
||||
:class:`PassSuppressFormatter` so that passing scenarios produce no
|
||||
output. Coverage mode falls back to ``config.default_format`` (normally
|
||||
``pretty``) so that slipcover can instrument a single sequential process
|
||||
without output interference.
|
||||
"""
|
||||
from behave.configuration import Configuration
|
||||
from behave.formatter._registry import register_as
|
||||
from behave.runner import Runner
|
||||
from behave.runner_util import reset_runtime
|
||||
|
||||
reset_runtime()
|
||||
# Register our custom formatter so behave can resolve it by name when
|
||||
# make_formatters() is called during Runner initialisation.
|
||||
register_as(PassSuppressFormatter.name, PassSuppressFormatter)
|
||||
|
||||
args = list(behave_args) + [str(p) for p in feature_paths]
|
||||
config = Configuration(command_args=args)
|
||||
if not config.format:
|
||||
config.format = [config.default_format]
|
||||
# No explicit format was requested by the caller. Use pass_suppress
|
||||
# unless coverage instrumentation is active (BEHAVE_PARALLEL_COVERAGE=1),
|
||||
# where slipcover needs unmodified output from a single sequential process.
|
||||
coverage_mode = bool(os.environ.get("BEHAVE_PARALLEL_COVERAGE"))
|
||||
if coverage_mode:
|
||||
config.format = [config.default_format]
|
||||
else:
|
||||
config.format = [PassSuppressFormatter.name]
|
||||
return Runner(config)
|
||||
|
||||
|
||||
def _run_features_inprocess(
|
||||
feature_paths: list[str], behave_args: list[str]
|
||||
) -> tuple[bool, Summary]:
|
||||
def _run_features_inprocess(paths: list[str], args: list[str]) -> tuple[bool, Summary]:
|
||||
"""Run *all* feature_paths in a single behave Runner invocation.
|
||||
|
||||
Returns ``(failed: bool, summary: dict)``.
|
||||
"""
|
||||
runner = _make_runner(feature_paths, behave_args)
|
||||
runner = _make_runner(paths, args)
|
||||
failed = runner.run()
|
||||
summary = _extract_summary(runner)
|
||||
return failed, summary
|
||||
|
||||
|
||||
def _worker_run_features(
|
||||
payload: tuple[list[str], list[str]],
|
||||
) -> tuple[bool, str, str, Summary]:
|
||||
def _worker_run_features(payload: tuple[list[str], list[str]]) -> WorkerResult:
|
||||
"""Entry point for multiprocessing workers.
|
||||
|
||||
Runs a chunk of feature files in a forked child process. Heavy
|
||||
@@ -348,9 +364,7 @@ def _worker_run_features(
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _aggregate_worker_results(
|
||||
results: list[tuple[bool, str, str, Summary]],
|
||||
) -> Summary:
|
||||
def _aggregate_worker_results(results: list[WorkerResult]) -> Summary:
|
||||
"""Aggregate worker results, replaying logs only for failed chunks.
|
||||
|
||||
Iterates over the list of ``(worker_failed, stdout, stderr, summary)``
|
||||
|
||||
@@ -10,11 +10,12 @@ Based on issue #9579 and ``docs/specification.md`` ~lines 44405-44420.
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterator
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from enum import StrEnum
|
||||
from pathlib import Path
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class FileType(StrEnum):
|
||||
"""File type enumeration for index entries."""
|
||||
@@ -45,8 +46,7 @@ class TierLevel(StrEnum):
|
||||
ARCHIVE = "archive" # Reference
|
||||
|
||||
|
||||
@dataclass
|
||||
class IndexEntry:
|
||||
class IndexEntry(BaseModel):
|
||||
"""Represents a single indexed context entry.
|
||||
|
||||
Attributes:
|
||||
@@ -65,9 +65,9 @@ class IndexEntry:
|
||||
size_bytes: int
|
||||
created_at: datetime
|
||||
modified_at: datetime
|
||||
tags: set[str] = field(default_factory=set)
|
||||
tags: set[str] = Field(default_factory=set)
|
||||
tier: TierLevel = TierLevel.COLD
|
||||
metadata: dict[str, str] = field(default_factory=dict)
|
||||
metadata: dict[str, str] = Field(default_factory=dict)
|
||||
|
||||
def add_tag(self, tag: str) -> None:
|
||||
"""Add a tag to this entry.
|
||||
@@ -104,7 +104,6 @@ class IndexEntry:
|
||||
self.tier = tier
|
||||
|
||||
|
||||
@dataclass
|
||||
class ACMSIndex:
|
||||
"""ACMS Index for storing and querying indexed context entries.
|
||||
|
||||
@@ -115,7 +114,8 @@ class ACMSIndex:
|
||||
entries: Dictionary mapping file paths to IndexEntry objects
|
||||
"""
|
||||
|
||||
entries: dict[str, IndexEntry] = field(default_factory=dict)
|
||||
def __init__(self) -> None:
|
||||
self.entries: dict[str, IndexEntry] = {}
|
||||
|
||||
def add_entry(self, entry: IndexEntry) -> None:
|
||||
"""Add an index entry to the index.
|
||||
|
||||
@@ -1,14 +1,15 @@
|
||||
"""LSP resource type definitions for the Resource Registry.
|
||||
|
||||
Contains the built-in LSP-related type entries (``lsp-server``,
|
||||
``lsp-workspace``, ``lsp-document``) that are appended to
|
||||
``BUILTIN_TYPES`` in ``_resource_registry_data``.
|
||||
Contains the built-in LSP-related type entries (``executable``,
|
||||
``lsp-server``, ``lsp-workspace``, ``lsp-document``) that are appended
|
||||
to ``BUILTIN_TYPES`` in ``_resource_registry_data``.
|
||||
|
||||
These types bridge the resource system with the LSP runtime, allowing
|
||||
actors to discover available language tooling and the LSP system to know
|
||||
which resources it operates on.
|
||||
|
||||
Spec reference: docs/adr/ADR-040-lsp-resource-types.md (lsp-server,
|
||||
Spec reference: docs/adr/ADR-039-container-resource-types.md (executable),
|
||||
docs/adr/ADR-040-lsp-resource-types.md (lsp-server,
|
||||
lsp-workspace, lsp-document)
|
||||
|
||||
.. note::
|
||||
@@ -27,11 +28,52 @@ __all__ = ["LSP_RESOURCE_TYPES"]
|
||||
# === LSP Resource Types (#832) ===
|
||||
#
|
||||
# These types model language tooling infrastructure in the resource DAG.
|
||||
# - executable: system binary / interpreter / LSP server binary
|
||||
# - lsp-server: a configured or running LSP server instance
|
||||
# - lsp-workspace: workspace root tracked by an LSP server
|
||||
# - lsp-document: text document tracked by an LSP server
|
||||
|
||||
LSP_RESOURCE_TYPES: list[dict[str, Any]] = [
|
||||
# ── executable ───────────────────────────────────────────────
|
||||
{
|
||||
"name": "executable",
|
||||
"description": (
|
||||
"System executable (language runtime, compiler, LSP server "
|
||||
"binary). Discovered from container or host PATH."
|
||||
),
|
||||
"resource_kind": "physical",
|
||||
"sandbox_strategy": "none",
|
||||
"user_addable": True,
|
||||
"built_in": True,
|
||||
"cli_args": [
|
||||
{
|
||||
"name": "path",
|
||||
"type": "string",
|
||||
"required": True,
|
||||
"description": "Absolute path to the executable",
|
||||
},
|
||||
{
|
||||
"name": "version",
|
||||
"type": "string",
|
||||
"required": False,
|
||||
"description": "Executable version if detectable",
|
||||
},
|
||||
],
|
||||
"parent_types": ["container-exec-env", "fs-directory"],
|
||||
"child_types": [],
|
||||
"handler": "cleveragents.resource.handlers.executable:ExecutableHandler",
|
||||
"auto_discovery": {
|
||||
"trigger_types": ["container-exec-env", "fs-directory"],
|
||||
"scan_paths": [],
|
||||
"lazy": True,
|
||||
},
|
||||
"capabilities": {
|
||||
"read": True,
|
||||
"write": False,
|
||||
"sandbox": False,
|
||||
"checkpoint": False,
|
||||
},
|
||||
},
|
||||
# ── lsp-server ───────────────────────────────────────────────
|
||||
{
|
||||
"name": "lsp-server",
|
||||
|
||||
@@ -125,6 +125,7 @@ BUILTIN_TYPE_NAMES: frozenset[str] = frozenset(
|
||||
"submodule",
|
||||
"symlink",
|
||||
# LSP resource types (#832)
|
||||
"executable",
|
||||
"lsp-server",
|
||||
"lsp-workspace",
|
||||
"lsp-document",
|
||||
|
||||
Reference in New Issue
Block a user