- Replaced the e2e_tests nox session runner from sequential robot to the parallel pabot runner to shrink CI push times.
- Added _split_pabot_args and _pabot_parallel_args support to the e2e_tests session, mirroring the pattern used by integration_tests for consistent parallel execution control.
- CI workflow: added TEST_PROCESSES: "4" environment variable to the e2e_tests job to run four parallel workers. Rationale: E2E suites are IO-bound (LLM API calls) rather than CPU-bound, so increasing parallelism reduces wall-clock time without CPU contention.
- Updated the e2e_tests docstring to document parallelism control via the TEST_PROCESSES environment variable or the --processes positional argument.
- Pre-compiled bytecode comment updated to explain thundering-herd prevention for parallel workers.
- Template DB comment updated to explain the criticality of proper initialization for parallel pabot execution.
Key design decisions:
- Used pabot (robotframework-pabot>=4.0.0), a project-provided dependency, instead of introducing new tooling.
- Set TEST_PROCESSES=4 in CI (instead of the default min(cpu,2)) because E2E tasks are IO-bound and benefit from higher concurrency without CPU contention.
- Suite-level isolation via E2E Suite Setup (separate CLEVERAGENTS_HOME per suite) ensures workers do not share database state, enabling safe parallelism.
- Followed the exact same pattern as integration_tests for consistency and predictability.
Modules/components affected:
- e2e_tests nox session (parallelization logic and arg parsing)
- CI workflow (TEST_PROCESSES environment variable)
- E2E session docstring and related comments (documentation of parallelism and initialization)
- Inline comments for pre-compiled bytecode and template database initialization to reflect parallel execution considerations
ISSUES CLOSED: #1860
Both slow_integration_tests and e2e_tests sessions require a database
(via setup_workspace() in helper scripts) but were not using the
pre-migrated template DB optimization already present in unit_tests,
integration_tests, and coverage_report.
Changes:
- slow_integration_tests: add _create_template_db() call, set
CLEVERAGENTS_TEMPLATE_DB env var, upgrade from robot to pabot for
parallel execution, add NO_COLOR/PYTHONPATH/PATH/compileall setup,
add --include slow / --exclude discovery/code_blocks/wip/E2E/tdd_fixture
tag filters, and update docstring.
- e2e_tests: add _create_template_db() call and set
CLEVERAGENTS_TEMPLATE_DB env var. The existing setup_workspace()
in helper_e2e_common.py already checks this env var and copies the
template instead of running 25+ Alembic migrations.
No changes to test helper files are required since setup_workspace()
already implements the fast-path copy logic when CLEVERAGENTS_TEMPLATE_DB
is set.
ISSUES CLOSED: #2334
Move the large embedded `_BEHAVE_PARALLEL_CLI_SOURCE` string constant out of
noxfile.py and into a standalone `scripts/run_behave_parallel.py` module.
The `_install_behave_parallel()` helper now reads the script from disk via
`Path(__file__).parent / 'scripts' / 'run_behave_parallel.py'` instead of
embedding the source as a raw string literal. This allows ruff to lint and
type-check the runner independently, and makes noxfile.py significantly
shorter and easier to read.
No functional changes: the installed `behave-parallel` entry point is
identical to the previous embedded version. Parallel and sequential modes,
coverage integration, and the multiprocessing fork model are all preserved.
Fixed two SIM105 lint violations in the extracted script (replaced
try/except/pass with contextlib.suppress).
ISSUES CLOSED: #1538
Add the missing plan prompt CLI command and wire it through the local A2A facade into PlanLifecycleService so operator guidance can be injected into active execute-phase plans per the specification. The lifecycle flow now validates active state, records user-intervention decisions, and returns structured queue/decision metadata for rich and machine-readable output envelopes across all supported formats.
Also stabilize flaky quality gates discovered while implementing #885 by hardening integration helper timeouts, relaxing an overly strict CLI-core timing assertion, defaulting test worker concurrency to serial for determinism, switching ASV to spawn launch mode for runner stability, and documenting the controlled transform sandbox exec/compile path for static security hooks.
ISSUES CLOSED: #885
Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
## Summary
Add M6 parallel-scaling coverage for 10+ concurrent subplans:
- **15-subplan parallel scenario** with explicit peak-concurrency bound checks (`max_parallel=10`) and thread-safe concurrency tracking via `_build_executor()`.
- **Deep hierarchical decomposition** coverage (4+ levels) with adjusted leaf condition that only stops early when hitting `max_depth` or when the workset is trivially small (`min_files_per_subplan`).
- **Non-progress guard** in `_build_hierarchy` to prevent pathological recursion when clustering cannot meaningfully split the file set.
- **Small-project regression test** (< 50 files) verifying decomposition depth does not increase unexpectedly with the relaxed leaf condition.
- **ASV benchmark** for 15-subplan parallel execution with `max_parallel=10` to track scaling behavior.
### Removed from this PR
The `_build_hierarchy` child-linkage correctness fix (returning `node_id` from recursive calls instead of using `nodes[-1].node_id`) has been **removed** per review feedback — it is a separate bug fix and will be submitted as an independent issue/PR per CONTRIBUTING.md §Atomic Commits.
## Approach
- **Concurrency tracking:** The `_build_executor()` closure in step definitions detects `context.concurrency_counter` / `context.concurrency_lock` and performs thread-safe peak tracking in a try/finally block.
- **Leaf condition:** Replaced the `max_files_per_subplan` / `max_tokens_per_subplan` leaf check with a `min_files_per_subplan` check to allow deeper decomposition for large projects. Added a non-progress guard so clustering that cannot split the file set terminates immediately rather than recursing to `max_depth`.
- **Deterministic IDs:** `_ids_for_count()` preserves legacy fixed IDs for the first 5 subplans and generates additional deterministic IDs for scale scenarios.
## Validation
### Passing
- `nox -s lint` — all checks passed
- `nox -s typecheck` — 0 errors, 0 warnings
- `nox -s unit_tests` — 12,988 scenarios passed, 0 failed
- `nox -s coverage_report` — 97% (passes `--fail-under=97`)
Closes#855
Reviewed-on: cleveragents/cleveragents-core#1201
Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
## Summary
This PR adds Kubernetes Helm deployment support for the CleverAgents server.
Closes#928
### What this PR includes
- Helm chart under `k8s/` with Deployment, Service, optional Ingress, ConfigMap,
ServiceAccount, Secrets, NOTES, and optional Redis subchart configuration.
- Multi-stage `Dockerfile.server` for server runtime deployment.
- Deployment-focused docs in `k8s/README.md`.
- Behave + Robot + benchmark coverage for chart/deployment wiring.
### Review fixes applied (cycle 11 — hurui200320 review #2687)
**Critical fix:**
1. **CI SHA256 checksum verification fixed** — All 3 Helm install blocks (`unit_tests`, `integration_tests`, `helm` jobs) now save the tarball using its original filename (`helm-v3.16.4-linux-amd64.tar.gz`) instead of `helm.tgz`, so the `.sha256sum` file can correctly locate and verify it. This was causing all 3 CI Helm jobs to fail with "No such file or directory".
**Major fixes (test coverage gaps):**
2. **405 Allow header now tested** — The existing "POST to known path returns method not allowed" scenario now asserts that the `Allow: GET` header is present. Added a second 405 scenario testing `POST /live` for broader `_KNOWN_PATHS` coverage (review item #11).
3. **Security-hardening headers now tested** — New scenario "HTTP responses include security-hardening headers" verifies `content-length`, `x-content-type-options: nosniff`, and `cache-control: no-store` headers are present on HTTP responses.
4. **Lifespan warning logging now tested** — New scenario "Unrecognised lifespan message type logs warning and continues" queues `lifespan.startup` → `lifespan.bogus` → `lifespan.shutdown` and verifies: (a) the app completes the lifespan cycle cleanly, (b) a warning is logged mentioning the unrecognised type.
### Review fixes applied (cycle 10)
**Critical/Major fixes (from hurui200320's prior REQUEST_CHANGES):**
1. **Rebased branch onto master** — Removed merge commit per CONTRIBUTING.md rebase-only policy. Clean linear history restored.
2. **Fixed commit message body** — Replaced literal `\n` sequences with actual newlines. `ISSUES CLOSED: #928` footer is now on its own line after a blank separator.
3. **Moved uvicorn import to module top level** — `from uvicorn import run as uvicorn_run` is now at the top of `src/cleveragents/cli/commands/server.py` per Import Guidelines. Updated test mock target from `uvicorn.run` to `cleveragents.cli.commands.server.uvicorn_run`.
4. **Added SHA256 checksum verification** — All three Helm CLI install blocks in `.forgejo/workflows/ci.yml` now download and verify `helm.sha256sum` before extracting the binary.
**Minor fixes:**
5. **Added `Dockerfile.server` build to CI** — New "Build Docker image (Server)" step in the `docker` job validates the server Dockerfile.
6. **ASGI 405 Method Not Allowed** — Known paths (`/`, `/live`, `/ready`, `/health`) now return 405 with `Allow: GET` header for non-GET methods, per RFC 9110 §15.5.6. Added `_KNOWN_PATHS` frozenset.
7. **WebSocket close protocol fix** — App now calls `await receive()` to consume the `websocket.connect` event before closing. Changed close code from 1000 (Normal Closure) to 1008 (Policy Violation).
8. **Lifespan handler logging** — Unrecognised lifespan message types are now logged as warnings instead of silently consumed.
9. **Security-hardening headers** — `_send_response` now includes `content-length`, `x-content-type-options: nosniff`, and `cache-control: no-store` on all HTTP responses.
10. **`.dockerignore` credential patterns** — Added `*.pem`, `*.key`, `*.p12`, `*.pfx`, `credentials*.json`.
11. **`--log-level` validation** — Constrained to `click.Choice(["critical", "error", "warning", "info", "debug", "trace"])` for clean CLI validation errors.
12. **Reverted unrelated semgrep pre-commit change** — `pass_filenames` and `entry` restored to original values per atomic commit hygiene.
13. **Removed unused `ReceiveCallable` type alias** and `Callable`/`Awaitable` imports from `asgi_app_steps.py`.
14. **Fixed redundant `shutil.which("helm")` check** — `_skip_if_helm_missing` now returns `bool` to eliminate the duplicate check in `_render_chart`.
15. **Improved test deque error handling** — Lifespan test receive mock now raises descriptive `AssertionError` instead of opaque `IndexError`.
16. **Scope type dispatch** — Changed `if/if/if` to `if/elif/elif` for mutually exclusive ASGI scope types.
17. **Dockerfile.server base image** — Standardised to `python:3.13-slim` (floating minor) consistent with CLI Dockerfile.
18. **Dockerfile layer caching** — Split `uv pip install build` and `python -m build` into separate `RUN` instructions.
19. **Removed extraneous double blank line** in Dockerfile.server.
### Deferred items (acknowledged, not in scope)
- PodDisruptionBudget, HorizontalPodAutoscaler, NetworkPolicy — Follow-up for production hardening.
- `appVersion: "1.0.0"` placeholder — Needs tracking issue for release versioning alignment.
- Readiness probe with downstream dependency checks — Documented limitation.
- Cross-system test for probe paths matching ASGI routes — Test enhancement.
- Improved benchmarks (helm template timing vs PyYAML parsing) — Benchmark quality improvement.
- CI DRY violation (Helm install 3×) — Code quality improvement, consider composite action.
- File length limits exceeded (`k8s_helm_chart_steps.py` 551 lines, `helper_k8s_helm_chart.py` 678 lines) — Non-blocking, can be split in follow-up.
- `runAsGroup: 1000` in pod security context — Defense-in-depth improvement.
- HEAD method support on known paths — RFC compliance, does not affect K8s probes.
- `click.Choice` log-level validation via CLI runner test — Test gap.
### Scope note: status-check CI gate
The `status-check` job now includes `integration_tests`, `e2e_tests`, and `helm` in its `needs` list. The `helm` job is new in this PR. The `integration_tests` and `e2e_tests` additions fix previously-missing gate checks — included here since this PR modifies both of those jobs to install Helm.
### Quality gates
- `nox -e lint` ✅
- `nox -e typecheck` ✅
- `nox -e unit_tests` ✅ (12,321 scenarios passed, 4 skipped)
- `nox -e integration_tests` — 3 pre-existing failures in unrelated areas (plan correction, resource types)
- `nox -e e2e_tests` — pre-existing failures (LLM API keys not available in local env)
- `nox -e coverage_report` ✅ (**97.7%**)
Reviewed-on: cleveragents/cleveragents-core#1085
Reviewed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com>
Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Rename the TDD tag system from tdd_bug/tdd_bug_<N> to tdd_issue/tdd_issue_<N>
across the entire codebase. The tdd_expected_fail tag is unchanged.
The TDD expected-failure workflow is not limited to bug fixes — it applies
equally to any issue type (features, tasks, refactors). The _bug suffix was
misleading and narrowed the perceived scope. The new _issue suffix accurately
reflects that the TDD tagging system applies to any Forgejo issue.
Changes span 92 files:
- features/environment.py: validate_tdd_tags(), should_invert_result(), and
apply_tdd_inversion() updated — regex, variables, error messages
- robot/tdd_expected_fail_listener.py: _validate_tdd_tags(), _should_invert_result(),
start_test(), end_test() updated consistently
- 33 Behave .feature files: all @tdd_bug/@tdd_bug_<N> tags renamed
- 29 Robot .robot files: all tdd_bug/tdd_bug_<N> tags renamed
- 3 Robot fixture files renamed (tdd_bug_alone, tdd_missing_tdd_bug,
tdd_expected_fail_missing_bug_n) with content and references updated
- Tag validation tests and helpers updated (function names, command dispatch
keys, output strings, fixture references)
- CONTRIBUTING.md: section renamed from 'TDD Bug Test Tags' to
'TDD Issue Test Tags', all tag references and examples updated
- noxfile.py: comment references updated
- Step definition files, mock helpers, and benchmark files: docstring
references updated
ISSUES CLOSED: #965
## Summary
This PR adds a Robot Framework E2E test suite (`robot/e2e/tdd_acms_behavioral_validation.robot`) that proves bug #1028 exists — the ACMS indexing pipeline is not wired into the CLI, so `ContextTierService` starts empty on every invocation.
### Changes
- **New file**: `robot/e2e/tdd_acms_behavioral_validation.robot` — 4 E2E test cases tagged `tdd_expected_fail`, `tdd_bug`, `tdd_bug_1028`, `E2E`
- **Modified**: `robot/e2e/common_e2e.resource` — Extracted shared keywords (`Run CLI`, `Extract JSON From Stdout`, `Link Resource To Project`, `Create Synthetic Codebase`) from both `m5_acceptance.robot` and `tdd_acms_behavioral_validation.robot` to eliminate ~97 lines of duplication. `Create Synthetic Codebase` is parameterized with `project_label`.
- **Modified**: `robot/e2e/m5_acceptance.robot` — Removed duplicated keywords now provided by `common_e2e.resource`.
- **CHANGELOG.md**: Added entry under `## Unreleased` documenting the TDD tests for #1029.
### Test Cases
- **Test 1**: Context Simulate Returns Non-Empty Tier Data — asserts `fragment_count > 0` (fails, proving bug)
- **Test 2**: Context Inspect Shows Indexed Resources — asserts tier metrics total > 0 (fails, proving bug)
- **Test 3**: Budget Enforcement Excludes Oversized Files — asserts `fragment_count > 0` with `max_file_size` policy (fails, proving bug). Includes TODO comment for post-fix exclusion assertion.
- **Test 4**: Large Project Indexes Without Timeout — generates 10K+ files, asserts `fragment_count > 0` (fails, proving bug). Includes explicit developer responsibility note about git-tracking of generated files.
All tests pass CI through result inversion by the `tdd_expected_fail_listener.py` — failing assertions (bug confirmed) are inverted to PASS.
### Documentation & Robustness Improvements
- Suite-level documentation expanded with **known limitation** section explaining `tdd_expected_fail` result inversion scope.
- Suite setup error messages include `(rc=${var.rc}). Check DEBUG logs above.` for debugging consistency with `m5_acceptance.robot`.
- Redundant exit code assertions after `Run CLI` calls removed (Run CLI already validates rc internally).
- `Run CLI` keyword documentation includes API key security notes.
- Budget enforcement test (Test 3) includes `TODO(bugfix/...)` comment for the bug-fix developer.
- Large project test (Test 4) includes explicit NOTE assigning responsibility to the bug-fix developer to evaluate filesystem vs. git-tracked content indexing.
### Review Fix Round
Addressed all findings from Luis's review (review #2691):
- **C1 (CRITICAL)**: Restored the #845 `CorrectionService` changelog entry (50 lines) accidentally deleted during merge conflict resolution.
- **L1 (LOW)**: Extracted ~97 lines of duplicated keywords into `common_e2e.resource` (parameterized `Create Synthetic Codebase`, shared `Run CLI`, `Extract JSON From Stdout`, `Link Resource To Project`).
- **M1 (MEDIUM)**: Strengthened NOTE comment about 10K files not being git-committed — explicit developer MUST responsibility.
- **L3 (LOW)**: Made suite setup error messages verbose with `(rc=...). Check DEBUG logs above.`
- **L2 (LOW)**: Removed redundant exit code assertions after `Run CLI` calls.
- **M2 (MEDIUM)**: Acknowledged — partial assertion is acceptable for TDD capture phase (TODO documents the gap).
- **I1 (INFO)**: Acknowledged — `tdd_expected_fail` masking is documented and mitigated.
### Motivation
Per the Bug Fix Workflow in CONTRIBUTING.md, this TDD issue (#1029) is the prerequisite for bug fix#1028. The tests capture the buggy behavior so that when the fix is implemented, removing the `tdd_expected_fail` tag will cause the tests to pass normally.
### Quality Gates
| Gate | Result |
|------|--------|
| lint | PASS |
| typecheck | PASS |
| unit_tests | PASS |
| integration_tests | PASS |
| e2e_tests | PASS (41 tests, 4 TDD) |
| coverage_report | PASS (>=97%) |
Closes#1029
Reviewed-on: cleveragents/cleveragents-core#1124
Reviewed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com>
Co-authored-by: Rui Hu <rui.hu@cleverthis.com>
Co-committed-by: Rui Hu <rui.hu@cleverthis.com>
## Summary
TDD expected-fail tests proving bug #932 exists: the `plan apply` (`lifecycle-apply`) command does not accept the `--yes`/`-y` flag required by the specification. The flag should skip the confirmation prompt before applying plan changes.
### Tests Added
**Behave scenarios** (`features/tdd_plan_apply_yes_flag.feature`):
- `lifecycle-apply --yes` should be accepted (tests `--yes` long flag)
- `lifecycle-apply -y` should be accepted (tests `-y` short flag)
Tags: `@tdd_expected_fail @tdd_bug @tdd_bug_932`
**Robot Framework tests** (`robot/tdd_plan_apply_yes_flag.robot`):
- `check-yes-long` — invokes CLI with `--yes`, asserts no "No such option" error
- `check-yes-short` — invokes CLI with `-y`, asserts no "No such option" error
### How the Bug Is Proven
The `lifecycle_apply_plan` function (`cleveragents.cli.commands.plan`) defines only `plan_id` and `--format` parameters — no `--yes`/`-y`. When tests invoke `lifecycle-apply --yes`, Typer/Click rejects it with `"No such option: --yes"` and exit code 2. The assertion that this error is absent **fails**, confirming bug #932. The `@tdd_expected_fail` tag inverts this to a pass.
### Quality Gates
| Session | Result |
|---|---|
| `nox -s lint` | PASS |
| `nox -s typecheck` | PASS (0 errors) |
| `nox -s unit_tests` | PASS (10,808 scenarios) |
| `nox -s integration_tests` | PASS (1,508 tests) |
| `nox -s coverage_report` | 98% (>= 97%) |
Closes#950
Reviewed-on: cleveragents/cleveragents-core#958
Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Implements the three-tag TDD bug-capture system in Robot Framework via a
Listener v3 module, paralleling the Behave implementation. Tests tagged
tdd_expected_fail that fail have their result inverted to PASS (bug still
exists); tests that unexpectedly pass are inverted to FAIL with guidance.
Addresses all 15 findings from code review (PR !673, reviewer hamza.khyari):
P2 fixes:
- Added idempotency guard (_processed_tests set) to prevent double-inversion
when the listener is loaded twice in the same process.
- Rewrote normal-test-unaffected check to run alongside a tdd_expected_fail
fixture in a single Robot invocation, proving the listener is loaded and
selectively applies rather than being a tautological pass.
P3 fixes:
- Added output.xml existence guard with clear diagnostics in _run_fixture.
- Documented intentional use of data.tags (static definition) vs result.tags
(runtime-modifiable) in end_test docstring.
- Added SKIP status test fixture and integration test case.
- Added message content assertion in cmd_expected_fail_inverted.
- Tightened substring assertions to match specific error text.
- Added tdd_expected_fail-alone fixture (both companions missing).
- Added close() hook to clear _validation_errors and _processed_tests.
- Simplified _run_fixture return type to tuple[str, str].
- Changed listener path resolution from CWD-relative to __file__-relative
in noxfile.py (integration_tests, slow_integration_tests, e2e_tests).
P4 fixes:
- Added __all__ declaration to helper module.
- Changed module docstring from "mirroring" to "paralleling".
- Added comment documenting accepted XML parsing risk (self-generated XML).
Additional fixes:
- Increased M4 E2E plan-tree test timeout from 30s to 120s (pre-existing
timeout failure unrelated to this feature).
Quality gates (post-rebase onto latest master):
- nox -s lint: PASS
- nox -s typecheck: PASS (0 errors)
- nox -s unit_tests: PASS (10,700 scenarios)
- nox -s integration_tests: PASS (1,505 tests)
- nox -s coverage_report: PASS (97.9% >= 97% threshold)
- nox -s benchmark: PASS
- nox -s docs: PASS
- nox -s build: PASS
- nox -s security_scan: PASS
- nox -s dead_code: PASS
ISSUES CLOSED: #628
Added dedicated E2E test infrastructure completely separate from the existing
integration test suite. E2E tests use zero mocking — they exercise the real
CleverAgents CLI with real LLM API keys.
Key changes:
- New e2e_tests nox session running Robot Framework with --include E2E tag
filter against robot/e2e/ directory. Uses sequential robot (not pabot)
since E2E tests hit real API endpoints with rate limits. Propagates
ANTHROPIC_API_KEY, OPENAI_API_KEY, and GOOGLE_API_KEY from environment.
Output goes to build/reports/robot-e2e/ to avoid artifact collisions.
- Existing integration_tests session now excludes E2E-tagged tests via
--exclude E2E on the pabot invocation.
- New robot/e2e/common_e2e.resource provides shared E2E keywords: suite
setup/teardown with per-suite isolation (no mock AI), graceful skip when
LLM API keys are absent, CLI runner keyword, flexible output assertions,
and temporary git repo fixture creation.
- Minimal smoke test (robot/e2e/smoke_test.robot) validates the harness by
running agents --version and agents --help. Does not require LLM keys.
- Dedicated e2e_tests CI job in .forgejo/workflows/ci.yml injects LLM API
keys from Forgejo secrets. Runs independently (no needs dependencies)
and does not block regular CI.
- e2e_tests is deliberately NOT in the default nox sessions list since it
requires real API keys not present in all environments.
ISSUES CLOSED: #740
Implement TDD bug-capture tests for bug #570 where `agents session create`
fails because `_get_session_service()` calls `container.db()` which does
not exist on the DI Container class (AttributeError). Same root cause as
bug #554.
Behave BDD scenarios tagged @tdd_bug @tdd_bug_570 @tdd_expected_fail
exercise the real DI path (no mocks). Includes Robot Framework integration
smoke tests with self-inverting helper and ASV benchmark baseline.
ISSUES CLOSED: #631
Add TDD regression tests for bug #570 where `_get_session_service()`
calls `container.db()` but the DI `Container` class has no `db`
provider, raising `AttributeError`. Same root cause as bug #554.
Includes 4 Behave BDD scenarios tagged `@tdd_bug @tdd_bug_570
@tdd_expected_fail`, Robot Framework integration smoke tests with
`--format plain`, and ASV service-layer benchmarks. Tests exercise the
real DI path by resetting `_service = None` and using a file-based
SQLite database.
Implements the `@tdd_expected_fail` inversion infrastructure:
- Behave: `after_scenario` hook in `features/environment.py` inverts
pass/fail for scenarios tagged `@tdd_expected_fail`
- Robot: `robot/tdd_expected_fail_listener.py` listener (API v3)
performs the same inversion for Robot test cases
- `noxfile.py`: registers the listener via `--listener` in both the
`integration_tests` and `slow_integration_tests` sessions
Migrates 18 existing TDD scenarios across 5 feature files from the old
`@tdd @bugNNN` convention to the standardised `@tdd_bug @tdd_bug_NNN`
tags per CONTRIBUTING.md § TDD Bug Test Tags.
Refs: #570
Implement TDD bug-capture tests for bug #554 where `agents session list`
fails because `_get_session_service()` calls `container.db()` which does
not exist on the DI Container class (AttributeError).
Behave BDD scenarios tagged @tdd_bug @tdd_bug_554 @tdd_expected_fail
exercise the real DI path (no mocks) and assert correct behavior. The
@tdd_expected_fail handler in environment.py inverts failed→passed while
the bug is present, keeping CI green.
Also adds:
- @tdd_expected_fail infrastructure in features/environment.py
(tag validation + status inversion in after_scenario hook)
- behave-parallel exit logic fix to use summary-based failure
detection (compatible with TDD status inversion)
- Robot Framework integration smoke tests with self-inverting helper
- ASV benchmark baseline for session list command throughput
ISSUES CLOSED: #630
Replace the subprocess-per-feature execution model (342 Python interpreter
startups) with direct use of behave's Runner API for in-process
execution.
Sequential mode (--processes 1 or BEHAVE_PARALLEL_COVERAGE=1): All
features run in a single Runner.run() call. Steps and hooks load once.
Parallel mode (--processes N, N>1): Features split into N chunks,
dispatched via multiprocessing.Pool with fork. Heavy modules shared
copy-on-write.
Proper format defaulting (mirrors behave.__main__.run_behave() logic
for -q flag). Summary extracted from runner.features status attributes
instead of regex-parsing stdout.
Simplified coverage pipeline: single slipcover invocation wraps the
entire behave-parallel process. No per-worker UUID files, no --merge
step needed. Coverage data produced in one build/coverage.json file.
Removed: behave-parallel tarball download from PyPI, tarfile and
urllib.request imports, per-worker subprocess.run() calls,
__SLIPCOVER_OUT__ placeholder mechanism, _build_base_args(),
_parse_summary(), regex-based summary parsing.
Results: nox -s unit_tests 24m21s -> 2m05s (91%); nox -s coverage_report
75m20s -> 3m00s (96%). Coverage: 98% (above 97% threshold).
ISSUES CLOSED: #481
Created scripts/create_template_db.py that builds a pre-migrated SQLite
template database using Base.metadata.create_all() + alembic stamp
(~5ms for 34 tables, vs ~0.5-3s x 25 Alembic migrations per scenario).
Nox unit_tests and coverage_report sessions generate the template before
test execution and propagate CLEVERAGENTS_TEMPLATE_DB env var to all
workers.
features/environment.py before_all() installs a monkey-patch on
MigrationRunner.init_or_upgrade that copies the template for fresh
scenario temp DBs, falling through to real migrations for :memory:,
existing files, and migration-runner unit tests.
Quick wins: sleep(0.5) -> sleep(0.05) in cli_streaming wait step;
removed redundant Background re-declaration in cli_streaming.feature
scenario 7.
ISSUES CLOSED: #483
Replace coverage.py (sys.settrace-based) with slipcover (bytecode-based
instrumentation) for significantly faster coverage collection:
- Each behave-parallel worker runs under slipcover, producing per-feature
JSON coverage files with unique UUIDs to avoid write contention.
- After all workers finish, slipcover --merge combines per-worker data
into a single build/coverage.json report.
- XML report generated via slipcover --merge --xml for CI tooling.
- Terminal report with --fail-under=97 threshold enforcement.
- Robust JSON key-fallback logic handles both slipcover and coverage.py
output formats.
- CI workflow (ci.yml, nightly-quality.yml) updated with defensive key
lookup instead of hardcoded coverage.py format.
- Documentation updated to reflect slipcover as the coverage tool.
- CHANGELOG.md updated.
ISSUES CLOSED: #482
- Remove AutomationLevel imports from cli_robot_flow_bench.py and
persistence_robot_bench.py (enum was removed by master's automation
refactor); replace with AutomationProfileRef where needed.
- Use typer.echo() instead of console.print() for machine-readable
output (JSON/YAML/plain) in config.py and session.py to prevent
Rich from injecting ANSI escape codes that corrupt json.loads().
- Set NO_COLOR=1 in noxfile unit_tests, integration_tests, and
coverage_report sessions as a belt-and-suspenders safeguard for
all CLI commands that route format_output through Rich.
Add python -m compileall -q src/ step to the integration_tests nox
session before launching pabot. On CI runners with high core counts
(e.g. 32 processes) the first Robot test to spawn a Python subprocess
could fail because 30+ processes simultaneously cold-compile the
entire source tree from scratch. Pre-compiling eliminates the
thundering-herd race and lets all workers read cached .pyc files.
freemo
brent.edwards
hurui200320
CoreRasurae
Brent E. Edwards
mngrif
Jeffrey Phillips Freeman