Implement TDD bug-capture tests for bug #570 where `agents session create`
fails because `_get_session_service()` calls `container.db()` which does
not exist on the DI Container class (AttributeError). Same root cause as
bug #554.
Behave BDD scenarios tagged @tdd_bug @tdd_bug_570 @tdd_expected_fail
exercise the real DI path (no mocks). Includes Robot Framework integration
smoke tests with self-inverting helper and ASV benchmark baseline.
ISSUES CLOSED: #631
Add TDD regression tests for bug #570 where `_get_session_service()`
calls `container.db()` but the DI `Container` class has no `db`
provider, raising `AttributeError`. Same root cause as bug #554.
Includes 4 Behave BDD scenarios tagged `@tdd_bug @tdd_bug_570
@tdd_expected_fail`, Robot Framework integration smoke tests with
`--format plain`, and ASV service-layer benchmarks. Tests exercise the
real DI path by resetting `_service = None` and using a file-based
SQLite database.
Implements the `@tdd_expected_fail` inversion infrastructure:
- Behave: `after_scenario` hook in `features/environment.py` inverts
pass/fail for scenarios tagged `@tdd_expected_fail`
- Robot: `robot/tdd_expected_fail_listener.py` listener (API v3)
performs the same inversion for Robot test cases
- `noxfile.py`: registers the listener via `--listener` in both the
`integration_tests` and `slow_integration_tests` sessions
Migrates 18 existing TDD scenarios across 5 feature files from the old
`@tdd @bugNNN` convention to the standardised `@tdd_bug @tdd_bug_NNN`
tags per CONTRIBUTING.md § TDD Bug Test Tags.
Refs: #570
Implement TDD bug-capture tests for bug #554 where `agents session list`
fails because `_get_session_service()` calls `container.db()` which does
not exist on the DI Container class (AttributeError).
Behave BDD scenarios tagged @tdd_bug @tdd_bug_554 @tdd_expected_fail
exercise the real DI path (no mocks) and assert correct behavior. The
@tdd_expected_fail handler in environment.py inverts failed→passed while
the bug is present, keeping CI green.
Also adds:
- @tdd_expected_fail infrastructure in features/environment.py
(tag validation + status inversion in after_scenario hook)
- behave-parallel exit logic fix to use summary-based failure
detection (compatible with TDD status inversion)
- Robot Framework integration smoke tests with self-inverting helper
- ASV benchmark baseline for session list command throughput
ISSUES CLOSED: #630
Replace the subprocess-per-feature execution model (342 Python interpreter
startups) with direct use of behave's Runner API for in-process
execution.
Sequential mode (--processes 1 or BEHAVE_PARALLEL_COVERAGE=1): All
features run in a single Runner.run() call. Steps and hooks load once.
Parallel mode (--processes N, N>1): Features split into N chunks,
dispatched via multiprocessing.Pool with fork. Heavy modules shared
copy-on-write.
Proper format defaulting (mirrors behave.__main__.run_behave() logic
for -q flag). Summary extracted from runner.features status attributes
instead of regex-parsing stdout.
Simplified coverage pipeline: single slipcover invocation wraps the
entire behave-parallel process. No per-worker UUID files, no --merge
step needed. Coverage data produced in one build/coverage.json file.
Removed: behave-parallel tarball download from PyPI, tarfile and
urllib.request imports, per-worker subprocess.run() calls,
__SLIPCOVER_OUT__ placeholder mechanism, _build_base_args(),
_parse_summary(), regex-based summary parsing.
Results: nox -s unit_tests 24m21s -> 2m05s (91%); nox -s coverage_report
75m20s -> 3m00s (96%). Coverage: 98% (above 97% threshold).
ISSUES CLOSED: #481
Created scripts/create_template_db.py that builds a pre-migrated SQLite
template database using Base.metadata.create_all() + alembic stamp
(~5ms for 34 tables, vs ~0.5-3s x 25 Alembic migrations per scenario).
Nox unit_tests and coverage_report sessions generate the template before
test execution and propagate CLEVERAGENTS_TEMPLATE_DB env var to all
workers.
features/environment.py before_all() installs a monkey-patch on
MigrationRunner.init_or_upgrade that copies the template for fresh
scenario temp DBs, falling through to real migrations for :memory:,
existing files, and migration-runner unit tests.
Quick wins: sleep(0.5) -> sleep(0.05) in cli_streaming wait step;
removed redundant Background re-declaration in cli_streaming.feature
scenario 7.
ISSUES CLOSED: #483
Replace coverage.py (sys.settrace-based) with slipcover (bytecode-based
instrumentation) for significantly faster coverage collection:
- Each behave-parallel worker runs under slipcover, producing per-feature
JSON coverage files with unique UUIDs to avoid write contention.
- After all workers finish, slipcover --merge combines per-worker data
into a single build/coverage.json report.
- XML report generated via slipcover --merge --xml for CI tooling.
- Terminal report with --fail-under=97 threshold enforcement.
- Robust JSON key-fallback logic handles both slipcover and coverage.py
output formats.
- CI workflow (ci.yml, nightly-quality.yml) updated with defensive key
lookup instead of hardcoded coverage.py format.
- Documentation updated to reflect slipcover as the coverage tool.
- CHANGELOG.md updated.
ISSUES CLOSED: #482
- Remove AutomationLevel imports from cli_robot_flow_bench.py and
persistence_robot_bench.py (enum was removed by master's automation
refactor); replace with AutomationProfileRef where needed.
- Use typer.echo() instead of console.print() for machine-readable
output (JSON/YAML/plain) in config.py and session.py to prevent
Rich from injecting ANSI escape codes that corrupt json.loads().
- Set NO_COLOR=1 in noxfile unit_tests, integration_tests, and
coverage_report sessions as a belt-and-suspenders safeguard for
all CLI commands that route format_output through Rich.
Add python -m compileall -q src/ step to the integration_tests nox
session before launching pabot. On CI runners with high core counts
(e.g. 32 processes) the first Robot test to spawn a Python subprocess
could fail because 30+ processes simultaneously cold-compile the
entire source tree from scratch. Pre-compiling eliminates the
thundering-herd race and lets all workers read cached .pyc files.
- Restore pabot-based integration_tests with conservative parallelism
(<=2 processes by default) and support PABOT_PROCESSES/--processes
- Remove the temporary CI debug dump from noxfile.py
- Add robot/discovery_common.resource to silence non-fatal warning
- Document the change in implementation_plan.md
Three fixes targeting CI integration_tests failures (18 failures on e8aa5ac):
1. Merge duplicate *** Settings *** blocks in 14 robot files into single
blocks. Multiple Settings sections are non-standard RF practice and
may cause resource import failures in certain Robot Framework versions
or CI environments.
2. Replace bare 'python' with ${PYTHON} variable in all Run Process
calls (14 files). Noxfile now passes --variable PYTHON:<venv-path>
to robot so tests use the venv interpreter regardless of PATH. This
fixes '/usr/local/bin/python: No module named cleveragents' on CI.
3. Add comprehensive CI debug output in noxfile.py: file existence
checks for .resource files, PATH/Python resolution, fixture dir
checks, and RF version. This will diagnose any remaining resource
import issues.
Also: remove hardcoded '/app/src' sys.path.insert in
system_prompt_template_rendering.robot (not portable to CI), and
add trailing newline to common.resource.
All 204 tests pass locally (4 excluded: 2 slow, 2 discovery).
- Restore session.env["PATH"] in integration_tests nox session to ensure
Run Process uses venv Python instead of system Python
- Convert bare Resource references to ${CURDIR}/ absolute paths across
30 robot files to fix CI resolution failures
- Add timeout=30s to all Run Process calls in rxpy_route_validation.robot
to prevent hanging tests
- Tag 2 rxpy tests as slow (require running actors unavailable on CI)
- Fix LangGraph test to use correct config file (LANGGRAPH_CONFIG)
- All 204 tests pass (4 excluded: 2 slow + 2 discovery)
- integration_tests: replace pabot with sequential robot execution to
eliminate FileNotFoundError caused by subprocess/FD exhaustion in
constrained CI containers (pabot spawns a robot subprocess per suite;
after ~24 suites the container cannot execve new processes)
- integration_tests: add resource debug output (open FD count, ulimit
values) to help diagnose future CI container issues
- integration_tests: remove _pabot_parallel_args (no longer needed);
slow_integration_tests session still available for parallel runs
- load_context_test: add env:TERM=dumb alongside NO_COLOR=1 to disable
all Rich terminal styling (NO_COLOR only disables color, not bold/
reset ANSI codes that may split substrings)
- load_context_test: add repr() debug logging around the --load-context
match to reveal any invisible characters on CI
- security_scan: create build/ directory before bandit writes its JSON
report (fails on fresh CI checkout where directory does not exist)
- integration_tests: cap pabot parallelism to 2 processes and explicitly
propagate venv bin/ to PATH, preventing FileNotFoundError for the
robot binary under CI resource constraints
- integration_tests: add env:NO_COLOR=1 to load_context_test.robot help
text assertions so Rich ANSI escape codes do not break substring
matching on CI
- integration_tests: tag initial_next_command_test as slow (requires
OPENAI_API_KEY for LLM agent invocation, unavailable on CI)
- Fix Rich Console line-wrapping breaking assertions in
context_unit_tests_steps.py: collapse newlines before checking for
filenames and overflow summaries (CI temp paths exceed 80 columns)
- Fix features.mocks import failure in database_integration.robot:
replace hardcoded sys.path '/app' with portable ${CURDIR}/..
- Fix --load-context help text assertions in load_context_test.robot:
merge stderr into stdout via stderr=STDOUT and remove duplicate test
- Add standalone dead_code nox session running vulture directly
- Rename security nox session to security_scan to match CI references
- Restore --exclude discovery to integration_tests nox session (lost
during merge conflict resolution)
- Rewrite .forgejo/workflows/ci.yml to route all jobs through nox sessions
- Fix coverage_report nox session: serial behave mode replaces broken parallel
mode (22% -> 97% accuracy), raise fail-under from 85% to 97%
- Pass posargs through format nox session for CI --check support
- Add 11 CI workflow validation scenarios (Behave) + Robot smoke test + ASV bench
- Add 108 new Behave scenarios covering 6 largest coverage gaps to reach 97%:
yaml_template_engine, actor/config, actor/registry, message_router,
context_analysis, context_service
- Update docs/development/ci-cd.md with nox-based CI docs and 97% threshold
- Restore implementation_plan.md verbose style, check off completed CI tasks
Verified: 1673 scenarios pass, 97% coverage, lint clean, typecheck clean
brent.edwards
Brent E. Edwards
freemo
mngrif
CoreRasurae
Jeffrey Phillips Freeman