Implements the three-tag TDD bug-capture system in Robot Framework via a
Listener v3 module, paralleling the Behave implementation. Tests tagged
tdd_expected_fail that fail have their result inverted to PASS (bug still
exists); tests that unexpectedly pass are inverted to FAIL with guidance.
Addresses all 15 findings from code review (PR !673, reviewer hamza.khyari):
P2 fixes:
- Added idempotency guard (_processed_tests set) to prevent double-inversion
when the listener is loaded twice in the same process.
- Rewrote normal-test-unaffected check to run alongside a tdd_expected_fail
fixture in a single Robot invocation, proving the listener is loaded and
selectively applies rather than being a tautological pass.
P3 fixes:
- Added output.xml existence guard with clear diagnostics in _run_fixture.
- Documented intentional use of data.tags (static definition) vs result.tags
(runtime-modifiable) in end_test docstring.
- Added SKIP status test fixture and integration test case.
- Added message content assertion in cmd_expected_fail_inverted.
- Tightened substring assertions to match specific error text.
- Added tdd_expected_fail-alone fixture (both companions missing).
- Added close() hook to clear _validation_errors and _processed_tests.
- Simplified _run_fixture return type to tuple[str, str].
- Changed listener path resolution from CWD-relative to __file__-relative
in noxfile.py (integration_tests, slow_integration_tests, e2e_tests).
P4 fixes:
- Added __all__ declaration to helper module.
- Changed module docstring from "mirroring" to "paralleling".
- Added comment documenting accepted XML parsing risk (self-generated XML).
Additional fixes:
- Increased M4 E2E plan-tree test timeout from 30s to 120s (pre-existing
timeout failure unrelated to this feature).
Quality gates (post-rebase onto latest master):
- nox -s lint: PASS
- nox -s typecheck: PASS (0 errors)
- nox -s unit_tests: PASS (10,700 scenarios)
- nox -s integration_tests: PASS (1,505 tests)
- nox -s coverage_report: PASS (97.9% >= 97% threshold)
- nox -s benchmark: PASS
- nox -s docs: PASS
- nox -s build: PASS
- nox -s security_scan: PASS
- nox -s dead_code: PASS
ISSUES CLOSED: #628
Added dedicated E2E test infrastructure completely separate from the existing
integration test suite. E2E tests use zero mocking — they exercise the real
CleverAgents CLI with real LLM API keys.
Key changes:
- New e2e_tests nox session running Robot Framework with --include E2E tag
filter against robot/e2e/ directory. Uses sequential robot (not pabot)
since E2E tests hit real API endpoints with rate limits. Propagates
ANTHROPIC_API_KEY, OPENAI_API_KEY, and GOOGLE_API_KEY from environment.
Output goes to build/reports/robot-e2e/ to avoid artifact collisions.
- Existing integration_tests session now excludes E2E-tagged tests via
--exclude E2E on the pabot invocation.
- New robot/e2e/common_e2e.resource provides shared E2E keywords: suite
setup/teardown with per-suite isolation (no mock AI), graceful skip when
LLM API keys are absent, CLI runner keyword, flexible output assertions,
and temporary git repo fixture creation.
- Minimal smoke test (robot/e2e/smoke_test.robot) validates the harness by
running agents --version and agents --help. Does not require LLM keys.
- Dedicated e2e_tests CI job in .forgejo/workflows/ci.yml injects LLM API
keys from Forgejo secrets. Runs independently (no needs dependencies)
and does not block regular CI.
- e2e_tests is deliberately NOT in the default nox sessions list since it
requires real API keys not present in all environments.
ISSUES CLOSED: #740
Implement TDD bug-capture tests for bug #570 where `agents session create`
fails because `_get_session_service()` calls `container.db()` which does
not exist on the DI Container class (AttributeError). Same root cause as
bug #554.
Behave BDD scenarios tagged @tdd_bug @tdd_bug_570 @tdd_expected_fail
exercise the real DI path (no mocks). Includes Robot Framework integration
smoke tests with self-inverting helper and ASV benchmark baseline.
ISSUES CLOSED: #631
Add TDD regression tests for bug #570 where `_get_session_service()`
calls `container.db()` but the DI `Container` class has no `db`
provider, raising `AttributeError`. Same root cause as bug #554.
Includes 4 Behave BDD scenarios tagged `@tdd_bug @tdd_bug_570
@tdd_expected_fail`, Robot Framework integration smoke tests with
`--format plain`, and ASV service-layer benchmarks. Tests exercise the
real DI path by resetting `_service = None` and using a file-based
SQLite database.
Implements the `@tdd_expected_fail` inversion infrastructure:
- Behave: `after_scenario` hook in `features/environment.py` inverts
pass/fail for scenarios tagged `@tdd_expected_fail`
- Robot: `robot/tdd_expected_fail_listener.py` listener (API v3)
performs the same inversion for Robot test cases
- `noxfile.py`: registers the listener via `--listener` in both the
`integration_tests` and `slow_integration_tests` sessions
Migrates 18 existing TDD scenarios across 5 feature files from the old
`@tdd @bugNNN` convention to the standardised `@tdd_bug @tdd_bug_NNN`
tags per CONTRIBUTING.md § TDD Bug Test Tags.
Refs: #570
Implement TDD bug-capture tests for bug #554 where `agents session list`
fails because `_get_session_service()` calls `container.db()` which does
not exist on the DI Container class (AttributeError).
Behave BDD scenarios tagged @tdd_bug @tdd_bug_554 @tdd_expected_fail
exercise the real DI path (no mocks) and assert correct behavior. The
@tdd_expected_fail handler in environment.py inverts failed→passed while
the bug is present, keeping CI green.
Also adds:
- @tdd_expected_fail infrastructure in features/environment.py
(tag validation + status inversion in after_scenario hook)
- behave-parallel exit logic fix to use summary-based failure
detection (compatible with TDD status inversion)
- Robot Framework integration smoke tests with self-inverting helper
- ASV benchmark baseline for session list command throughput
ISSUES CLOSED: #630
Replace the subprocess-per-feature execution model (342 Python interpreter
startups) with direct use of behave's Runner API for in-process
execution.
Sequential mode (--processes 1 or BEHAVE_PARALLEL_COVERAGE=1): All
features run in a single Runner.run() call. Steps and hooks load once.
Parallel mode (--processes N, N>1): Features split into N chunks,
dispatched via multiprocessing.Pool with fork. Heavy modules shared
copy-on-write.
Proper format defaulting (mirrors behave.__main__.run_behave() logic
for -q flag). Summary extracted from runner.features status attributes
instead of regex-parsing stdout.
Simplified coverage pipeline: single slipcover invocation wraps the
entire behave-parallel process. No per-worker UUID files, no --merge
step needed. Coverage data produced in one build/coverage.json file.
Removed: behave-parallel tarball download from PyPI, tarfile and
urllib.request imports, per-worker subprocess.run() calls,
__SLIPCOVER_OUT__ placeholder mechanism, _build_base_args(),
_parse_summary(), regex-based summary parsing.
Results: nox -s unit_tests 24m21s -> 2m05s (91%); nox -s coverage_report
75m20s -> 3m00s (96%). Coverage: 98% (above 97% threshold).
ISSUES CLOSED: #481
Created scripts/create_template_db.py that builds a pre-migrated SQLite
template database using Base.metadata.create_all() + alembic stamp
(~5ms for 34 tables, vs ~0.5-3s x 25 Alembic migrations per scenario).
Nox unit_tests and coverage_report sessions generate the template before
test execution and propagate CLEVERAGENTS_TEMPLATE_DB env var to all
workers.
features/environment.py before_all() installs a monkey-patch on
MigrationRunner.init_or_upgrade that copies the template for fresh
scenario temp DBs, falling through to real migrations for :memory:,
existing files, and migration-runner unit tests.
Quick wins: sleep(0.5) -> sleep(0.05) in cli_streaming wait step;
removed redundant Background re-declaration in cli_streaming.feature
scenario 7.
ISSUES CLOSED: #483
Replace coverage.py (sys.settrace-based) with slipcover (bytecode-based
instrumentation) for significantly faster coverage collection:
- Each behave-parallel worker runs under slipcover, producing per-feature
JSON coverage files with unique UUIDs to avoid write contention.
- After all workers finish, slipcover --merge combines per-worker data
into a single build/coverage.json report.
- XML report generated via slipcover --merge --xml for CI tooling.
- Terminal report with --fail-under=97 threshold enforcement.
- Robust JSON key-fallback logic handles both slipcover and coverage.py
output formats.
- CI workflow (ci.yml, nightly-quality.yml) updated with defensive key
lookup instead of hardcoded coverage.py format.
- Documentation updated to reflect slipcover as the coverage tool.
- CHANGELOG.md updated.
ISSUES CLOSED: #482
- Remove AutomationLevel imports from cli_robot_flow_bench.py and
persistence_robot_bench.py (enum was removed by master's automation
refactor); replace with AutomationProfileRef where needed.
- Use typer.echo() instead of console.print() for machine-readable
output (JSON/YAML/plain) in config.py and session.py to prevent
Rich from injecting ANSI escape codes that corrupt json.loads().
- Set NO_COLOR=1 in noxfile unit_tests, integration_tests, and
coverage_report sessions as a belt-and-suspenders safeguard for
all CLI commands that route format_output through Rich.
Add python -m compileall -q src/ step to the integration_tests nox
session before launching pabot. On CI runners with high core counts
(e.g. 32 processes) the first Robot test to spawn a Python subprocess
could fail because 30+ processes simultaneously cold-compile the
entire source tree from scratch. Pre-compiling eliminates the
thundering-herd race and lets all workers read cached .pyc files.
- Restore pabot-based integration_tests with conservative parallelism
(<=2 processes by default) and support PABOT_PROCESSES/--processes
- Remove the temporary CI debug dump from noxfile.py
- Add robot/discovery_common.resource to silence non-fatal warning
- Document the change in implementation_plan.md
Three fixes targeting CI integration_tests failures (18 failures on e8aa5ac):
1. Merge duplicate *** Settings *** blocks in 14 robot files into single
blocks. Multiple Settings sections are non-standard RF practice and
may cause resource import failures in certain Robot Framework versions
or CI environments.
2. Replace bare 'python' with ${PYTHON} variable in all Run Process
calls (14 files). Noxfile now passes --variable PYTHON:<venv-path>
to robot so tests use the venv interpreter regardless of PATH. This
fixes '/usr/local/bin/python: No module named cleveragents' on CI.
3. Add comprehensive CI debug output in noxfile.py: file existence
checks for .resource files, PATH/Python resolution, fixture dir
checks, and RF version. This will diagnose any remaining resource
import issues.
Also: remove hardcoded '/app/src' sys.path.insert in
system_prompt_template_rendering.robot (not portable to CI), and
add trailing newline to common.resource.
All 204 tests pass locally (4 excluded: 2 slow, 2 discovery).
- Restore session.env["PATH"] in integration_tests nox session to ensure
Run Process uses venv Python instead of system Python
- Convert bare Resource references to ${CURDIR}/ absolute paths across
30 robot files to fix CI resolution failures
- Add timeout=30s to all Run Process calls in rxpy_route_validation.robot
to prevent hanging tests
- Tag 2 rxpy tests as slow (require running actors unavailable on CI)
- Fix LangGraph test to use correct config file (LANGGRAPH_CONFIG)
- All 204 tests pass (4 excluded: 2 slow + 2 discovery)
- integration_tests: replace pabot with sequential robot execution to
eliminate FileNotFoundError caused by subprocess/FD exhaustion in
constrained CI containers (pabot spawns a robot subprocess per suite;
after ~24 suites the container cannot execve new processes)
- integration_tests: add resource debug output (open FD count, ulimit
values) to help diagnose future CI container issues
- integration_tests: remove _pabot_parallel_args (no longer needed);
slow_integration_tests session still available for parallel runs
- load_context_test: add env:TERM=dumb alongside NO_COLOR=1 to disable
all Rich terminal styling (NO_COLOR only disables color, not bold/
reset ANSI codes that may split substrings)
- load_context_test: add repr() debug logging around the --load-context
match to reveal any invisible characters on CI
- security_scan: create build/ directory before bandit writes its JSON
report (fails on fresh CI checkout where directory does not exist)
- integration_tests: cap pabot parallelism to 2 processes and explicitly
propagate venv bin/ to PATH, preventing FileNotFoundError for the
robot binary under CI resource constraints
- integration_tests: add env:NO_COLOR=1 to load_context_test.robot help
text assertions so Rich ANSI escape codes do not break substring
matching on CI
- integration_tests: tag initial_next_command_test as slow (requires
OPENAI_API_KEY for LLM agent invocation, unavailable on CI)
- Fix Rich Console line-wrapping breaking assertions in
context_unit_tests_steps.py: collapse newlines before checking for
filenames and overflow summaries (CI temp paths exceed 80 columns)
- Fix features.mocks import failure in database_integration.robot:
replace hardcoded sys.path '/app' with portable ${CURDIR}/..
- Fix --load-context help text assertions in load_context_test.robot:
merge stderr into stdout via stderr=STDOUT and remove duplicate test
- Add standalone dead_code nox session running vulture directly
- Rename security nox session to security_scan to match CI references
- Restore --exclude discovery to integration_tests nox session (lost
during merge conflict resolution)
hurui200320
freemo
brent.edwards
Brent E. Edwards
mngrif
CoreRasurae
Jeffrey Phillips Freeman