Files
cleveragents-core/robot/common.resource
Brent E. Edwards d88dad94f6
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 25s
CI / quality (pull_request) Successful in 25s
CI / build (pull_request) Successful in 26s
CI / typecheck (pull_request) Successful in 1m9s
CI / security (pull_request) Successful in 1m18s
CI / integration_tests (pull_request) Successful in 3m43s
CI / unit_tests (pull_request) Successful in 4m8s
CI / docker (pull_request) Successful in 43s
CI / coverage (pull_request) Successful in 5m35s
CI / benchmark-regression (pull_request) Successful in 32m44s
fix(test): resolve race condition in M4 validation integration test
Three-pronged fix for intermittent pabot-parallel race condition in
M4 validation integration tests:

1. Composable Setup Database Isolation keyword in common.resource
   gives each suite a unique CLEVERAGENTS_DATABASE_URL so concurrent
   pabot workers never contend on the same SQLite file.

2. Per-suite CLEVERAGENTS_HOME directories prevent shared temp
   directory cleanup from racing between workers.

3. Centralised reset_global_state() in robot/helpers_common.py clears
   Settings singleton, DI container, provider registry, and engine
   cache between chained CLI invocations in helper processes.

Also:
- Setup Test Environment now accepts optional mock_ai and
  auto_apply_migrations arguments (default TRUE) for backward
  compatibility while allowing suites to opt out.
- Added Suite Teardown to cli_plan_context_commands.robot.
- Fixed _COMMANDS typing in two helpers to eliminate type: ignore.
- Updated docs/development/testing.md to reflect helpers_common
  delegation pattern.
- Added timeout=30s to all Run Process calls in
  m4_e2e_verification.robot.

Fixes: #563
2026-03-10 19:04:57 +00:00

115 lines
6.0 KiB
Plaintext

*** Settings ***
Documentation Common resources and keywords for Robot Framework tests.
...
... Provides per-suite and per-test isolation for parallel execution
... via pabot. Each suite receives a unique CLEVERAGENTS_HOME and
... CLEVERAGENTS_DATABASE_URL so that concurrent workers never
... contend on the same SQLite file or on shared global state.
Library OperatingSystem
Library String
Library Collections
Library Process
*** Variables ***
${WORKSPACE} ${CURDIR}/..
${SRC_DIR} ${WORKSPACE}/src/cleveragents
# Use the Python interpreter that's running Robot (from nox venv)
# This ensures we use the same Python with all dependencies installed
# ${PYTHON} will be set dynamically in Setup Test Environment
*** Keywords ***
Setup Test Environment
[Documentation] Setup common test environment with per-suite isolation.
...
... Creates a unique CLEVERAGENTS_HOME directory so that
... pabot workers never collide on configuration files.
... Does NOT set CLEVERAGENTS_DATABASE_URL — suites that
... run helper scripts via ``Run Process`` should also call
... ``Setup Database Isolation`` (or ``Setup Test Environment
... With Database Isolation``) so that concurrent workers
... never contend on the same SQLite file.
...
... Optional arguments:
... - ``auto_apply_migrations``: Set CLEVERAGENTS_AUTO_APPLY_MIGRATIONS
... (default: ``${TRUE}``).
... - ``mock_ai``: Set CLEVERAGENTS_TESTING_USE_MOCK_AI
... (default: ``${TRUE}``). Pass ``${FALSE}`` in suites
... that intentionally need real AI responses.
[Arguments] ${auto_apply_migrations}=${TRUE} ${mock_ai}=${TRUE}
Log Setting up test environment
${safe_suite}= Replace String ${SUITE NAME} ${SPACE} _
${home}= Set Variable ${TEMPDIR}${/}.cleveragents_${safe_suite}
Run Keyword And Ignore Error Remove Directory ${home} recursive=True
Create Directory ${home}
Set Environment Variable CLEVERAGENTS_HOME ${home}
Set Suite Variable ${SUITE_HOME} ${home}
# Prevent migration prompts in helper sub-processes (opt-in via argument)
Run Keyword If ${auto_apply_migrations}
... Set Environment Variable CLEVERAGENTS_AUTO_APPLY_MIGRATIONS true
# Enable mock AI in helper sub-processes (opt-in via argument)
Run Keyword If ${mock_ai}
... Set Environment Variable CLEVERAGENTS_TESTING_USE_MOCK_AI true
# Get the actual Python executable being used
${python_exec}= Evaluate sys.executable sys
Set Suite Variable ${PYTHON} ${python_exec}
# Don't set AGENTS_EXECUTABLE as a single variable - we'll use ${PYTHON} -m cleveragents directly
Setup Database Isolation
[Documentation] Set per-suite unique database URLs for helper-based suites.
...
... Call this keyword AFTER ``Setup Test Environment`` in
... suites that invoke Python helper scripts via ``Run
... Process``. Without this, helper processes fall back to
... the default ``sqlite:///cleveragents.db`` (relative to
... CWD), which causes concurrent pabot workers to contend
... on the same file.
...
... Do NOT call this in suites that run the real
... ``cleveragents`` CLI as a subprocess (e.g.
... ``cli_plan_context_commands.robot``), because those
... commands determine their own database path at runtime.
${home}= Get Environment Variable CLEVERAGENTS_HOME
${db_path}= Set Variable ${home}${/}cleveragents_suite.db
Set Environment Variable CLEVERAGENTS_DATABASE_URL sqlite:///${db_path}
Set Suite Variable ${SUITE_DB_PATH} ${db_path}
${test_db_path}= Set Variable ${home}${/}cleveragents_suite_test.db
Set Environment Variable CLEVERAGENTS_TEST_DATABASE_URL sqlite:///${test_db_path}
Set Suite Variable ${SUITE_TEST_DB_PATH} ${test_db_path}
Setup Test Environment With Database Isolation
[Documentation] Convenience keyword: calls both ``Setup Test Environment``
... and ``Setup Database Isolation``. Use this in suites
... that execute Python helper scripts via ``Run Process``.
Setup Test Environment
Setup Database Isolation
Cleanup Test Environment
[Documentation] Clean up after tests.
...
... Removes the per-suite CLEVERAGENTS_HOME directory and
... any associated SQLite files. Uses the ``${SUITE_HOME}``
... suite variable set during setup rather than re-reading
... the environment variable, which could be stale if a test
... modified ``CLEVERAGENTS_HOME``.
Log Cleaning up test environment
Run Keyword And Ignore Error Remove Directory ${SUITE_HOME} recursive=True
# Clean env vars to avoid leaking into subsequent suites
Remove Environment Variable CLEVERAGENTS_HOME
Remove Environment Variable CLEVERAGENTS_AUTO_APPLY_MIGRATIONS
Remove Environment Variable CLEVERAGENTS_TESTING_USE_MOCK_AI
Remove Environment Variable CLEVERAGENTS_DATABASE_URL
Remove Environment Variable CLEVERAGENTS_TEST_DATABASE_URL
File Should Contain Pattern
[Arguments] ${file_path} ${pattern}
[Documentation] Check if file contains a pattern
${content}= Get File ${file_path}
Should Contain ${content} ${pattern}
Count Files With Extension
[Arguments] ${directory} ${extension}
[Documentation] Count files with specific extension
@{files}= List Files In Directory ${directory} *.${extension} recursive=True
${count}= Get Length ${files}
RETURN ${count}