d88dad94f6
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 25s
CI / quality (pull_request) Successful in 25s
CI / build (pull_request) Successful in 26s
CI / typecheck (pull_request) Successful in 1m9s
CI / security (pull_request) Successful in 1m18s
CI / integration_tests (pull_request) Successful in 3m43s
CI / unit_tests (pull_request) Successful in 4m8s
CI / docker (pull_request) Successful in 43s
CI / coverage (pull_request) Successful in 5m35s
CI / benchmark-regression (pull_request) Successful in 32m44s
Three-pronged fix for intermittent pabot-parallel race condition in M4 validation integration tests: 1. Composable Setup Database Isolation keyword in common.resource gives each suite a unique CLEVERAGENTS_DATABASE_URL so concurrent pabot workers never contend on the same SQLite file. 2. Per-suite CLEVERAGENTS_HOME directories prevent shared temp directory cleanup from racing between workers. 3. Centralised reset_global_state() in robot/helpers_common.py clears Settings singleton, DI container, provider registry, and engine cache between chained CLI invocations in helper processes. Also: - Setup Test Environment now accepts optional mock_ai and auto_apply_migrations arguments (default TRUE) for backward compatibility while allowing suites to opt out. - Added Suite Teardown to cli_plan_context_commands.robot. - Fixed _COMMANDS typing in two helpers to eliminate type: ignore. - Updated docs/development/testing.md to reflect helpers_common delegation pattern. - Added timeout=30s to all Run Process calls in m4_e2e_verification.robot. Fixes: #563
115 lines
6.0 KiB
Plaintext
115 lines
6.0 KiB
Plaintext
*** Settings ***
|
|
Documentation Common resources and keywords for Robot Framework tests.
|
|
...
|
|
... Provides per-suite and per-test isolation for parallel execution
|
|
... via pabot. Each suite receives a unique CLEVERAGENTS_HOME and
|
|
... CLEVERAGENTS_DATABASE_URL so that concurrent workers never
|
|
... contend on the same SQLite file or on shared global state.
|
|
Library OperatingSystem
|
|
Library String
|
|
Library Collections
|
|
Library Process
|
|
|
|
*** Variables ***
|
|
${WORKSPACE} ${CURDIR}/..
|
|
${SRC_DIR} ${WORKSPACE}/src/cleveragents
|
|
# Use the Python interpreter that's running Robot (from nox venv)
|
|
# This ensures we use the same Python with all dependencies installed
|
|
# ${PYTHON} will be set dynamically in Setup Test Environment
|
|
|
|
*** Keywords ***
|
|
Setup Test Environment
|
|
[Documentation] Setup common test environment with per-suite isolation.
|
|
...
|
|
... Creates a unique CLEVERAGENTS_HOME directory so that
|
|
... pabot workers never collide on configuration files.
|
|
... Does NOT set CLEVERAGENTS_DATABASE_URL — suites that
|
|
... run helper scripts via ``Run Process`` should also call
|
|
... ``Setup Database Isolation`` (or ``Setup Test Environment
|
|
... With Database Isolation``) so that concurrent workers
|
|
... never contend on the same SQLite file.
|
|
...
|
|
... Optional arguments:
|
|
... - ``auto_apply_migrations``: Set CLEVERAGENTS_AUTO_APPLY_MIGRATIONS
|
|
... (default: ``${TRUE}``).
|
|
... - ``mock_ai``: Set CLEVERAGENTS_TESTING_USE_MOCK_AI
|
|
... (default: ``${TRUE}``). Pass ``${FALSE}`` in suites
|
|
... that intentionally need real AI responses.
|
|
[Arguments] ${auto_apply_migrations}=${TRUE} ${mock_ai}=${TRUE}
|
|
Log Setting up test environment
|
|
${safe_suite}= Replace String ${SUITE NAME} ${SPACE} _
|
|
${home}= Set Variable ${TEMPDIR}${/}.cleveragents_${safe_suite}
|
|
Run Keyword And Ignore Error Remove Directory ${home} recursive=True
|
|
Create Directory ${home}
|
|
Set Environment Variable CLEVERAGENTS_HOME ${home}
|
|
Set Suite Variable ${SUITE_HOME} ${home}
|
|
# Prevent migration prompts in helper sub-processes (opt-in via argument)
|
|
Run Keyword If ${auto_apply_migrations}
|
|
... Set Environment Variable CLEVERAGENTS_AUTO_APPLY_MIGRATIONS true
|
|
# Enable mock AI in helper sub-processes (opt-in via argument)
|
|
Run Keyword If ${mock_ai}
|
|
... Set Environment Variable CLEVERAGENTS_TESTING_USE_MOCK_AI true
|
|
# Get the actual Python executable being used
|
|
${python_exec}= Evaluate sys.executable sys
|
|
Set Suite Variable ${PYTHON} ${python_exec}
|
|
# Don't set AGENTS_EXECUTABLE as a single variable - we'll use ${PYTHON} -m cleveragents directly
|
|
|
|
Setup Database Isolation
|
|
[Documentation] Set per-suite unique database URLs for helper-based suites.
|
|
...
|
|
... Call this keyword AFTER ``Setup Test Environment`` in
|
|
... suites that invoke Python helper scripts via ``Run
|
|
... Process``. Without this, helper processes fall back to
|
|
... the default ``sqlite:///cleveragents.db`` (relative to
|
|
... CWD), which causes concurrent pabot workers to contend
|
|
... on the same file.
|
|
...
|
|
... Do NOT call this in suites that run the real
|
|
... ``cleveragents`` CLI as a subprocess (e.g.
|
|
... ``cli_plan_context_commands.robot``), because those
|
|
... commands determine their own database path at runtime.
|
|
${home}= Get Environment Variable CLEVERAGENTS_HOME
|
|
${db_path}= Set Variable ${home}${/}cleveragents_suite.db
|
|
Set Environment Variable CLEVERAGENTS_DATABASE_URL sqlite:///${db_path}
|
|
Set Suite Variable ${SUITE_DB_PATH} ${db_path}
|
|
${test_db_path}= Set Variable ${home}${/}cleveragents_suite_test.db
|
|
Set Environment Variable CLEVERAGENTS_TEST_DATABASE_URL sqlite:///${test_db_path}
|
|
Set Suite Variable ${SUITE_TEST_DB_PATH} ${test_db_path}
|
|
|
|
Setup Test Environment With Database Isolation
|
|
[Documentation] Convenience keyword: calls both ``Setup Test Environment``
|
|
... and ``Setup Database Isolation``. Use this in suites
|
|
... that execute Python helper scripts via ``Run Process``.
|
|
Setup Test Environment
|
|
Setup Database Isolation
|
|
|
|
Cleanup Test Environment
|
|
[Documentation] Clean up after tests.
|
|
...
|
|
... Removes the per-suite CLEVERAGENTS_HOME directory and
|
|
... any associated SQLite files. Uses the ``${SUITE_HOME}``
|
|
... suite variable set during setup rather than re-reading
|
|
... the environment variable, which could be stale if a test
|
|
... modified ``CLEVERAGENTS_HOME``.
|
|
Log Cleaning up test environment
|
|
Run Keyword And Ignore Error Remove Directory ${SUITE_HOME} recursive=True
|
|
# Clean env vars to avoid leaking into subsequent suites
|
|
Remove Environment Variable CLEVERAGENTS_HOME
|
|
Remove Environment Variable CLEVERAGENTS_AUTO_APPLY_MIGRATIONS
|
|
Remove Environment Variable CLEVERAGENTS_TESTING_USE_MOCK_AI
|
|
Remove Environment Variable CLEVERAGENTS_DATABASE_URL
|
|
Remove Environment Variable CLEVERAGENTS_TEST_DATABASE_URL
|
|
|
|
File Should Contain Pattern
|
|
[Arguments] ${file_path} ${pattern}
|
|
[Documentation] Check if file contains a pattern
|
|
${content}= Get File ${file_path}
|
|
Should Contain ${content} ${pattern}
|
|
|
|
Count Files With Extension
|
|
[Arguments] ${directory} ${extension}
|
|
[Documentation] Count files with specific extension
|
|
@{files}= List Files In Directory ${directory} *.${extension} recursive=True
|
|
${count}= Get Length ${files}
|
|
RETURN ${count}
|