Files
HAL9000 7aa50ac489 fix(e2e): restore M5/WF14 tests broken by JSON envelope and stale config
- Add NO_COLOR=1 to E2E Suite Setup and Test Environment to prevent
  Rich from injecting ANSI escape codes into JSON output, which breaks
  Extract JSON From Stdout and JSON assertions in M5/WF14 acceptance tests.

- Add reset_global_state() calls to all M1/M2/M4/M5 E2E verification
  helpers to clear Settings singleton, DI container, provider registry,
  and SQLAlchemy engine cache between parallel pabot worker runs.

- Propagate NO_COLOR=1 at suite-setup level in E2E and integration
  test resource files so all Run Process subprocesses inherit clean
  terminal output mode.

Root causes:
1. Missing NO_COLOR=1 caused Rich ANSI codes in CLI JSON output,
   making json.loads() failures in robot Extract JSON From Stdout
2. Stale Settings singleton carried provider keys and DB paths between
   pabot workers, causing UNIQUE constraint violations and stale config
   lookups in context policy and server mode E2E tests.

Related: PR #9912
2026-05-01 00:59:02 +00:00

116 lines
6.1 KiB
Plaintext

*** Settings ***
Documentation Common resources and keywords for Robot Framework tests.
...
... Provides per-suite and per-test isolation for parallel execution
... via pabot. Each suite receives a unique CLEVERAGENTS_HOME and
... CLEVERAGENTS_DATABASE_URL so that concurrent workers never
... contend on the same SQLite file or on shared global state.
Library OperatingSystem
Library String
Library Collections
Library Process
*** Variables ***
${WORKSPACE} ${CURDIR}/..
${SRC_DIR} ${WORKSPACE}/src/cleveragents
# Use the Python interpreter that's running Robot (from nox venv)
# This ensures we use the same Python with all dependencies installed
# ${PYTHON} will be set dynamically in Setup Test Environment
*** Keywords ***
Setup Test Environment
[Documentation] Setup common test environment with per-suite isolation.
...
... Creates a unique CLEVERAGENTS_HOME directory so that
... pabot workers never collide on configuration files.
... Does NOT set CLEVERAGENTS_DATABASE_URL — suites that
... run helper scripts via ``Run Process`` should also call
... ``Setup Database Isolation`` (or ``Setup Test Environment
... With Database Isolation``) so that concurrent workers
... never contend on the same SQLite file.
...
... Optional arguments:
... - ``auto_apply_migrations``: Set CLEVERAGENTS_AUTO_APPLY_MIGRATIONS
... (default: ``${TRUE}``).
... - ``mock_ai``: Set CLEVERAGENTS_TESTING_USE_MOCK_AI
... (default: ``${TRUE}``). Pass ``${FALSE}`` in suites
... that intentionally need real AI responses.
[Arguments] ${auto_apply_migrations}=${TRUE} ${mock_ai}=${TRUE}
Log Setting up test environment
Set Environment Variable NO_COLOR 1
${safe_suite}= Replace String ${SUITE NAME} ${SPACE} _
${home}= Set Variable ${TEMPDIR}${/}.cleveragents_${safe_suite}
Run Keyword And Ignore Error Remove Directory ${home} recursive=True
Create Directory ${home}
Set Environment Variable CLEVERAGENTS_HOME ${home}
Set Suite Variable ${SUITE_HOME} ${home}
# Prevent migration prompts in helper sub-processes (opt-in via argument)
Run Keyword If ${auto_apply_migrations}
... Set Environment Variable CLEVERAGENTS_AUTO_APPLY_MIGRATIONS true
# Enable mock AI in helper sub-processes (opt-in via argument)
Run Keyword If ${mock_ai}
... Set Environment Variable CLEVERAGENTS_TESTING_USE_MOCK_AI true
# Get the actual Python executable being used
${python_exec}= Evaluate sys.executable sys
Set Suite Variable ${PYTHON} ${python_exec}
# Don't set AGENTS_EXECUTABLE as a single variable - we'll use ${PYTHON} -m cleveragents directly
Setup Database Isolation
[Documentation] Set per-suite unique database URLs for helper-based suites.
...
... Call this keyword AFTER ``Setup Test Environment`` in
... suites that invoke Python helper scripts via ``Run
... Process``. Without this, helper processes fall back to
... the default ``sqlite:///cleveragents.db`` (relative to
... CWD), which causes concurrent pabot workers to contend
... on the same file.
...
... Do NOT call this in suites that run the real
... ``cleveragents`` CLI as a subprocess (e.g.
... ``cli_plan_context_commands.robot``), because those
... commands determine their own database path at runtime.
${home}= Get Environment Variable CLEVERAGENTS_HOME
${db_path}= Set Variable ${home}${/}cleveragents_suite.db
Set Environment Variable CLEVERAGENTS_DATABASE_URL sqlite:///${db_path}
Set Suite Variable ${SUITE_DB_PATH} ${db_path}
${test_db_path}= Set Variable ${home}${/}cleveragents_suite_test.db
Set Environment Variable CLEVERAGENTS_TEST_DATABASE_URL sqlite:///${test_db_path}
Set Suite Variable ${SUITE_TEST_DB_PATH} ${test_db_path}
Setup Test Environment With Database Isolation
[Documentation] Convenience keyword: calls both ``Setup Test Environment``
... and ``Setup Database Isolation``. Use this in suites
... that execute Python helper scripts via ``Run Process``.
Setup Test Environment
Setup Database Isolation
Cleanup Test Environment
[Documentation] Clean up after tests.
...
... Removes the per-suite CLEVERAGENTS_HOME directory and
... any associated SQLite files. Uses the ``${SUITE_HOME}``
... suite variable set during setup rather than re-reading
... the environment variable, which could be stale if a test
... modified ``CLEVERAGENTS_HOME``.
Log Cleaning up test environment
Run Keyword And Ignore Error Remove Directory ${SUITE_HOME} recursive=True
# Clean env vars to avoid leaking into subsequent suites
Remove Environment Variable CLEVERAGENTS_HOME
Remove Environment Variable CLEVERAGENTS_AUTO_APPLY_MIGRATIONS
Remove Environment Variable CLEVERAGENTS_TESTING_USE_MOCK_AI
Remove Environment Variable CLEVERAGENTS_DATABASE_URL
Remove Environment Variable CLEVERAGENTS_TEST_DATABASE_URL
File Should Contain Pattern
[Arguments] ${file_path} ${pattern}
[Documentation] Check if file contains a pattern
${content}= Get File ${file_path}
Should Contain ${content} ${pattern}
Count Files With Extension
[Arguments] ${directory} ${extension}
[Documentation] Count files with specific extension
@{files}= List Files In Directory ${directory} *.${extension} recursive=True
${count}= Get Length ${files}
RETURN ${count}