UAT: `robot/common.resource` enables mock AI (`CLEVERAGENTS_TESTING_USE_MOCK_AI=true`) by default for all integration test suites #4074

New issue

Open

opened 2026-04-06 09:57:16 +00:00 by freemo · 0 comments

freemo commented

2026-04-06 09:57:16 +00:00

Owner

Metadata

Branch: fix/integration-tests-real-ai-default
Commit Message: fix(robot): disable mock AI by default in integration test common resource
Milestone: (none — backlog)
Parent Epic: (Integration Testing Epic)

Background and Context

CONTRIBUTING.md (lines 568–570) states:

Integration tests must exercise real services, real endpoints, and real dependencies — mocking of any kind is strictly prohibited in integration tests.

The CLEVERAGENTS_TESTING_USE_MOCK_AI environment variable is a testing-only mechanism that replaces the real LLM provider with a mock/fake implementation. Setting this to true in integration tests means the AI provider is mocked, violating the no-mock rule.

What Was Tested

Code analysis of robot/common.resource (the shared setup resource used by nearly all Robot Framework integration test suites) and robot/helper_e2e_common.py.

Expected Behavior (from spec/CONTRIBUTING.md)

Integration tests should use real dependencies. The CLEVERAGENTS_TESTING_USE_MOCK_AI flag should:

Default to false (or be unset) in integration tests
Only be explicitly enabled in specific test suites that are intentionally testing non-AI behavior (e.g., CLI argument parsing, database persistence, config management)
Never be the global default for all integration tests

Actual Behavior

robot/common.resource sets CLEVERAGENTS_TESTING_USE_MOCK_AI=true by default for ALL integration test suites:

# robot/common.resource (lines 38, 50-51)
[Arguments]    ${auto_apply_migrations}=${TRUE}    ${mock_ai}=${TRUE}

# Enable mock AI in helper sub-processes (opt-in via argument)
Run Keyword If    ${mock_ai}
...    Set Environment Variable    CLEVERAGENTS_TESTING_USE_MOCK_AI    true

The mock_ai parameter defaults to ${TRUE}, meaning every suite that calls Setup Test Environment without explicitly passing mock_ai=${FALSE} will have mock AI enabled.

robot/helper_e2e_common.py also hardcodes mock AI:

# robot/helper_e2e_common.py (lines 45-46)
def run_cli(*args, workspace, env_extra=None, timeout=120):
    env = os.environ.copy()
    env.setdefault("CLEVERAGENTS_AUTO_APPLY_MIGRATIONS", "true")
    env.setdefault("CLEVERAGENTS_TESTING_USE_MOCK_AI", "true")  # ← Always enabled!

This means even the "real CLI subprocess" tests in helper_m1_e2e_verification.py use mock AI.

Impact

With CLEVERAGENTS_TESTING_USE_MOCK_AI=true enabled globally:

The AI provider (LLM) is replaced with a mock/fake implementation in all integration tests
Integration tests cannot detect regressions in the real LLM provider integration
Integration tests cannot detect issues with prompt formatting, token limits, or provider-specific behavior
The integration test suite provides false confidence that the AI integration works correctly
Tests that should exercise real AI-driven plan generation, strategy creation, and execution are silently using fake responses

Affected Suites

All suites that call Setup Test Environment without mock_ai=${FALSE} are affected. This includes the vast majority of integration test suites in robot/. Only a small number of suites explicitly opt out:

# Example of correct opt-out (rare):
Suite Setup    Setup Test Environment    mock_ai=${FALSE}

Steps to Reproduce

grep -n "mock_ai\|TESTING_USE_MOCK_AI" robot/common.resource robot/helper_e2e_common.py

Output shows mock_ai=${TRUE} as the default and env.setdefault("CLEVERAGENTS_TESTING_USE_MOCK_AI", "true") in the CLI helper.

Subtasks

Change mock_ai default in robot/common.resource Setup Test Environment from ${TRUE} to ${FALSE}
Remove env.setdefault("CLEVERAGENTS_TESTING_USE_MOCK_AI", "true") from robot/helper_e2e_common.py
Audit all suites that rely on mock AI and add explicit mock_ai=${TRUE} where intentional (e.g., CLI-only tests that don't exercise AI)
Ensure suites that test AI-driven workflows (plan generation, strategy, execution) use real AI or are tagged as E2E (requiring API keys)
Document in robot/common.resource which suites require real AI keys and which can run without them
Run nox -s integration_tests to verify non-AI tests still pass
Run nox -s e2e_tests to verify AI-driven tests pass with real keys

Definition of Done

mock_ai defaults to ${FALSE} in Setup Test Environment
helper_e2e_common.py does not set CLEVERAGENTS_TESTING_USE_MOCK_AI by default
All suites that intentionally use mock AI explicitly declare mock_ai=${TRUE}
nox -s integration_tests passes for non-AI tests
Coverage >= 97%

Supporting Information

Code location: robot/common.resource lines 38, 50–51
Code location: robot/helper_e2e_common.py lines 45–46
Rule source: CONTRIBUTING.md lines 568–570
Note: Tests that require real LLM API keys should be tagged with [Tags] E2E and run under nox -s e2e_tests (which requires API keys). Tests that don't exercise AI at all (CLI parsing, database persistence, config) can legitimately use mock_ai=${TRUE} but should be explicit about it.

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-uat-tester

## Metadata - **Branch**: `fix/integration-tests-real-ai-default` - **Commit Message**: `fix(robot): disable mock AI by default in integration test common resource` - **Milestone**: (none — backlog) - **Parent Epic**: (Integration Testing Epic) ## Background and Context `CONTRIBUTING.md` (lines 568–570) states: > Integration tests must exercise real services, real endpoints, and real dependencies — mocking of any kind is strictly prohibited in integration tests. The `CLEVERAGENTS_TESTING_USE_MOCK_AI` environment variable is a testing-only mechanism that replaces the real LLM provider with a mock/fake implementation. Setting this to `true` in integration tests means the AI provider is mocked, violating the no-mock rule. ## What Was Tested Code analysis of `robot/common.resource` (the shared setup resource used by nearly all Robot Framework integration test suites) and `robot/helper_e2e_common.py`. ## Expected Behavior (from spec/CONTRIBUTING.md) Integration tests should use real dependencies. The `CLEVERAGENTS_TESTING_USE_MOCK_AI` flag should: - Default to `false` (or be unset) in integration tests - Only be explicitly enabled in specific test suites that are intentionally testing non-AI behavior (e.g., CLI argument parsing, database persistence, config management) - Never be the global default for all integration tests ## Actual Behavior **`robot/common.resource` sets `CLEVERAGENTS_TESTING_USE_MOCK_AI=true` by default for ALL integration test suites:** ```robotframework # robot/common.resource (lines 38, 50-51) [Arguments] ${auto_apply_migrations}=${TRUE} ${mock_ai}=${TRUE} # Enable mock AI in helper sub-processes (opt-in via argument) Run Keyword If ${mock_ai} ... Set Environment Variable CLEVERAGENTS_TESTING_USE_MOCK_AI true ``` The `mock_ai` parameter defaults to `${TRUE}`, meaning every suite that calls `Setup Test Environment` without explicitly passing `mock_ai=${FALSE}` will have mock AI enabled. **`robot/helper_e2e_common.py` also hardcodes mock AI:** ```python # robot/helper_e2e_common.py (lines 45-46) def run_cli(*args, workspace, env_extra=None, timeout=120): env = os.environ.copy() env.setdefault("CLEVERAGENTS_AUTO_APPLY_MIGRATIONS", "true") env.setdefault("CLEVERAGENTS_TESTING_USE_MOCK_AI", "true") # ← Always enabled! ``` This means even the "real CLI subprocess" tests in `helper_m1_e2e_verification.py` use mock AI. ## Impact With `CLEVERAGENTS_TESTING_USE_MOCK_AI=true` enabled globally: - The AI provider (LLM) is replaced with a mock/fake implementation in all integration tests - Integration tests cannot detect regressions in the real LLM provider integration - Integration tests cannot detect issues with prompt formatting, token limits, or provider-specific behavior - The integration test suite provides false confidence that the AI integration works correctly - Tests that should exercise real AI-driven plan generation, strategy creation, and execution are silently using fake responses ## Affected Suites All suites that call `Setup Test Environment` without `mock_ai=${FALSE}` are affected. This includes the vast majority of integration test suites in `robot/`. Only a small number of suites explicitly opt out: ```robotframework # Example of correct opt-out (rare): Suite Setup Setup Test Environment mock_ai=${FALSE} ``` ## Steps to Reproduce ```bash grep -n "mock_ai\|TESTING_USE_MOCK_AI" robot/common.resource robot/helper_e2e_common.py ``` Output shows `mock_ai=${TRUE}` as the default and `env.setdefault("CLEVERAGENTS_TESTING_USE_MOCK_AI", "true")` in the CLI helper. ## Subtasks - [ ] Change `mock_ai` default in `robot/common.resource` `Setup Test Environment` from `${TRUE}` to `${FALSE}` - [ ] Remove `env.setdefault("CLEVERAGENTS_TESTING_USE_MOCK_AI", "true")` from `robot/helper_e2e_common.py` - [ ] Audit all suites that rely on mock AI and add explicit `mock_ai=${TRUE}` where intentional (e.g., CLI-only tests that don't exercise AI) - [ ] Ensure suites that test AI-driven workflows (plan generation, strategy, execution) use real AI or are tagged as `E2E` (requiring API keys) - [ ] Document in `robot/common.resource` which suites require real AI keys and which can run without them - [ ] Run `nox -s integration_tests` to verify non-AI tests still pass - [ ] Run `nox -s e2e_tests` to verify AI-driven tests pass with real keys ## Definition of Done - [ ] `mock_ai` defaults to `${FALSE}` in `Setup Test Environment` - [ ] `helper_e2e_common.py` does not set `CLEVERAGENTS_TESTING_USE_MOCK_AI` by default - [ ] All suites that intentionally use mock AI explicitly declare `mock_ai=${TRUE}` - [ ] `nox -s integration_tests` passes for non-AI tests - [ ] Coverage >= 97% ## Supporting Information - **Code location**: `robot/common.resource` lines 38, 50–51 - **Code location**: `robot/helper_e2e_common.py` lines 45–46 - **Rule source**: `CONTRIBUTING.md` lines 568–570 - **Note**: Tests that require real LLM API keys should be tagged with `[Tags] E2E` and run under `nox -s e2e_tests` (which requires API keys). Tests that don't exercise AI at all (CLI parsing, database persistence, config) can legitimately use `mock_ai=${TRUE}` but should be explicit about it. --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-uat-tester

freemo referenced this issue

2026-04-06 09:57:43 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo referenced this issue

2026-04-06 10:15:25 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo referenced this issue

2026-04-06 10:35:53 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775