freemo
8ea00f5185
CI / unit_tests (push) Has been cancelled
CI / benchmark-publish (push) Has been cancelled
CI / lint (push) Has been cancelled
CI / typecheck (push) Has been cancelled
CI / security (push) Has been cancelled
CI / quality (push) Has been cancelled
CI / integration_tests (push) Has been cancelled
CI / e2e_tests (push) Has been cancelled
CI / coverage (push) Has been cancelled
CI / benchmark-regression (push) Has been cancelled
CI / build (push) Has been cancelled
CI / push-validation (push) Has been cancelled
CI / status-check (push) Has been cancelled
CI / docker (push) Has been cancelled
CI / helm (push) Has been cancelled
Co-authored-by: Jeffrey Phillips Freeman <the@jeffreyfreeman.me> Co-committed-by: Jeffrey Phillips Freeman <the@jeffreyfreeman.me>
45 lines
2.3 KiB
Plaintext
45 lines
2.3 KiB
Plaintext
*** Settings ***
|
|
Documentation TDD Issue #658 — E2E verification suites use mocks instead of real system
|
|
... Integration tests verifying that at least one CLI-facing E2E test
|
|
... function exercises the real CLI via subprocess invocation without
|
|
... mocking the service layer. Bug #658 documents that all 21 CLI-facing
|
|
... tests in the M1-M6 helpers use unittest.mock.patch + CliRunner.
|
|
Resource ${CURDIR}/common.resource
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
${HELPER} ${CURDIR}/helper_tdd_e2e_mock_only_coverage.py
|
|
|
|
*** Test Cases ***
|
|
TDD At Least One CLI Test Uses Subprocess
|
|
[Documentation] Verify that at least one CLI-facing E2E test invokes
|
|
... the real ``agents`` CLI via subprocess instead of
|
|
... Typer's in-process CliRunner.
|
|
[Tags] tdd_issue tdd_issue_658 tdd_issue_4178
|
|
${result}= Run Process ${PYTHON} ${HELPER} check-subprocess-usage cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} tdd-e2e-subprocess-check-ok
|
|
|
|
TDD At Least One CLI Test Skips Service Mocking
|
|
[Documentation] Verify that at least one CLI-facing E2E test exercises
|
|
... the real service layer without mocking service factories.
|
|
[Tags] tdd_issue tdd_issue_658 tdd_issue_4178
|
|
${result}= Run Process ${PYTHON} ${HELPER} check-unmocked-services cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} tdd-e2e-unmocked-services-ok
|
|
|
|
TDD No Suite Has 100 Percent Mocked CLI Tests
|
|
[Documentation] Verify that every E2E suite with CLI tests has at
|
|
... least one test that exercises real code paths.
|
|
[Tags] tdd_issue tdd_issue_658 tdd_issue_4178
|
|
${result}= Run Process ${PYTHON} ${HELPER} check-per-suite-coverage cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} tdd-e2e-per-suite-coverage-ok
|