forked from cleveragents/cleveragents-core
freemo
d4a1a0d87b
Added dedicated E2E test infrastructure completely separate from the existing integration test suite. E2E tests use zero mocking — they exercise the real CleverAgents CLI with real LLM API keys. Key changes: - New e2e_tests nox session running Robot Framework with --include E2E tag filter against robot/e2e/ directory. Uses sequential robot (not pabot) since E2E tests hit real API endpoints with rate limits. Propagates ANTHROPIC_API_KEY, OPENAI_API_KEY, and GOOGLE_API_KEY from environment. Output goes to build/reports/robot-e2e/ to avoid artifact collisions. - Existing integration_tests session now excludes E2E-tagged tests via --exclude E2E on the pabot invocation. - New robot/e2e/common_e2e.resource provides shared E2E keywords: suite setup/teardown with per-suite isolation (no mock AI), graceful skip when LLM API keys are absent, CLI runner keyword, flexible output assertions, and temporary git repo fixture creation. - Minimal smoke test (robot/e2e/smoke_test.robot) validates the harness by running agents --version and agents --help. Does not require LLM keys. - Dedicated e2e_tests CI job in .forgejo/workflows/ci.yml injects LLM API keys from Forgejo secrets. Runs independently (no needs dependencies) and does not block regular CI. - e2e_tests is deliberately NOT in the default nox sessions list since it requires real API keys not present in all environments. ISSUES CLOSED: #740
26 lines
1.0 KiB
Plaintext
26 lines
1.0 KiB
Plaintext
*** Settings ***
|
|
Documentation Minimal E2E smoke test to validate the E2E test infrastructure.
|
|
...
|
|
... This test exercises ``agents --version`` to verify that the
|
|
... E2E harness (nox session, Robot Framework tag filtering,
|
|
... and common resource keywords) works correctly. It does
|
|
... NOT require LLM API keys.
|
|
Resource common_e2e.resource
|
|
Suite Setup E2E Suite Setup
|
|
Suite Teardown E2E Suite Teardown
|
|
|
|
*** Test Cases ***
|
|
CleverAgents Version Smoke Test
|
|
[Documentation] Verify the CleverAgents CLI responds to --version.
|
|
[Tags] E2E
|
|
${result}= Run CleverAgents Command --version
|
|
Should Not Be Empty ${result.stdout}
|
|
Output Should Contain ${result} cleveragents
|
|
|
|
CleverAgents Help Smoke Test
|
|
[Documentation] Verify the CleverAgents CLI responds to --help.
|
|
[Tags] E2E
|
|
${result}= Run CleverAgents Command --help
|
|
Should Not Be Empty ${result.stdout}
|
|
Output Should Contain ${result} usage
|