diff --git a/CHANGELOG.md b/CHANGELOG.md index 66630d6f..707a2dec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -493,6 +493,12 @@ Framework variables. (`robot/e2e/wf07_cicd.robot`, `robot/e2e/common_e2e.resource`, `robot/common_vars.py`) (#753) +- Added E2E Robot Framework test for Specification Workflow Example 14: Server + Mode — Team Collaboration. Exercises server mode configuration (server URL, + token, namespace), diagnostics, action publishing to team namespace with + namespace-scoped listing, actor registration, plan list smoke test, and + `supervised` automation profile verification with threshold field assertions + via real CLI with zero mocking. (`robot/e2e/wf14_server_mode.robot`) (#760) - Fixed `agents session list`, `agents session create`, and other session subcommands raising `AttributeError: 'DynamicContainer' object has no attribute 'db'` after `agents init`. Root cause: `_get_session_service()` diff --git a/robot/e2e/wf14_server_mode.robot b/robot/e2e/wf14_server_mode.robot new file mode 100644 index 00000000..a1aef8ac --- /dev/null +++ b/robot/e2e/wf14_server_mode.robot @@ -0,0 +1,135 @@ +*** Settings *** +Documentation E2E test for Workflow Example 14: server mode team collaboration. +... +... Validates the ``supervised`` automation profile in a distributed +... team scenario where multiple engineers share actions, actors, and +... projects via CleverAgents server mode. All commands exercise the +... real CLI with zero mocking. +... +... Coverage: Steps 1–2 (server config, resource publishing). +... Steps 3–4 (multi-machine usage, team monitoring) require a +... live server and are deferred. +Resource common_e2e.resource +Suite Setup WF14 Suite Setup +Suite Teardown WF14 Suite Teardown +Force Tags E2E + +*** Keywords *** +WF14 Suite Setup + [Documentation] E2E Suite Setup plus database initialisation for WF14 tests. + E2E Suite Setup + # Initialise the database so action/actor/plan commands work in all tests. + ${init}= Run CleverAgents Command init --yes + Should Be Equal As Integers ${init.rc} 0 + # Generate a unique suffix for entity names to avoid UNIQUE + # constraint collisions on repeated E2E runs or parallel CI. + ${suffix}= Evaluate __import__('uuid').uuid4().hex[:12] + Set Suite Variable ${RUN_SUFFIX} ${suffix} + +WF14 Suite Teardown + [Documentation] Restore config state and clean up E2E test environment. + # Reset config keys set during the server config test to avoid + # contaminating subsequent test suites. + Run Keyword And Ignore Error Run CleverAgents Command config set server.url ${EMPTY} expected_rc=None + Run Keyword And Ignore Error Run CleverAgents Command config set server.token ${EMPTY} expected_rc=None + Run Keyword And Ignore Error Run CleverAgents Command config set core.namespace local expected_rc=None + E2E Suite Teardown + +*** Test Cases *** +WF14 E2E Server Config Setup + [Documentation] Configure server URL, authentication token, and team namespace + ... for server mode per Specification Example 14 Step 1. + # Set and verify server URL + ${set_url}= Run CleverAgents Command config set server.url https://agents.example.com + Output Should Contain ${set_url} server.url + ${get_url}= Run CleverAgents Command config get server.url + Output Should Contain ${get_url} https://agents.example.com + # Set and verify server token (spec Step 1 includes token configuration) + ${set_token}= Run CleverAgents Command config set server.token tok_e2e_test_placeholder + Output Should Contain ${set_token} server.token + ${get_token}= Run CleverAgents Command config get server.token + Output Should Contain ${get_token} tok_e2e_test_placeholder + # Set and verify team namespace + ${set_ns}= Run CleverAgents Command config set core.namespace myteam + Output Should Contain ${set_ns} core.namespace + ${get_ns}= Run CleverAgents Command config get core.namespace + Output Should Contain ${get_ns} myteam + +WF14 E2E Diagnostics + [Documentation] Run diagnostics and verify basic health-check output. + ... Diagnostics checks are all local (config, database, disk, + ... API keys, git, etc.) so the command should succeed even + ... without a live server. Server-mode diagnostics (Server + ... connectivity, Namespace membership) shown in the spec are + ... not yet implemented in the diagnostics command (spec gap). + ${result}= Run CleverAgents Command diagnostics --format plain expected_rc=0 + # Basic diagnostic categories should appear in the output + ${combined}= Set Variable ${result.stdout}\n${result.stderr} + Should Contain Any ${combined} config Config configuration + Should Contain Any ${combined} database Database db + Should Contain Any ${combined} disk Disk space storage + +WF14 E2E Action Create For Namespace + [Documentation] Create an action in the team namespace and verify it persists. + ... Uses ``myteam/`` prefix per Specification Example 14 Step 2. + ${action_name}= Set Variable myteam/gen-tests-${RUN_SUFFIX} + ${yaml_content}= Catenate SEPARATOR=\n + ... name: ${action_name} + ... description: Generate unit tests for source modules + ... strategy_actor: local/strategist + ... execution_actor: local/executor + ... definition_of_done: All source modules have corresponding unit tests. + ... reusable: true + ... read_only: true + ${yaml_file}= Set Variable ${SUITE_HOME}${/}generate-tests-action.yaml + Create File ${yaml_file} ${yaml_content} + ${create_result}= Run CleverAgents Command action create --config ${yaml_file} + Output Should Contain ${create_result} gen-tests + # Verify action persists across separate CLI invocations via action show + ${show_result}= Run CleverAgents Command action show ${action_name} + Output Should Contain ${show_result} gen-tests + # Make the action name available for the namespace listing test + Set Suite Variable ${WF14_ACTION_NAME} ${action_name} + +WF14 E2E Action List By Namespace + [Documentation] List shared actions filtered by team namespace. + ... Validates Specification Example 14 Step 2: + ... ``agents action list --namespace myteam``. + ${list_result}= Run CleverAgents Command action list --namespace myteam --format plain + Output Should Contain ${list_result} gen-tests + +WF14 E2E Actor Add For Namespace + [Documentation] Add an actor from YAML configuration and verify it appears. + ... Custom actors must use the ``local/`` namespace prefix per + ... the current ActorService validation. + ${actor_name}= Set Variable local/team-rev-${RUN_SUFFIX} + ${yaml_content}= Catenate SEPARATOR=\n + ... provider: openai + ... model: gpt-4 + ... description: Team code reviewer actor for collaborative reviews + ${yaml_file}= Set Variable ${SUITE_HOME}${/}team-reviewer-actor.yaml + Create File ${yaml_file} ${yaml_content} + ${add_result}= Run CleverAgents Command actor add ${actor_name} --config ${yaml_file} + Output Should Contain ${add_result} team-rev + # Use --format plain to avoid table truncation + ${list_result}= Run CleverAgents Command actor list --format plain + Output Should Contain ${list_result} team-rev + +WF14 E2E Plan List + [Documentation] Verify that the plan list command executes without crashing. + ... No plans exist after init, so the output may be an empty table + ... or an informational message. The ``--namespace`` flag is not + ... yet implemented for plan list (spec gap). + ${result}= Run CleverAgents Command plan list expected_rc=0 + # After init, plan list should succeed (rc=0) even with an empty plan table + Should Not Be Empty ${result.stdout}${result.stderr} + +WF14 E2E Supervised Profile Verification + [Documentation] Verify the built-in supervised automation profile exists and + ... contains the expected confidence threshold fields. + ${result}= Run CleverAgents Command automation-profile show supervised + Output Should Contain ${result} supervised + # Verify supervised-specific threshold fields are present in the output + Output Should Contain ${result} auto_strategize + Output Should Contain ${result} auto_execute + Output Should Contain ${result} auto_apply