cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity

fix: Correct TDD tag format in e2e tests

Browse Source 
- Fixed tag format from @tdd_issue:N to @tdd_issue_N (underscore instead of colon)
- Added missing @tdd_issue base tag to all tests with @tdd_expected_fail
- This ensures the TDD expected-fail listener properly recognizes and inverts the test results

The TDD system requires three tags:
1. @tdd_issue (base tag for filtering)
2. @tdd_issue_N (specific issue reference)
3. @tdd_expected_fail (to invert the result)

Previously e2e tests were using @tdd_issue:N format which the listener doesn't recognize.
This commit is contained in:
Jeffrey Phillips Freeman
2026-04-06 23:26:23 +00:00
committed by Forgejo
parent 7a09c8a043
commit 696a30581a
11 changed files with 45 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
robot/e2e/m2_acceptance.robot
+1 -1
View File
@@ -23,7 +23,7 @@ M2 Full Actor Compiler And LLM Integration
... resource and project setup, action creation referencing a
... custom actor, and the full plan lifecycle (use, execute
... strategize, execute, diff, apply) with real LLM API keys.
[Tags] E2E @tdd_issue:4189 @tdd_expected_fail
[Tags] E2E @tdd_issue @tdd_issue_4189 @tdd_expected_fail
Skip If No LLM Keys
# ---- Step 1: Create temp git repo with sample project files ----
${repo_dir}= Create Temp Git Repo m2-e2e-repo
robot/e2e/m5_acceptance.robot
+14 -14
View File
@@ -110,7 +110,7 @@ Plan Test Setup
# -----------------------------------------------------------------------
Context Assembly — Add Files To Context
[Documentation] ``context-load`` adds files; ``context list`` reflects them.
[Tags] E2E @tdd_issue:4189 @tdd_expected_fail
[Tags] E2E @tdd_issue @tdd_issue_4189 @tdd_expected_fail
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
... msg=Prerequisite not met: suite setup did not complete
${result}= Run CLI context-load main.py utils.py
@@ -121,7 +121,7 @@ Context Assembly — Add Files To Context
Context Assembly — Show File Content
[Documentation] ``context show <file>`` displays file content.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Tags] E2E
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
... msg=Prerequisite not met: suite setup did not complete
@@ -136,7 +136,7 @@ Context Assembly — Show Context Summary
... keywords, and that the CLI exit code is 0. Also
... checks the output is not just an error message by
... verifying the absence of common error indicators.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Tags] E2E
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
... msg=Prerequisite not met: suite setup did not complete
@@ -156,7 +156,7 @@ Context Assembly — Clear Context
[Documentation] ``context clear --yes`` removes all loaded files.
... Asserts files are present before clearing so the
... ``Should Not Contain`` checks are not vacuously true.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Tags] E2E
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
... msg=Prerequisite not met: suite setup did not complete
@@ -192,7 +192,7 @@ Context Scaling — Structural Plumbing for 10K File Projects
... of M5 criterion *"Projects with 10,000+ files index
... without timeout"* is deferred until the full ACMS
... indexing pipeline is wired.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Tags] E2E
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
... msg=Prerequisite not met: suite setup did not complete
@@ -245,7 +245,7 @@ Context Scaling — Structural Plumbing for 10K File Projects
# -----------------------------------------------------------------------
Context Policy — Create Project For Policy Tests
[Documentation] Pre-requisite: create a project and link the workspace resource.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Tags] E2E
${result}= Run CLI project create ${PROJECT_POLICY}
Should Be Equal As Integers ${result.rc} 0
@@ -255,7 +255,7 @@ Context Policy — Create Project For Policy Tests
Context Policy — Set Default View
[Documentation] Configure the default context view with include/exclude
... paths and file-size limits.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Tags] E2E
[Setup] Variable Should Exist ${POLICY_PROJECT_CREATED}
... msg=Prerequisite not met: policy project not created
@@ -271,7 +271,7 @@ Context Policy — Set Default View
Context Policy — Set Strategize View Override
[Documentation] The strategize view overrides the default with tighter
... limits and additional include paths.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Tags] E2E
[Setup] Variable Should Exist ${POLICY_PROJECT_CREATED}
... msg=Prerequisite not met: policy project not created
@@ -288,7 +288,7 @@ Context Policy — Verify Default View
... configured values. Uses parsed JSON assertions to
... avoid substring collisions (e.g. ``1024`` matching
... inside ``10240``).
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Tags] E2E
[Setup] Variable Should Exist ${POLICY_PROJECT_CREATED}
... msg=Prerequisite not met: policy project not created
@@ -311,7 +311,7 @@ Context Policy — Verify Default View
Context Policy — Verify Strategize View
[Documentation] ``project context show --view strategize`` returns the
... overridden values. Uses parsed JSON assertions.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Tags] E2E
[Setup] Variable Should Exist ${POLICY_PROJECT_CREATED}
... msg=Prerequisite not met: policy project not created
@@ -407,7 +407,7 @@ Context Analysis — Create Project With ACMS Config
[Documentation] Create a project and configure ACMS pipeline parameters.
... Uses **non-default** values to avoid vacuous assertions —
... defaults are 8000 / 500 / 5000.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Tags] E2E
${result}= Run CLI project create ${PROJECT_ANALYSIS}
Should Be Equal As Integers ${result.rc} 0
@@ -435,7 +435,7 @@ Context Analysis — Inspect Context Tiers (Structural)
... was indexed — the ``ContextTierService`` is empty per
... process. These assertions verify JSON schema correctness,
... not ACMS behavioral state.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Tags] E2E
[Setup] Variable Should Exist ${ANALYSIS_PROJECT_CREATED}
... msg=Prerequisite not met: analysis project not created
@@ -480,7 +480,7 @@ Context Analysis — Simulate Produces Structured Output
... process, so ``total_tokens`` is 0 and ``budget_used``
... is 0.0. These assertions verify JSON schema
... correctness, not ACMS behavioral output.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Tags] E2E
[Setup] Variable Should Exist ${ANALYSIS_PROJECT_CREATED}
... msg=Prerequisite not met: analysis project not created
@@ -508,7 +508,7 @@ Context Analysis — Show Full Policy With ACMS Config
[Documentation] ``project context show`` includes the ACMS pipeline config.
... Uses parsed JSON assertions to avoid substring collisions
... (e.g. 500 matching inside 3500).
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Tags] E2E
[Setup] Variable Should Exist ${ANALYSIS_PROJECT_CREATED}
... msg=Prerequisite not met: analysis project not created
robot/e2e/m6_acceptance.robot
+1 -1
View File
@@ -409,7 +409,7 @@ M6 E2E Hierarchical Decomposition Via Plan Tree
M6 E2E Full Autonomy Acceptance Flow
[Documentation] End-to-end: init, resource, project, action, plan use, execute, status, apply.
[Tags] @tdd_issue:4189 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4189 @tdd_expected_fail
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
Skip If No LLM Keys
${proj_name}= Setup Plan Test Resources flow
robot/e2e/wf04_multi_project.robot
+1 -1
View File
@@ -330,7 +330,7 @@ WF04 Multi Project Dependency Update Supervised Profile
... targeting all 4 projects with --automation-profile supervised,
... execute with child plan spawning in dependency order,
... verify per-project validation, and apply in dependency order.
[Tags] @tdd_issue:4189 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4189 @tdd_expected_fail
[Timeout] 25 minutes
[Teardown] WF04 Test Teardown
Skip If No LLM Keys
robot/e2e/wf05_db_migration.robot
+1 -1
View File
@@ -121,7 +121,7 @@ WF05 Database Schema Migration With Safety Nets Review Profile
... register git-checkout resource, create custom skill with
... spec-aligned DB tools (query_db, execute_migration,
... backfill_column), create migration action with review
[Tags] @tdd_issue:4189 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4189 @tdd_expected_fail
... automation profile, exercise phased child plan execution,
... attempt checkpoint-based rollback, verify migration changes.
[Timeout] 30 minutes
robot/e2e/wf07_cicd.robot
+6 -6
View File
@@ -27,7 +27,7 @@ ${CI_TESTS_VAL} local/ci-tests
WF07 E2E CI Profile Configuration
[Documentation] Set and verify the ci automation profile, json format,
... and log level per specification Step 1.
[Tags] E2E @tdd_issue:4189 @tdd_expected_fail
[Tags] E2E @tdd_issue @tdd_issue_4189 @tdd_expected_fail
[Teardown] Log CI Profile Configuration teardown complete
# Automation profile
Run CleverAgents Command config set core.automation-profile ci
@@ -50,7 +50,7 @@ WF07 E2E CI Profile Configuration
WF07 E2E Idempotent Resource Registration
[Documentation] Register a git-checkout resource twice with --path and
... --branch, and verify it appears in the resource list.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Tags] E2E
[Teardown] Log Idempotent Resource Registration teardown complete
${repo_dir}= Create Temp Git Repo With Issues ci-repo
@@ -70,7 +70,7 @@ WF07 E2E Idempotent Resource Registration
WF07 E2E Idempotent Project Registration
[Documentation] Create a project linked to the resource twice and verify
... it appears in the project list (idempotent).
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Tags] E2E
[Teardown] Log Idempotent Project Registration teardown complete
# First creation with resource link and description (per spec Step 3)
@@ -93,7 +93,7 @@ WF07 E2E Validation Registration
[Documentation] Register lint, typecheck, and test validations, attach
... them to the project, and verify they appear in the tool
... list and project show output.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Tags] E2E
[Teardown] Log Validation Registration teardown complete
# --- Lint validation ---
@@ -155,7 +155,7 @@ WF07 E2E CI Plan Launch
[Documentation] Create an action and launch a plan with --automation-profile ci,
... --format json, and --arg flags per specification Steps 2-3.
... Polls for plan completion. Requires LLM API keys.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Tags] E2E
[Teardown] Log CI Plan Launch teardown complete
Skip If No LLM Keys
@@ -266,7 +266,7 @@ WF07 E2E CI Plan Launch
WF07 E2E JSON Output Verification
[Documentation] Verify that --format json produces valid, parseable JSON
... output from the version command.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Tags] E2E
[Teardown] Log JSON Output Verification teardown complete
${result}= Run CleverAgents Command --format json version
robot/e2e/wf12_hierarchical.robot
+2 -2
View File
@@ -18,7 +18,7 @@ Force Tags E2E
*** Keywords ***
WF12 Suite Setup
[Documentation] E2E Suite Setup plus workspace init and unique run suffix for name isolation.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
E2E Suite Setup
# Initialise the database so plan/resource/project commands work in all tests.
# Use --force because the workspace may already contain an initialised project.
@@ -127,7 +127,7 @@ WF12 Large Scale Hierarchical Feature Implementation
... Once available, this test should add a ``plan prompt`` step
... after tree inspection to provide user guidance and verify
... the ``user_intervention`` decision is created.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Timeout] 35 minutes
# ---- Gate: skip if no LLM API keys ----
robot/e2e/wf14_server_mode.robot
+9 -9
View File
@@ -17,7 +17,7 @@ Force Tags E2E
*** Keywords ***
WF14 Suite Setup
[Documentation] E2E Suite Setup plus unique run suffix generation.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
E2E Suite Setup
# Generate a unique suffix for entity names to avoid UNIQUE
# constraint collisions on repeated E2E runs or parallel CI.
@@ -26,7 +26,7 @@ WF14 Suite Setup
WF14 Suite Teardown
[Documentation] Restore config state and clean up E2E test environment.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
# Reset config keys set during the server config test to avoid
# contaminating subsequent test suites.
Run Keyword And Ignore Error Run CleverAgents Command config set server.url ${EMPTY} expected_rc=None
@@ -38,7 +38,7 @@ WF14 Suite Teardown
WF14 E2E Server Config Setup
[Documentation] Configure server URL, authentication token, and team namespace
... for server mode per Specification Example 14 Step 1.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
# Set and verify server URL
${set_url}= Run CleverAgents Command config set server.url https://agents.example.com
Output Should Contain ${set_url} server.url
@@ -62,7 +62,7 @@ WF14 E2E Diagnostics
... without a live server. Server-mode diagnostics (Server
... connectivity, Namespace membership) shown in the spec are
... not yet implemented in the diagnostics command (spec gap).
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
${result}= Run CleverAgents Command --format plain diagnostics expected_rc=0
# Basic diagnostic categories should appear in the output
${combined}= Set Variable ${result.stdout}\n${result.stderr}
@@ -73,7 +73,7 @@ WF14 E2E Diagnostics
WF14 E2E Action Create For Namespace
[Documentation] Create an action in the team namespace and verify it persists.
... Uses ``myteam/`` prefix per Specification Example 14 Step 2.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
${action_name}= Set Variable myteam/gen-tests-${RUN_SUFFIX}
${yaml_content}= Catenate SEPARATOR=\n
... name: ${action_name}
@@ -97,7 +97,7 @@ WF14 E2E Action List By Namespace
[Documentation] List shared actions filtered by team namespace.
... Validates Specification Example 14 Step 2:
... ``agents action list --namespace myteam``.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
${list_result}= Run CleverAgents Command action list --namespace myteam --format plain
Output Should Contain ${list_result} gen-tests
@@ -105,7 +105,7 @@ WF14 E2E Actor Add For Namespace
[Documentation] Add an actor from YAML configuration and verify it appears.
... Custom actors must use the ``local/`` namespace prefix per
... the current ActorService validation.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
${actor_name}= Set Variable local/team-rev-${RUN_SUFFIX}
${yaml_content}= Catenate SEPARATOR=\n
... name: ${actor_name}
@@ -125,7 +125,7 @@ WF14 E2E Plan List
... No plans exist after init, so the output may be an empty table
... or an informational message. The ``--namespace`` flag is not
... yet implemented for plan list (spec gap).
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
${result}= Run CleverAgents Command plan list expected_rc=0
# After init, plan list should succeed (rc=0) even with an empty plan table
Should Not Be Empty ${result.stdout}${result.stderr}
@@ -133,7 +133,7 @@ WF14 E2E Plan List
WF14 E2E Supervised Profile Verification
[Documentation] Verify the built-in supervised automation profile exists and
... contains the expected confidence threshold fields.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
${result}= Run CleverAgents Command automation-profile show supervised
Output Should Contain ${result} supervised
# Verify supervised-specific threshold fields are present in the output
robot/e2e/wf16_devcontainer.robot
+3 -3
View File
@@ -33,7 +33,7 @@ ${PROJECT_PREFIX} local/wf16-devcontainer-project
WF16 Suite Setup
[Documentation] E2E Suite Setup plus database initialisation, unique suffix
... generation, and dynamic actor selection for WF16 tests.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
E2E Suite Setup
# Initialise the database so CLI commands work in all tests.
# Use --force because the workspace may already contain an initialised project.
@@ -107,7 +107,7 @@ Create Devcontainer Repo
WF16 Test Teardown
[Documentation] Log diagnostic context on failure for debugging.
... Captures plan status when a plan ID is available.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
${plan_id}= Get Variable Value ${WF16_PLAN_ID} ${EMPTY}
IF '${plan_id}' != ''
${status} ${result}= Run Keyword And Ignore Error
@@ -132,7 +132,7 @@ WF16 Devcontainer Driven Development Supervised Profile
... Tagged ``tdd_expected_fail`` because devcontainer features
... are not yet fully wired; the listener inverts the failure
... to a CI pass until all AC indicators are present.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Tags] tdd_expected_fail tdd_issue tdd_issue_1208
# Timeout budget: 35 minutes test-level timeout. Per-step timeouts sum
# higher in theory, but retries and conditional paths are mutually
robot/e2e/wf17_explicit_container.robot
+5 -5
View File
@@ -24,7 +24,7 @@ ${PROJECT_BASE} local/wf17-mount-project
WF17 Suite Setup
[Documentation] E2E Suite Setup plus workspace initialisation and
... dynamic actor selection for WF17 container tests.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
E2E Suite Setup
# Initialise the workspace so all CLI commands work.
${init}= Run CleverAgents Command init --force --yes
@@ -69,7 +69,7 @@ WF17 Explicit Container With Directory Mount Trusted Profile
... of tool invocations without user confirmation. This is
... intentional for E2E testing of the WF17 scenario where
... tool invocations should route to the container.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Timeout] 20 minutes
Skip If No LLM Keys
@@ -314,7 +314,7 @@ WF17 TDD Dual Mount Registration
... Bug #1078 is now fixed — --mount flag is implemented on
... ``resource add container-instance``.
... Tracked in #1078.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Tags] tdd_issue tdd_issue_1078 tdd_expected_fail tdd_issue_4178
[Timeout] 5 minutes
@@ -340,7 +340,7 @@ WF17 TDD Project Level Execution Env Priority Override
[Documentation] TDD test for AC #3: Set execution environment override priority
... at the project level via ``project context set
... --execution-env-priority override``.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
... Regression test for #1079: --execution-env-priority flag
... is now implemented on ``project context set``.
[Tags] tdd_issue tdd_issue_1079 tdd_expected_fail tdd_issue_4178
@@ -378,7 +378,7 @@ WF17 TDD Precedence Level 2 Project Override Resolution
[Documentation] TDD test for AC #5: Verify execution environment resolves via
... project-level override (precedence level 2) when no plan-level
... override is present.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
... Regression test for #1079 and #1080: project-level
... execution-env-priority is now implemented and the
... resolution logic honours it.
robot/e2e/wf18_container_clone.robot
+2 -2
View File
@@ -23,7 +23,7 @@ ${PROJECT_PREFIX} local/wf18-clone-proj
WF18 Suite Setup
[Documentation] E2E Suite Setup plus unique suffix generation and dynamic
... actor selection for WF18 tests.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
E2E Suite Setup
# Initialise the workspace so CLI commands work.
${init}= Run CleverAgents Command init --force --yes
@@ -99,7 +99,7 @@ WF18 Container With Remote Repo Clone Trusted Profile
... --clone-into for remote repo clone, two-step project setup,
... plan-level execution environment with fallback priority, and
... full plan lifecycle including apply with container commit/push.
[Tags] @tdd_issue:4188 @tdd_expected_fail
[Tags] @tdd_issue @tdd_issue_4188 @tdd_expected_fail
[Timeout] 20 minutes
[Teardown] Log WF18 Container Clone test completed.
Skip If No LLM Keys