hurui200320
51aab18411
CI / benchmark-publish (pull_request) Has been skipped
CI / helm (pull_request) Successful in 26s
CI / lint (pull_request) Successful in 29s
CI / push-validation (pull_request) Successful in 28s
CI / quality (pull_request) Successful in 31s
CI / build (pull_request) Successful in 32s
CI / typecheck (pull_request) Successful in 59s
CI / security (pull_request) Successful in 59s
CI / e2e_tests (pull_request) Successful in 3m0s
CI / integration_tests (pull_request) Successful in 4m3s
CI / unit_tests (pull_request) Successful in 5m21s
CI / docker (pull_request) Successful in 1m19s
CI / coverage (pull_request) Successful in 10m25s
CI / status-check (pull_request) Successful in 1s
CI / push-validation (push) Successful in 17s
CI / build (push) Successful in 27s
CI / helm (push) Successful in 29s
CI / quality (push) Successful in 32s
CI / lint (push) Successful in 35s
CI / security (push) Successful in 51s
CI / typecheck (push) Successful in 51s
CI / benchmark-regression (push) Has been skipped
CI / e2e_tests (push) Successful in 3m7s
CI / integration_tests (push) Successful in 4m0s
CI / unit_tests (push) Successful in 4m54s
CI / docker (push) Successful in 21s
CI / coverage (push) Successful in 10m24s
CI / status-check (push) Successful in 1s
CI / benchmark-regression (pull_request) Successful in 59m2s
CI / benchmark-publish (push) Successful in 1h13m43s
Add three guard conditions to the Robot Framework tdd_expected_fail_listener end_test() function, paralleling the Behave apply_tdd_inversion() guards in features/environment.py. These guards prevent the listener from blindly inverting ALL test failures to passes, which was masking infrastructure errors and causing flaky CI behavior. Guards added: 1. Setup/teardown error guard (_has_setup_teardown_failure): checks result.setup/teardown status and message prefix to detect infrastructure failures where the test body never executed. 2. Non-assertion failure guard (_is_infrastructure_error): checks result.message against known infrastructure error patterns (keyword resolution errors, Python exception types, network errors) to avoid inverting failures that are not the captured bug. 3. Dry-run guard: detects Robot Framework dry-run mode by checking if all body keywords have status NOT RUN, preventing meaningless inversions when no test actually executed. Also fixes collateral issues exposed by the guards: - Fixed Variable Should Exist syntax error in e2e test files (single space was being parsed as part of keyword name instead of separator) - Removed tdd_expected_fail from 4 context assembly e2e tests where the bugs appear to be fixed (previously masked by syntax error + blind inversion) ISSUES CLOSED: #5436
597 lines
31 KiB
Plaintext
597 lines
31 KiB
Plaintext
*** Settings ***
|
||
Documentation M5 (v3.4.0) E2E acceptance test — ACMS v1 and context scaling.
|
||
...
|
||
... Exercises context policy configuration, context assembly,
|
||
... context analysis, and plan execution through real CLI
|
||
... commands with real LLM API keys. This suite uses **zero
|
||
... mocking** — every LLM call hits a real provider endpoint.
|
||
...
|
||
... **Structural vs. behavioural scope:** Tests in sections
|
||
... 1b–4 that use ``project context simulate`` or ``inspect``
|
||
... are *structural / plumbing* validations — they verify CLI
|
||
... execution, JSON serialization, and stored configuration but
|
||
... do **not** exercise actual ACMS indexing or budget enforcement
|
||
... because the ``ContextTierService`` is an in-memory singleton
|
||
... that starts empty per CLI process. Each affected test has
|
||
... a ``[Documentation]`` note explaining this limitation.
|
||
... Behavioural ACMS validation is deferred until the full
|
||
... indexing pipeline is wired.
|
||
Resource common_e2e.resource
|
||
Library OperatingSystem
|
||
Library String
|
||
Library Collections
|
||
Library Process
|
||
Suite Setup M5 Acceptance Suite Setup
|
||
Suite Teardown M5 Acceptance Suite Teardown
|
||
|
||
*** Variables ***
|
||
${PROJECT_POLICY} local/m5-policy
|
||
${PROJECT_BUDGET} local/m5-budget
|
||
${PROJECT_ANALYSIS} local/m5-analysis
|
||
${PROJECT_PLAN} local/m5-plan
|
||
|
||
*** Keywords ***
|
||
M5 Acceptance Suite Setup
|
||
[Documentation] Create an isolated workspace with ``agents init``, build a
|
||
... synthetic codebase, initialise a git repo, register it as a
|
||
... resource, and disable mock AI.
|
||
E2E Suite Setup
|
||
# Create workspace directory inside the suite home
|
||
${ws}= Set Variable ${SUITE_HOME}${/}workspace
|
||
Create Directory ${ws}
|
||
Set Suite Variable ${WS} ${ws}
|
||
# Initialize the CleverAgents workspace
|
||
${result}= Run CLI init m5-workspace
|
||
Log Init stdout: ${result.stdout} level=DEBUG
|
||
Log Init stderr: ${result.stderr} level=DEBUG
|
||
Should Be Equal As Integers ${result.rc} 0 msg=Workspace init failed (rc=${result.rc}). Check DEBUG-level log entries above.
|
||
# Create synthetic source files for context testing
|
||
Create Synthetic Codebase ${ws} M5 test
|
||
# Initialize a git repository — check every return code
|
||
${git_init}= Run Process git init cwd=${ws} timeout=60s on_timeout=kill
|
||
Should Be Equal As Integers ${git_init.rc} 0 msg=git init failed (rc=${git_init.rc}). Check DEBUG logs above.
|
||
${git_cfg_name}= Run Process git config user.name E2E Test cwd=${ws} timeout=60s on_timeout=kill
|
||
Should Be Equal As Integers ${git_cfg_name.rc} 0 msg=git config user.name failed
|
||
${git_cfg_email}= Run Process git config user.email e2e@test.local cwd=${ws} timeout=60s on_timeout=kill
|
||
Should Be Equal As Integers ${git_cfg_email.rc} 0 msg=git config user.email failed
|
||
${git_cfg_gpg}= Run Process git config commit.gpgsign false cwd=${ws} timeout=60s on_timeout=kill
|
||
Should Be Equal As Integers ${git_cfg_gpg.rc} 0 msg=git config commit.gpgsign failed
|
||
${git_add}= Run Process git add . cwd=${ws} timeout=60s on_timeout=kill
|
||
Should Be Equal As Integers ${git_add.rc} 0 msg=git add failed (rc=${git_add.rc}). Check DEBUG logs above.
|
||
${git_commit}= Run Process git commit -m Initial commit cwd=${ws} timeout=60s on_timeout=kill
|
||
Should Be Equal As Integers ${git_commit.rc} 0 msg=git commit failed (rc=${git_commit.rc}). Check DEBUG logs above.
|
||
# Detect the default branch created by git init
|
||
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${ws} timeout=60s on_timeout=kill
|
||
Should Be Equal As Integers ${branch_result.rc} 0 msg=git rev-parse failed
|
||
${branch}= Strip String ${branch_result.stdout}
|
||
Set Suite Variable ${WS_BRANCH} ${branch}
|
||
# Register the workspace as a git-checkout resource and make it
|
||
# available for linking to individual projects.
|
||
${res_name}= Set Variable local/m5-ws-resource
|
||
Set Suite Variable ${WS_RESOURCE} ${res_name}
|
||
${r_add}= Run CLI resource add git-checkout ${res_name} --path ${ws} --branch ${branch}
|
||
Should Be Equal As Integers ${r_add.rc} 0 msg=resource add failed (rc=${r_add.rc}). Check DEBUG logs above.
|
||
# Mark suite setup as complete so Section 1 tests can skip cleanly
|
||
# if setup failed partway through.
|
||
Set Suite Variable ${SUITE_SETUP_COMPLETE} ${TRUE}
|
||
|
||
M5 Acceptance Suite Teardown
|
||
[Documentation] Delegate to the common E2E teardown.
|
||
E2E Suite Teardown
|
||
|
||
Combined Output
|
||
[Documentation] Return the concatenation of stdout and stderr (lowercased).
|
||
[Arguments] ${result}
|
||
${raw}= Set Variable ${result.stdout}\n${result.stderr}
|
||
${combined}= Convert To Lower Case ${raw}
|
||
RETURN ${combined}
|
||
|
||
Skip If No OpenAI Key
|
||
[Documentation] Skip the current test if ``OPENAI_API_KEY`` is not set.
|
||
... Uses ``Evaluate`` with ``os.environ.get`` to avoid
|
||
... storing the key value in any Robot variable.
|
||
${is_missing}= Evaluate len(os.environ.get('OPENAI_API_KEY', '')) == 0 modules=os
|
||
Skip If ${is_missing} OPENAI_API_KEY not set — required for openai/gpt-4o-mini tests.
|
||
|
||
Plan Test Setup
|
||
[Documentation] Setup for plan execution tests — skip if no API key
|
||
... and verify prerequisite variables exist.
|
||
[Arguments] @{required_vars}
|
||
Skip If No OpenAI Key
|
||
FOR ${var} IN @{required_vars}
|
||
Variable Should Exist \${${var}}
|
||
... msg=Prerequisite not met: ${var} not set — earlier test may have failed
|
||
END
|
||
|
||
*** Test Cases ***
|
||
# ----------------------------------------------------------------------- # 1. Context Assembly — context add / list / show / clear
|
||
# -----------------------------------------------------------------------
|
||
Context Assembly — Add Files To Context
|
||
[Documentation] ``context-load`` adds files; ``context list`` reflects them.
|
||
[Tags] E2E tdd_issue tdd_issue_4189
|
||
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
|
||
... msg=Prerequisite not met: suite setup did not complete
|
||
${result}= Run CLI context-load main.py utils.py
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
${list}= Run CLI context list
|
||
Should Contain ${list.stdout} main.py
|
||
Should Contain ${list.stdout} utils.py
|
||
|
||
Context Assembly — Show File Content
|
||
[Documentation] ``context show <file>`` displays file content.
|
||
[Tags] E2E tdd_issue tdd_issue_4188
|
||
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
|
||
... msg=Prerequisite not met: suite setup did not complete
|
||
# Ensure files are loaded (idempotent)
|
||
Run CLI context-load main.py
|
||
${result}= Run CLI context show main.py
|
||
Should Contain ${result.stdout} Hello from M5 test
|
||
|
||
Context Assembly — Show Context Summary
|
||
[Documentation] ``context show`` (no args) displays a summary.
|
||
... Verifies the output is non-empty, contains structural
|
||
... keywords, and that the CLI exit code is 0. Also
|
||
... checks the output is not just an error message by
|
||
... verifying the absence of common error indicators.
|
||
[Tags] E2E tdd_issue tdd_issue_4188
|
||
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
|
||
... msg=Prerequisite not met: suite setup did not complete
|
||
Run CLI context-load main.py utils.py
|
||
${result}= Run CLI context show
|
||
Should Not Be Empty ${result.stdout}
|
||
# The summary lists aggregate stats (file count, total size) — not
|
||
# individual file names. Verify the structural summary content.
|
||
${combined}= Combined Output ${result}
|
||
Should Contain ${combined} context summary msg=Expected summary header in context show output
|
||
Should Contain ${combined} total files msg=Expected file count in context show output
|
||
# Guard against false positives: ensure the output is not an error.
|
||
Should Not Contain ${combined} traceback msg=Output should not contain a traceback
|
||
Should Not Contain ${combined} error: msg=Output should not contain an error message
|
||
|
||
Context Assembly — Clear Context
|
||
[Documentation] ``context clear --yes`` removes all loaded files.
|
||
... Asserts files are present before clearing so the
|
||
... ``Should Not Contain`` checks are not vacuously true.
|
||
[Tags] E2E tdd_issue tdd_issue_4188
|
||
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
|
||
... msg=Prerequisite not met: suite setup did not complete
|
||
Run CLI context-load config.py
|
||
# Verify precondition: config.py IS loaded before we clear
|
||
${list_before}= Run CLI context list
|
||
Should Contain ${list_before.stdout} config.py
|
||
... msg=Precondition failed: config.py should be loaded before clear
|
||
${clear}= Run CLI context clear --yes
|
||
Should Be Equal As Integers ${clear.rc} 0
|
||
${list}= Run CLI context list
|
||
Should Not Contain ${list.stdout} config.py msg=config.py should be removed after clear
|
||
Should Not Contain ${list.stdout} main.py msg=main.py should be removed after clear
|
||
Should Not Contain ${list.stdout} utils.py msg=utils.py should be removed after clear
|
||
|
||
# ----------------------------------------------------------------------- # 1b. Context Scaling — 10,000+ files
|
||
# -----------------------------------------------------------------------
|
||
Context Scaling — Structural Plumbing for 10K File Projects
|
||
[Documentation] **Structural / plumbing test.** Generates 10,000+ small
|
||
... Python files, creates a project with ``--include-path``,
|
||
... and verifies that ``project context simulate`` completes
|
||
... without timeout and returns well-formed JSON.
|
||
...
|
||
... *Limitation:* ``project context simulate`` does **not**
|
||
... perform filesystem scanning — the ``ContextTierService``
|
||
... is an in-memory singleton that starts empty per CLI
|
||
... process. Therefore ``fragment_count`` is 0 and
|
||
... ``total_tokens`` is 0 regardless of the generated files.
|
||
... This test validates the CLI plumbing (policy storage,
|
||
... simulate command execution, JSON serialization) but
|
||
... **not** actual 10K-file indexing. Behavioral validation
|
||
... of M5 criterion *"Projects with 10,000+ files index
|
||
... without timeout"* is deferred until the full ACMS
|
||
... indexing pipeline is wired.
|
||
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
|
||
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
|
||
... msg=Prerequisite not met: suite setup did not complete
|
||
# Generate 10,000 tiny .py files in a subdirectory
|
||
${scale_dir}= Set Variable ${WS}${/}scale_src
|
||
Create Directory ${scale_dir}
|
||
${script}= Catenate SEPARATOR=\n
|
||
... import os, sys
|
||
... d = sys.argv[1]
|
||
... for i in range(10000):
|
||
... ${SPACE}${SPACE}${SPACE}${SPACE}with open(os.path.join(d, f"mod_{i:05d}.py"), "w") as f:
|
||
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}f.write(f"# module {i}\\ndef fn_{i}(): return {i}\\n")
|
||
${gen_script}= Set Variable ${WS}${/}_gen_10k.py
|
||
Create File ${gen_script} ${script}
|
||
${gen}= Run Process ${PYTHON} ${gen_script} ${scale_dir}
|
||
... timeout=120s on_timeout=kill
|
||
Should Be Equal As Integers ${gen.rc} 0 msg=10K file generation failed
|
||
# Create a project with the scale directory and link resource
|
||
${scale_proj}= Set Variable local/m5-scale
|
||
${result}= Run CLI project create ${scale_proj}
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
Link Resource To Project ${scale_proj}
|
||
${result}= Run CLI
|
||
... project context set ${scale_proj}
|
||
... --view default
|
||
... --include-path scale_src/**/*.py
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
# Simulate must complete within 600s (10 minutes)
|
||
${result}= Run CLI
|
||
... project context simulate ${scale_proj}
|
||
... --format json timeout=600s
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
# Parse JSON and verify simulation output is structurally valid.
|
||
# NOTE: All simulate assertions below are structural only — they verify
|
||
# the CLI returns well-formed JSON with the expected schema. Because the
|
||
# ContextTierService is empty (no indexing pipeline), total_tokens and
|
||
# fragment_count will be 0. This is a known limitation documented above.
|
||
${sim_json}= Extract JSON From Stdout ${result.stdout}
|
||
Should Be True 'total_tokens' in $sim_json
|
||
... msg=Simulate JSON missing 'total_tokens' field
|
||
Should Be True 'fragment_count' in $sim_json
|
||
... msg=Simulate JSON missing 'fragment_count' field
|
||
Should Be True type($sim_json.get('total_tokens')) in (int, float)
|
||
... msg=total_tokens should be numeric
|
||
Should Be True type($sim_json.get('fragment_count')) in (int, float)
|
||
... msg=fragment_count should be numeric
|
||
|
||
# ----------------------------------------------------------------------- # 2. Context Policy Configuration — view-specific settings
|
||
# -----------------------------------------------------------------------
|
||
Context Policy — Create Project For Policy Tests
|
||
[Documentation] Pre-requisite: create a project and link the workspace resource.
|
||
[Tags] E2E tdd_issue tdd_issue_4188
|
||
${result}= Run CLI project create ${PROJECT_POLICY}
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
Link Resource To Project ${PROJECT_POLICY}
|
||
Set Suite Variable ${POLICY_PROJECT_CREATED} ${TRUE}
|
||
|
||
Context Policy - Set Default View
|
||
[Documentation] Configure the default context view with include/exclude
|
||
... paths and file-size limits.
|
||
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
|
||
[Setup] Variable Should Exist ${POLICY_PROJECT_CREATED}
|
||
... msg=Prerequisite not met: policy project not created
|
||
${result}= Run CLI
|
||
... project context set ${PROJECT_POLICY}
|
||
... --view default
|
||
... --include-path src/**/*.py
|
||
... --exclude-path **/__pycache__/**
|
||
... --max-file-size 262144
|
||
... --max-total-size 1048576
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
|
||
Context Policy - Set Strategize View Override
|
||
[Documentation] The strategize view overrides the default with tighter
|
||
... limits and additional include paths.
|
||
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
|
||
[Setup] Variable Should Exist ${POLICY_PROJECT_CREATED}
|
||
... msg=Prerequisite not met: policy project not created
|
||
${result}= Run CLI
|
||
... project context set ${PROJECT_POLICY}
|
||
... --view strategize
|
||
... --include-path src/**/*.py
|
||
... --include-path docs/**/*.md
|
||
... --max-file-size 131072
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
|
||
Context Policy - Verify Default View
|
||
[Documentation] ``project context show --view default`` returns the
|
||
... configured values. Uses parsed JSON assertions to
|
||
... avoid substring collisions (e.g. ``1024`` matching
|
||
... inside ``10240``).
|
||
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
|
||
[Setup] Variable Should Exist ${POLICY_PROJECT_CREATED}
|
||
... msg=Prerequisite not met: policy project not created
|
||
${result}= Run CLI
|
||
... project context show ${PROJECT_POLICY}
|
||
... --view default --format json
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
${policy_json}= Extract JSON From Stdout ${result.stdout}
|
||
# When --view is specified, the JSON uses 'resolved_view' (not 'default_view').
|
||
${rv}= Evaluate $policy_json.get('resolved_view', {})
|
||
Should Be True $rv.get('max_file_size') == 262144
|
||
... msg=resolved default view max_file_size should be 262144
|
||
Should Be True $rv.get('max_total_size') == 1048576
|
||
... msg=resolved default view max_total_size should be 1048576
|
||
Should Be True 'src/**/*.py' in $rv.get('include_paths', [])
|
||
... msg=resolved default view should include src/**/*.py
|
||
Should Be True '**/__pycache__/**' in $rv.get('exclude_paths', [])
|
||
... msg=resolved default view should exclude **/__pycache__/**
|
||
|
||
Context Policy - Verify Strategize View
|
||
[Documentation] ``project context show --view strategize`` returns the
|
||
... overridden values. Uses parsed JSON assertions.
|
||
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
|
||
[Setup] Variable Should Exist ${POLICY_PROJECT_CREATED}
|
||
... msg=Prerequisite not met: policy project not created
|
||
${result}= Run CLI
|
||
... project context show ${PROJECT_POLICY}
|
||
... --view strategize --format json
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
${policy_json}= Extract JSON From Stdout ${result.stdout}
|
||
# When --view is specified, the JSON uses 'resolved_view'.
|
||
${rv}= Evaluate $policy_json.get('resolved_view', {})
|
||
Should Be True $rv.get('max_file_size') == 131072
|
||
... msg=resolved strategize view max_file_size should be 131072
|
||
Should Be True 'docs/**/*.md' in $rv.get('include_paths', [])
|
||
... msg=resolved strategize view should include docs/**/*.md
|
||
|
||
# ----------------------------------------------------------------------- # 3. Budget Enforcement — max_file_size / max_total_size
|
||
# -----------------------------------------------------------------------
|
||
Budget Enforcement — Create Project With Tight Budget
|
||
[Documentation] Create a project, link resource, and set tight budget constraints.
|
||
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
|
||
${result}= Run CLI project create ${PROJECT_BUDGET}
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
Link Resource To Project ${PROJECT_BUDGET}
|
||
${result}= Run CLI
|
||
... project context set ${PROJECT_BUDGET}
|
||
... --view default
|
||
... --include-path **/*.py
|
||
... --max-file-size 1024
|
||
... --max-total-size 4096
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
Set Suite Variable ${BUDGET_PROJECT_CREATED} ${TRUE}
|
||
|
||
Budget Enforcement — Verify Constraints Stored
|
||
[Documentation] The JSON policy output must contain the budget values.
|
||
... Uses parsed JSON assertions to avoid substring
|
||
... collisions (e.g. ``1024`` matching inside ``10240``).
|
||
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
|
||
Variable Should Exist ${BUDGET_PROJECT_CREATED}
|
||
... msg=Prerequisite not met: budget project not created
|
||
${result}= Run CLI
|
||
... project context show ${PROJECT_BUDGET}
|
||
... --view default --format json
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
${policy_json}= Extract JSON From Stdout ${result.stdout}
|
||
# When --view is specified, the JSON uses 'resolved_view'.
|
||
${rv}= Evaluate $policy_json.get('resolved_view', {})
|
||
Should Be True $rv.get('max_file_size') == 1024
|
||
... msg=Budget resolved view max_file_size should be 1024
|
||
Should Be True $rv.get('max_total_size') == 4096
|
||
... msg=Budget resolved view max_total_size should be 4096
|
||
|
||
Budget Enforcement — Simulate Context Assembly (Structural)
|
||
[Documentation] **Structural / plumbing test.** Verifies that
|
||
... ``project context simulate`` returns well-formed JSON
|
||
... reflecting the configured budget constraints in the
|
||
... ``acms_config`` section.
|
||
...
|
||
... *Limitation:* ``max_file_size`` filtering is only enforced
|
||
... in the repo indexing path (``repo_indexing_utils``), not
|
||
... in the simulate command. The ``ContextTierService`` is
|
||
... empty per process, so simulate operates on zero fragments.
|
||
... Behavioral budget enforcement (verifying ``large_file.py``
|
||
... exclusion) requires the full ACMS indexing pipeline and
|
||
... is deferred to a follow-up issue.
|
||
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
|
||
Variable Should Exist ${BUDGET_PROJECT_CREATED}
|
||
... msg=Prerequisite not met: budget project not created
|
||
${result}= Run CLI
|
||
... project context simulate ${PROJECT_BUDGET}
|
||
... --format json
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
# Parse JSON and verify structural fields present.
|
||
# NOTE: Assertions are structural only — they verify the CLI returns
|
||
# well-formed JSON with the expected schema. The ContextTierService is
|
||
# empty, so no actual file inclusion/exclusion is tested here.
|
||
# Budget constraints (max_file_size, max_total_size) are stored in the
|
||
# ContextView policy, not in the simulate output's acms_config.
|
||
${sim_json}= Extract JSON From Stdout ${result.stdout}
|
||
Should Be True 'acms_config' in $sim_json
|
||
... msg=Simulate JSON missing 'acms_config' field
|
||
Should Be True 'total_tokens' in $sim_json
|
||
... msg=Simulate JSON missing 'total_tokens' field
|
||
Should Be True 'fragment_count' in $sim_json
|
||
... msg=Simulate JSON missing 'fragment_count' field
|
||
Should Be True type($sim_json['total_tokens']) in (int, float)
|
||
... msg=total_tokens should be numeric
|
||
|
||
# ----------------------------------------------------------------------- # 4. Context Analysis — meaningful summaries
|
||
# -----------------------------------------------------------------------
|
||
Context Analysis — Create Project With ACMS Config
|
||
[Documentation] Create a project and configure ACMS pipeline parameters.
|
||
... Uses **non-default** values to avoid vacuous assertions —
|
||
... defaults are 8000 / 500 / 5000.
|
||
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
|
||
${result}= Run CLI project create ${PROJECT_ANALYSIS}
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
Link Resource To Project ${PROJECT_ANALYSIS}
|
||
${result}= Run CLI
|
||
... project context set ${PROJECT_ANALYSIS}
|
||
... --view default
|
||
... --include-path **/*.py
|
||
... --hot-max-tokens 12000
|
||
... --warm-max-decisions 750
|
||
... --cold-max-decisions 3500
|
||
... --temporal-scope current
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
Set Suite Variable ${ANALYSIS_PROJECT_CREATED} ${TRUE}
|
||
|
||
Context Analysis - Inspect Context Tiers (Structural)
|
||
[Documentation] **Structural / plumbing test.** Verifies that
|
||
... ``project context inspect`` returns well-formed JSON with
|
||
... the expected ``tier_metrics`` and ``tier_budget`` sections,
|
||
... and that each section contains the documented field names.
|
||
...
|
||
... *Limitation:* ``tier_metrics`` always has 7 keys with
|
||
... zero-value counters and ``tier_budget`` always has 3 keys
|
||
... with the configured values, regardless of whether any data
|
||
... was indexed — the ``ContextTierService`` is empty per
|
||
... process. These assertions verify JSON schema correctness,
|
||
... not ACMS behavioral state.
|
||
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
|
||
Variable Should Exist ${ANALYSIS_PROJECT_CREATED}
|
||
... msg=Prerequisite not met: analysis project not created
|
||
${result}= Run CLI
|
||
... project context inspect ${PROJECT_ANALYSIS}
|
||
... --format json
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
# Parse inspect output and verify tier data schema.
|
||
${inspect_json}= Extract JSON From Stdout ${result.stdout}
|
||
Should Be True 'tier_metrics' in $inspect_json
|
||
... msg=Inspect JSON missing 'tier_metrics' field
|
||
Should Be True 'tier_budget' in $inspect_json
|
||
... msg=Inspect JSON missing 'tier_budget' field
|
||
# Verify tier_metrics contains the expected field names (schema check).
|
||
${metrics}= Evaluate $inspect_json.get('tier_metrics', {})
|
||
Should Be True 'hot_count' in $metrics
|
||
... msg=tier_metrics missing 'hot_count' field
|
||
Should Be True 'warm_count' in $metrics
|
||
... msg=tier_metrics missing 'warm_count' field
|
||
Should Be True 'cold_count' in $metrics
|
||
... msg=tier_metrics missing 'cold_count' field
|
||
# Verify tier_budget has the expected field names (schema check).
|
||
# NOTE: tier_budget comes from the ContextTierService default TierBudget,
|
||
# not the ACMS config. Values are always the TierBudget defaults
|
||
# (8000/500/5000) because the tier service is a fresh singleton per
|
||
# CLI process with no indexed data.
|
||
${budget}= Evaluate $inspect_json.get('tier_budget', {})
|
||
Should Be True 'max_tokens_hot' in $budget
|
||
... msg=tier_budget missing 'max_tokens_hot' field
|
||
Should Be True 'max_decisions_warm' in $budget
|
||
... msg=tier_budget missing 'max_decisions_warm' field
|
||
Should Be True 'max_decisions_cold' in $budget
|
||
... msg=tier_budget missing 'max_decisions_cold' field
|
||
|
||
Context Analysis - Simulate Produces Structured Output
|
||
[Documentation] **Structural / plumbing test.** Verifies that
|
||
... ``project context simulate`` returns well-formed JSON
|
||
... containing the required schema fields (``total_tokens``,
|
||
... ``budget_used``, ``strategies_used``, ``context_hash``).
|
||
...
|
||
... *Limitation:* The ``ContextTierService`` is empty per
|
||
... process, so ``total_tokens`` is 0 and ``budget_used``
|
||
... is 0.0. These assertions verify JSON schema
|
||
... correctness, not ACMS behavioral output.
|
||
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
|
||
Variable Should Exist ${ANALYSIS_PROJECT_CREATED}
|
||
... msg=Prerequisite not met: analysis project not created
|
||
${result}= Run CLI
|
||
... project context simulate ${PROJECT_ANALYSIS}
|
||
... --format json
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
# Parse JSON and verify all expected schema fields are present.
|
||
${sim_json}= Extract JSON From Stdout ${result.stdout}
|
||
Should Be True 'total_tokens' in $sim_json
|
||
... msg=Simulate JSON missing 'total_tokens' field
|
||
Should Be True 'budget_used' in $sim_json
|
||
... msg=Simulate JSON missing 'budget_used' field
|
||
Should Be True 'strategies_used' in $sim_json
|
||
... msg=Simulate JSON missing 'strategies_used' field
|
||
Should Be True 'context_hash' in $sim_json
|
||
... msg=Simulate JSON missing 'context_hash' field
|
||
# Verify types are correct (structural schema check).
|
||
Should Be True type($sim_json['total_tokens']) in (int, float)
|
||
... msg=total_tokens should be numeric
|
||
Should Be True type($sim_json['budget_used']) in (int, float)
|
||
... msg=budget_used should be numeric
|
||
|
||
Context Analysis - Show Full Policy With ACMS Config
|
||
[Documentation] ``project context show`` includes the ACMS pipeline config.
|
||
... Uses parsed JSON assertions to avoid substring collisions
|
||
... (e.g. 500 matching inside 3500).
|
||
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
|
||
Variable Should Exist ${ANALYSIS_PROJECT_CREATED}
|
||
... msg=Prerequisite not met: analysis project not created
|
||
${result}= Run CLI
|
||
... project context show ${PROJECT_ANALYSIS}
|
||
... --format json
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
# Parse JSON and verify ACMS config values via dict access, avoiding
|
||
# fragile substring matches.
|
||
${policy_json}= Extract JSON From Stdout ${result.stdout}
|
||
${acms}= Evaluate $policy_json.get('acms_config', {})
|
||
Should Be True $acms.get('hot_max_tokens') == 12000
|
||
... msg=ACMS hot_max_tokens should be 12000 (configured value)
|
||
Should Be True $acms.get('warm_max_decisions') == 750
|
||
... msg=ACMS warm_max_decisions should be 750 (configured value)
|
||
Should Be True $acms.get('cold_max_decisions') == 3500
|
||
... msg=ACMS cold_max_decisions should be 3500 (configured value)
|
||
|
||
# ----------------------------------------------------------------------- # 5. Plan Execution — ACMS context with real LLM calls
|
||
# -----------------------------------------------------------------------
|
||
Plan Execution — Create Project And Configure ACMS
|
||
[Documentation] Set up a project with ACMS context for plan tests.
|
||
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
|
||
[Setup] Plan Test Setup
|
||
${result}= Run CLI project create ${PROJECT_PLAN}
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
Link Resource To Project ${PROJECT_PLAN}
|
||
${result}= Run CLI
|
||
... project context set ${PROJECT_PLAN}
|
||
... --view default
|
||
... --include-path **/*.py
|
||
... --hot-max-tokens 4000
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
# Track prerequisite completion for downstream tests
|
||
Set Suite Variable ${PLAN_PROJECT_CREATED} ${TRUE}
|
||
|
||
Plan Execution — Create Action
|
||
[Documentation] Create an action definition from YAML config.
|
||
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
|
||
Plan Test Setup PLAN_PROJECT_CREATED
|
||
${action_yaml}= Catenate SEPARATOR=\n
|
||
... name: local/m5-e2e-action
|
||
... description: M5 E2E acceptance test action
|
||
... strategy_actor: openai/gpt-4o-mini
|
||
... execution_actor: openai/gpt-4o-mini
|
||
... definition_of_done: |
|
||
... ${SPACE}${SPACE}The plan produces a valid strategy.
|
||
... reusable: true
|
||
... automation_profile: trusted
|
||
Create File ${WS}${/}m5_action.yaml ${action_yaml}
|
||
${result}= Run CLI action create --config m5_action.yaml
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
# Track prerequisite completion for downstream tests
|
||
Set Suite Variable ${PLAN_ACTION_CREATED} ${TRUE}
|
||
|
||
Plan Execution — Create Plan With Plan Use
|
||
[Documentation] ``plan use`` creates a plan in the Strategize phase
|
||
... referencing the ACMS-configured project. Uses the
|
||
... ``Extract JSON From Stdout`` keyword consistently with
|
||
... the rest of the suite (instead of fragile ``rindex``).
|
||
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
|
||
Plan Test Setup PLAN_ACTION_CREATED
|
||
${result}= Run CLI
|
||
... plan use local/m5-e2e-action ${PROJECT_PLAN}
|
||
... --format json timeout=300s
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
Should Contain ${result.stdout} plan_id msg=Plan use did not return JSON with plan_id
|
||
# Use Extract JSON From Stdout consistently with plan resume and other
|
||
# tests — avoids fragile rindex-based extraction that can pick the wrong
|
||
# '{' if structlog lines also contain plan_id.
|
||
${plan_data}= Extract JSON From Stdout ${result.stdout}
|
||
Should Be True 'plan_id' in $plan_data
|
||
... msg=Plan use JSON missing 'plan_id' field
|
||
Set Suite Variable ${PLAN_ID} ${plan_data}[plan_id]
|
||
|
||
Plan Execution — Resume Plan For LLM Processing
|
||
[Documentation] ``plan resume`` transitions the plan from queued to
|
||
... processing, triggering real LLM strategy generation
|
||
... using the ACMS context pipeline. Assertions are
|
||
... outside the TRY block so that field-level failures
|
||
... report the actual missing field, not a misleading
|
||
... "did not return valid JSON" message.
|
||
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
|
||
Plan Test Setup PLAN_ID
|
||
${result}= Run CLI
|
||
... plan resume ${PLAN_ID} --format json
|
||
... timeout=300s
|
||
Should Be Equal As Integers ${result.rc} 0
|
||
# Parse JSON — TRY block only guards the extraction itself.
|
||
TRY
|
||
${resume_json}= Extract JSON From Stdout ${result.stdout}
|
||
EXCEPT AS ${err}
|
||
Fail Plan resume did not return valid JSON: ${err}
|
||
END
|
||
# Assertions outside TRY so failures report the actual issue.
|
||
Should Be True 'plan_id' in $resume_json
|
||
... msg=Plan resume JSON missing 'plan_id' field
|
||
Should Be True 'phase' in $resume_json
|
||
... msg=Plan resume JSON missing 'phase' field
|
||
# Verify the phase transitioned to an expected state — a plan stuck
|
||
# in 'queued' would indicate resume had no effect.
|
||
${phase}= Evaluate $resume_json.get('phase', '')
|
||
Should Not Be Equal As Strings ${phase} queued
|
||
... msg=Plan phase should have transitioned from 'queued' after resume
|