aditya
f0442e835d
CI / typecheck (pull_request) Successful in 50s
CI / lint (pull_request) Successful in 3m31s
CI / security (pull_request) Successful in 4m14s
CI / build (pull_request) Successful in 26s
CI / quality (pull_request) Successful in 3m45s
CI / helm (pull_request) Successful in 40s
CI / integration_tests (pull_request) Successful in 3m57s
CI / unit_tests (pull_request) Successful in 4m24s
CI / docker (pull_request) Successful in 1m29s
CI / coverage (pull_request) Successful in 10m54s
CI / benchmark-publish (pull_request) Has been skipped
CI / e2e_tests (pull_request) Successful in 15m35s
CI / status-check (pull_request) Successful in 1s
CI / benchmark-regression (pull_request) Successful in 1h0m6s
Integrate ACMS execute-phase context assembly into LLMExecuteActor and inject assembled context into execute prompts with resilient fallback when assembly fails or returns empty output. Wire the plan CLI executor to use an ACMS-backed execute context assembler, add Behave coverage for context injection/fallback/empty context, and extend Robot M5 verification helpers to assert execute-phase ACMS context usage. ISSUES CLOSED: #850
363 lines
19 KiB
Plaintext
363 lines
19 KiB
Plaintext
*** Settings ***
|
|
Documentation End-to-end verification of M5 success criteria:
|
|
... large-project creation with 10,000+ files,
|
|
... resource registration and linking,
|
|
... indexing verification, context tier management
|
|
... (hot/warm/cold), ACMS v1 pipeline with phase
|
|
... view inheritance and scoped context output.
|
|
...
|
|
... CLI acceptance criteria from the v3.4.0 milestone
|
|
... description are tested via actual subprocess calls
|
|
... to ``agents`` (``python -m cleveragents``).
|
|
... Each CLI test creates an isolated workspace with its
|
|
... own ``CLEVERAGENTS_HOME`` to prevent shared state.
|
|
...
|
|
... Context tier management and ACMS scoped context output
|
|
... criteria are validated at the Python API level via
|
|
... ``helper_m5_e2e_verification.py`` because the CLI does
|
|
... not yet expose dedicated context/ACMS inspection
|
|
... commands.
|
|
Resource ${CURDIR}/common.resource
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
${HELPER} ${CURDIR}/helper_m5_e2e_verification.py
|
|
|
|
*** Test Cases ***
|
|
# ================================================================
|
|
# CLI Success Criteria (milestone description)
|
|
# ================================================================
|
|
|
|
CLI Project Create Large Project
|
|
[Documentation] ``agents project create local/large-project`` creates
|
|
... a project via the real CLI subprocess.
|
|
${tmpdir}= Evaluate __import__('tempfile').mkdtemp(prefix='m5_cli_create_')
|
|
${init}= Run Process ${PYTHON} -m cleveragents init m5-cli-test
|
|
... timeout=120s on_timeout=kill cwd=${tmpdir}
|
|
... env:CLEVERAGENTS_HOME=${tmpdir}
|
|
Log ${init.stdout}
|
|
Log ${init.stderr}
|
|
Should Be Equal As Integers ${init.rc} 0
|
|
... msg=agents init failed (rc ${init.rc}): ${init.stderr}
|
|
${create}= Run Process ${PYTHON} -m cleveragents
|
|
... project create local/large-project
|
|
... timeout=120s on_timeout=kill cwd=${tmpdir}
|
|
... env:CLEVERAGENTS_HOME=${tmpdir}
|
|
Log ${create.stdout}
|
|
Log ${create.stderr}
|
|
Should Be Equal As Integers ${create.rc} 0
|
|
... msg=project create failed (rc ${create.rc}): ${create.stderr}
|
|
# Verify the project is persisted via list
|
|
${list}= Run Process ${PYTHON} -m cleveragents
|
|
... project list --format plain
|
|
... timeout=120s on_timeout=kill cwd=${tmpdir}
|
|
... env:CLEVERAGENTS_HOME=${tmpdir}
|
|
Log ${list.stdout}
|
|
Should Be Equal As Integers ${list.rc} 0
|
|
Should Contain ${list.stdout} local/large-project
|
|
... msg=project list should contain local/large-project
|
|
[Teardown] Remove Directory ${tmpdir} recursive=True
|
|
|
|
CLI Resource Add Git Checkout
|
|
[Documentation] ``agents resource add git-checkout local/large-repo
|
|
... --path /path/to/large/repo --branch main`` registers
|
|
... a large repository resource via the real CLI subprocess.
|
|
${tmpdir}= Evaluate __import__('tempfile').mkdtemp(prefix='m5_cli_res_')
|
|
# Create a fake repo directory to use as --path.
|
|
# The CLI does not validate that the path is a real git repository
|
|
# at registration time; it only records the location for later use.
|
|
${repo_dir}= Set Variable ${tmpdir}${/}large-repo
|
|
Create Directory ${repo_dir}
|
|
# Initialize workspace
|
|
${init}= Run Process ${PYTHON} -m cleveragents init m5-res-test
|
|
... timeout=120s on_timeout=kill cwd=${tmpdir}
|
|
... env:CLEVERAGENTS_HOME=${tmpdir}
|
|
Should Be Equal As Integers ${init.rc} 0
|
|
... msg=agents init failed: ${init.stderr}
|
|
# Register git-checkout resource
|
|
${add}= Run Process ${PYTHON} -m cleveragents resource add
|
|
... git-checkout local/large-repo
|
|
... --path ${repo_dir} --branch main
|
|
... timeout=120s on_timeout=kill cwd=${tmpdir}
|
|
... env:CLEVERAGENTS_HOME=${tmpdir}
|
|
Log ${add.stdout}
|
|
Log ${add.stderr}
|
|
Should Be Equal As Integers ${add.rc} 0
|
|
... msg=resource add failed (rc ${add.rc}): ${add.stderr}
|
|
# Verify resource is visible via show
|
|
${show}= Run Process ${PYTHON} -m cleveragents resource show
|
|
... --format plain local/large-repo
|
|
... timeout=120s on_timeout=kill cwd=${tmpdir}
|
|
... env:CLEVERAGENTS_HOME=${tmpdir}
|
|
Log ${show.stdout}
|
|
Should Be Equal As Integers ${show.rc} 0
|
|
... msg=resource show failed: ${show.stderr}
|
|
Should Contain ${show.stdout} local/large-repo
|
|
... msg=resource show should display resource name
|
|
Should Match Regexp ${show.stdout} branch.*main
|
|
... msg=resource show should display the configured branch (main)
|
|
[Teardown] Remove Directory ${tmpdir} recursive=True
|
|
|
|
CLI Project Link Resource
|
|
[Documentation] ``agents project link-resource local/large-project
|
|
... local/large-repo`` links the resource to the project
|
|
... via the real CLI subprocess.
|
|
${tmpdir}= Evaluate __import__('tempfile').mkdtemp(prefix='m5_cli_link_')
|
|
# Create a fake repo directory (CLI records location without git validation)
|
|
${repo_dir}= Set Variable ${tmpdir}${/}link-repo
|
|
Create Directory ${repo_dir}
|
|
# Initialize workspace
|
|
${init}= Run Process ${PYTHON} -m cleveragents init m5-link-test
|
|
... timeout=120s on_timeout=kill cwd=${tmpdir}
|
|
... env:CLEVERAGENTS_HOME=${tmpdir}
|
|
Should Be Equal As Integers ${init.rc} 0
|
|
# Create project
|
|
${create}= Run Process ${PYTHON} -m cleveragents
|
|
... project create local/large-project
|
|
... timeout=120s on_timeout=kill cwd=${tmpdir}
|
|
... env:CLEVERAGENTS_HOME=${tmpdir}
|
|
Should Be Equal As Integers ${create.rc} 0
|
|
... msg=project create failed: ${create.stderr}
|
|
# Register resource
|
|
${add}= Run Process ${PYTHON} -m cleveragents resource add
|
|
... git-checkout local/large-repo
|
|
... --path ${repo_dir} --branch main
|
|
... timeout=120s on_timeout=kill cwd=${tmpdir}
|
|
... env:CLEVERAGENTS_HOME=${tmpdir}
|
|
Should Be Equal As Integers ${add.rc} 0
|
|
... msg=resource add failed: ${add.stderr}
|
|
# Link resource to project
|
|
${link}= Run Process ${PYTHON} -m cleveragents
|
|
... project link-resource local/large-project local/large-repo
|
|
... timeout=120s on_timeout=kill cwd=${tmpdir}
|
|
... env:CLEVERAGENTS_HOME=${tmpdir}
|
|
Log ${link.stdout}
|
|
Log ${link.stderr}
|
|
Should Be Equal As Integers ${link.rc} 0
|
|
... msg=project link-resource failed (rc ${link.rc}): ${link.stderr}
|
|
# Verify the link command output confirms the linked resource
|
|
Should Contain ${link.stdout} local/large-repo
|
|
... msg=link-resource output should confirm the linked resource
|
|
Should Contain ${link.stdout} local/large-project
|
|
... msg=link-resource output should confirm the target project
|
|
# Capture the resource ULID so we can verify the exact ID in project show
|
|
${res_show}= Run Process ${PYTHON} -m cleveragents resource show
|
|
... --format plain local/large-repo
|
|
... timeout=120s on_timeout=kill cwd=${tmpdir}
|
|
... env:CLEVERAGENTS_HOME=${tmpdir}
|
|
Should Be Equal As Integers ${res_show.rc} 0
|
|
... msg=resource show failed: ${res_show.stderr}
|
|
${ulid_match}= Get Regexp Matches ${res_show.stdout}
|
|
... "resource_id":\\s*"([0-9A-Za-z]{26})" 1
|
|
${has_ulid}= Evaluate len($ulid_match) > 0
|
|
# Verify the link was persisted by checking project show output
|
|
${show}= Run Process ${PYTHON} -m cleveragents project show
|
|
... --format plain local/large-project
|
|
... timeout=120s on_timeout=kill cwd=${tmpdir}
|
|
... env:CLEVERAGENTS_HOME=${tmpdir}
|
|
Log ${show.stdout}
|
|
Should Be Equal As Integers ${show.rc} 0
|
|
... msg=project show after link failed: ${show.stderr}
|
|
Should Contain ${show.stdout} "resource_id":
|
|
... msg=project show should include linked resource entries after linking
|
|
# If we captured the resource ULID, verify the exact ID appears
|
|
IF ${has_ulid}
|
|
Should Contain ${show.stdout} ${ulid_match}[0]
|
|
... msg=project show should contain the exact linked resource ULID
|
|
END
|
|
[Teardown] Remove Directory ${tmpdir} recursive=True
|
|
|
|
CLI Project Show Displays Linked Resource
|
|
[Documentation] ``agents project show local/large-project`` displays
|
|
... project details with the specific linked resource ULID
|
|
... via the real CLI subprocess.
|
|
...
|
|
... This test differs from *CLI Project Link Resource* by
|
|
... capturing the resource's ULID from ``resource show``
|
|
... and asserting that exact ULID appears in
|
|
... ``project show`` output. It also verifies
|
|
... ``resource show`` independently after linking.
|
|
...
|
|
... Context tier and indexing fields are validated at the
|
|
... Python API level via ``helper_m5_e2e_verification.py``
|
|
... because the CLI does not yet render those fields.
|
|
${tmpdir}= Evaluate __import__('tempfile').mkdtemp(prefix='m5_cli_show_')
|
|
# Create a fake repo directory (CLI records location without git validation)
|
|
${repo_dir}= Set Variable ${tmpdir}${/}show-repo
|
|
Create Directory ${repo_dir}
|
|
# Initialize, create, register, link (full flow)
|
|
${init}= Run Process ${PYTHON} -m cleveragents init m5-show-test
|
|
... timeout=120s on_timeout=kill cwd=${tmpdir}
|
|
... env:CLEVERAGENTS_HOME=${tmpdir}
|
|
Should Be Equal As Integers ${init.rc} 0
|
|
${create}= Run Process ${PYTHON} -m cleveragents
|
|
... project create local/large-project
|
|
... timeout=120s on_timeout=kill cwd=${tmpdir}
|
|
... env:CLEVERAGENTS_HOME=${tmpdir}
|
|
Should Be Equal As Integers ${create.rc} 0
|
|
${add}= Run Process ${PYTHON} -m cleveragents resource add
|
|
... git-checkout local/large-repo
|
|
... --path ${repo_dir} --branch main
|
|
... timeout=120s on_timeout=kill cwd=${tmpdir}
|
|
... env:CLEVERAGENTS_HOME=${tmpdir}
|
|
Should Be Equal As Integers ${add.rc} 0
|
|
# Capture the resource ULID from resource show output so we can
|
|
# assert the specific ID appears in project show later (not just
|
|
# the generic "resource_id": key).
|
|
${res_show}= Run Process ${PYTHON} -m cleveragents resource show
|
|
... --format plain local/large-repo
|
|
... timeout=120s on_timeout=kill cwd=${tmpdir}
|
|
... env:CLEVERAGENTS_HOME=${tmpdir}
|
|
Log ${res_show.stdout}
|
|
Should Be Equal As Integers ${res_show.rc} 0
|
|
... msg=resource show failed: ${res_show.stderr}
|
|
Should Contain ${res_show.stdout} local/large-repo
|
|
... msg=resource show should display resource name
|
|
# Extract the resource_id ULID (26-char alphanumeric string) from output
|
|
${match}= Get Regexp Matches ${res_show.stdout}
|
|
... "resource_id":\\s*"([0-9A-Za-z]{26})" 1
|
|
${has_ulid}= Evaluate len($match) > 0
|
|
# Link the resource
|
|
${link}= Run Process ${PYTHON} -m cleveragents
|
|
... project link-resource local/large-project local/large-repo
|
|
... timeout=120s on_timeout=kill cwd=${tmpdir}
|
|
... env:CLEVERAGENTS_HOME=${tmpdir}
|
|
Should Be Equal As Integers ${link.rc} 0
|
|
# Verify resource show still works after linking (independent check)
|
|
${res_show2}= Run Process ${PYTHON} -m cleveragents resource show
|
|
... --format plain local/large-repo
|
|
... timeout=120s on_timeout=kill cwd=${tmpdir}
|
|
... env:CLEVERAGENTS_HOME=${tmpdir}
|
|
Should Be Equal As Integers ${res_show2.rc} 0
|
|
... msg=resource show after linking failed: ${res_show2.stderr}
|
|
# Show project and verify it displays project name, linked resource,
|
|
# and (when available) the specific resource ULID.
|
|
${show}= Run Process ${PYTHON} -m cleveragents project show
|
|
... --format plain local/large-project
|
|
... timeout=120s on_timeout=kill cwd=${tmpdir}
|
|
... env:CLEVERAGENTS_HOME=${tmpdir}
|
|
Log ${show.stdout}
|
|
Log ${show.stderr}
|
|
Should Be Equal As Integers ${show.rc} 0
|
|
... msg=project show failed (rc ${show.rc}): ${show.stderr}
|
|
Should Contain ${show.stdout} local/large-project
|
|
... msg=project show should display project name
|
|
Should Contain ${show.stdout} "resource_id":
|
|
... msg=project show should display linked resource entries
|
|
# If we captured the ULID, verify the exact resource_id appears
|
|
IF ${has_ulid}
|
|
Should Contain ${show.stdout} ${match}[0]
|
|
... msg=project show should contain the exact resource ULID (${match}[0])
|
|
END
|
|
[Teardown] Remove Directory ${tmpdir} recursive=True
|
|
|
|
# ================================================================
|
|
# Python API Technical Criteria (domain model + persistence)
|
|
# ================================================================
|
|
|
|
Large Project Creation With Ten Thousand Files
|
|
[Documentation] Create a project, register a git-checkout resource
|
|
... at a directory with 10,000+ files, link the resource,
|
|
... and verify the project shows the linked resource.
|
|
${result}= Run Process ${PYTHON} ${HELPER} project-create-large cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} m5-project-create-large-ok
|
|
|
|
Resource Registration And Linking To Project
|
|
[Documentation] Register multiple git-checkout resources and link
|
|
... them to a project. Verifies that show_resource
|
|
... returns correct type and name for each resource.
|
|
${result}= Run Process ${PYTHON} ${HELPER} resource-register-link cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} m5-resource-register-link-ok
|
|
|
|
Indexing Completes Without Timeout
|
|
[Documentation] Verify indexing completes by confirming the resource
|
|
... link round-trips through the database, resource
|
|
... details survive re-fetch, and location is preserved.
|
|
${result}= Run Process ${PYTHON} ${HELPER} indexing-complete cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} m5-indexing-complete-ok
|
|
|
|
Context Tier Management Hot Warm Cold
|
|
[Documentation] Verify the ContextConfig domain model correctly
|
|
... stores and returns hot_max_tokens, warm_max_decisions,
|
|
... and cold_max_decisions tier configuration values.
|
|
... Also verifies a project can carry tier config.
|
|
${result}= Run Process ${PYTHON} ${HELPER} context-tier-config cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} m5-context-tier-config-ok
|
|
|
|
ACMS Context Policy Set And Show
|
|
[Documentation] Set a context policy with default and strategize
|
|
... views via SQL persistence helpers, then read it
|
|
... back and verify all fields survive the round-trip.
|
|
${result}= Run Process ${PYTHON} ${HELPER} context-policy-set-show cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} m5-context-policy-set-show-ok
|
|
|
|
ACMS Phase View Inheritance Chain
|
|
[Documentation] Verify the ACMS view inheritance chain:
|
|
... default -> strategize -> execute -> apply.
|
|
... When a phase view is not set, resolve_view walks
|
|
... up the chain to the nearest ancestor.
|
|
${result}= Run Process ${PYTHON} ${HELPER} acms-phase-inheritance cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} m5-acms-phase-inheritance-ok
|
|
|
|
ACMS Scoped Context Output Per Phase
|
|
[Documentation] Verify ACMS produces scoped context output for
|
|
... each phase with narrowing size limits from
|
|
... strategize through execute to apply. Includes
|
|
... persistence round-trip verification.
|
|
${result}= Run Process ${PYTHON} ${HELPER} acms-scoped-context cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} m5-acms-scoped-context-ok
|
|
|
|
Execute Phase LLM Uses ACMS Context
|
|
[Documentation] Verify execute-phase plan prompting injects
|
|
... ACMS assembled context using execute-view
|
|
... policy (not strategize view) and excludes
|
|
... out-of-view fragments.
|
|
${result}= Run Process ${PYTHON} ${HELPER} llm-execute-acms-context cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} m5-llm-execute-acms-context-ok
|
|
|
|
Context Policy Clear And Inheritance Fallback
|
|
[Documentation] Verify that clearing a phase view causes
|
|
... resolve_view to fall back to the parent phase.
|
|
... Tests clear + persistence round-trip.
|
|
${result}= Run Process ${PYTHON} ${HELPER} context-policy-clear cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} m5-context-policy-clear-ok
|
|
|
|
Context View Validation Rules
|
|
[Documentation] Verify ContextView validation: zero and negative
|
|
... size limits are rejected, None is allowed,
|
|
... VALID_PHASES is correct, and invalid phases
|
|
... raise ValueError.
|
|
${result}= Run Process ${PYTHON} ${HELPER} context-view-validation cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} m5-context-view-validation-ok
|