forked from HAL9000/cleveragents-core
freemo
8ea00f5185
Co-authored-by: Jeffrey Phillips Freeman <the@jeffreyfreeman.me> Co-committed-by: Jeffrey Phillips Freeman <the@jeffreyfreeman.me>
239 lines
13 KiB
Plaintext
239 lines
13 KiB
Plaintext
*** Settings ***
|
|
Documentation TDD Issue #1028 — ACMS indexing pipeline not wired into CLI.
|
|
...
|
|
... Behavioral E2E tests proving that ``ContextTierService`` starts
|
|
... empty on every CLI invocation because the ACMS indexing pipeline
|
|
... is not wired into the CLI entry points. The ``project context
|
|
... simulate`` and ``project context inspect`` commands operate on
|
|
... zero data even when run against a project directory containing
|
|
... files.
|
|
...
|
|
... See CONTRIBUTING.md > Bug Fix Workflow for the full TDD
|
|
... issue-capture lifecycle.
|
|
Resource common_e2e.resource
|
|
Library OperatingSystem
|
|
Library String
|
|
Library Collections
|
|
Library Process
|
|
Suite Setup ACMS Behavioral Suite Setup
|
|
Suite Teardown ACMS Behavioral Suite Teardown
|
|
|
|
*** Variables ***
|
|
${PROJECT_SIMULATE} local/tdd-1028-simulate
|
|
${PROJECT_INSPECT} local/tdd-1028-inspect
|
|
${PROJECT_BUDGET} local/tdd-1028-budget
|
|
${PROJECT_SCALE} local/tdd-1028-scale
|
|
|
|
*** Keywords ***
|
|
ACMS Behavioral Suite Setup
|
|
[Documentation] Create an isolated workspace with ``agents init``, build a
|
|
... synthetic codebase, initialise a git repo, register it as a
|
|
... resource, and prepare for behavioral ACMS tests.
|
|
E2E Suite Setup
|
|
# Create workspace directory inside the suite home
|
|
${ws}= Set Variable ${SUITE_HOME}${/}workspace
|
|
Create Directory ${ws}
|
|
Set Suite Variable ${WS} ${ws}
|
|
# Initialize the CleverAgents workspace
|
|
${result}= Run CLI init m5-tdd-workspace
|
|
Log Init stdout: ${result.stdout} level=DEBUG
|
|
Log Init stderr: ${result.stderr} level=DEBUG
|
|
Should Be Equal As Integers ${result.rc} 0 msg=Workspace init failed (rc=${result.rc}). Check DEBUG-level log entries above.
|
|
# Create synthetic source files for context testing
|
|
Create Synthetic Codebase ${ws} TDD 1028 test
|
|
# Initialize a git repository — check every return code
|
|
${git_init}= Run Process git init cwd=${ws} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${git_init.rc} 0 msg=git init failed (rc=${git_init.rc}). Check DEBUG logs above.
|
|
${git_cfg_name}= Run Process git config user.name E2E Test cwd=${ws} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${git_cfg_name.rc} 0 msg=git config user.name failed (rc=${git_cfg_name.rc}). Check DEBUG logs above.
|
|
${git_cfg_email}= Run Process git config user.email e2e@test.local cwd=${ws} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${git_cfg_email.rc} 0 msg=git config user.email failed (rc=${git_cfg_email.rc}). Check DEBUG logs above.
|
|
${git_add}= Run Process git add . cwd=${ws} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${git_add.rc} 0 msg=git add failed (rc=${git_add.rc}). Check DEBUG logs above.
|
|
${git_commit}= Run Process git commit -m Initial commit cwd=${ws} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${git_commit.rc} 0 msg=git commit failed (rc=${git_commit.rc}). Check DEBUG logs above.
|
|
# Detect the default branch created by git init
|
|
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${ws} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${branch_result.rc} 0 msg=git rev-parse failed (rc=${branch_result.rc}). Check DEBUG logs above.
|
|
${branch}= Strip String ${branch_result.stdout}
|
|
Set Suite Variable ${WS_BRANCH} ${branch}
|
|
# Register the workspace as a git-checkout resource
|
|
${res_name}= Set Variable local/tdd-1028-ws-resource
|
|
Set Suite Variable ${WS_RESOURCE} ${res_name}
|
|
${r_add}= Run CLI resource add git-checkout ${res_name} --path ${ws} --branch ${branch}
|
|
Should Be Equal As Integers ${r_add.rc} 0 msg=resource add failed (rc=${r_add.rc}). Check DEBUG logs above.
|
|
Set Suite Variable ${SUITE_SETUP_COMPLETE} ${TRUE}
|
|
|
|
ACMS Behavioral Suite Teardown
|
|
[Documentation] Delegate to the common E2E teardown.
|
|
E2E Suite Teardown
|
|
|
|
*** Test Cases ***
|
|
# ----------------------------------------------------------------------- # TDD Issue #1028 — Behavioral ACMS validation
|
|
# # NOTE: Each test guards against incomplete suite setup with
|
|
# [Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
|
|
# If suite setup fails, this guard raises an error and the test
|
|
# will fail without executing any assertions. The
|
|
# m5_acceptance.robot tests cover the same CLI plumbing and will
|
|
# surface setup failures independently.
|
|
# -----------------------------------------------------------------------
|
|
|
|
Context Simulate Returns Non-Empty Tier Data
|
|
[Documentation] Run ``project context simulate`` against a project with
|
|
... files and assert that the output contains actual indexed
|
|
... fragments (not empty tiers).
|
|
...
|
|
... **Expected bug behavior:** ``total_tokens`` is 0 and
|
|
... ``fragment_count`` is 0 because the ACMS indexing pipeline
|
|
... is not wired into the CLI — ``ContextTierService`` starts
|
|
... empty on every invocation.
|
|
...
|
|
[Tags] tdd_bug tdd_bug_1028 tdd_issue tdd_issue_1028 E2E tdd_issue tdd_issue_4306 tdd_expected_fail
|
|
|
|
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
|
|
... msg=Prerequisite not met: suite setup did not complete
|
|
# Create project and configure context policy
|
|
Run CLI project create ${PROJECT_SIMULATE}
|
|
Link Resource To Project ${PROJECT_SIMULATE}
|
|
Run CLI ... project context set ${PROJECT_SIMULATE}
|
|
... --view default
|
|
... --include-path **/*.py
|
|
# Run simulate and check for actual indexed data
|
|
${result}= Run CLI
|
|
... project context simulate ${PROJECT_SIMULATE}
|
|
... --format json
|
|
${sim_json}= Extract JSON From Stdout ${result.stdout}
|
|
# Behavioral assertion: the project has Python files, so simulate
|
|
# must produce non-zero fragment data after indexing.
|
|
${fragment_count}= Evaluate int($sim_json.get('fragment_count', 0))
|
|
Should Be True ${fragment_count} > 0
|
|
... msg=Bug #1028: fragment_count is ${fragment_count} (expected > 0). ACMS indexing pipeline is not wired into CLI — ContextTierService starts empty.
|
|
${total_tokens}= Evaluate int($sim_json.get('total_tokens', 0))
|
|
Should Be True ${total_tokens} > 0
|
|
... msg=Bug #1028: total_tokens is ${total_tokens} (expected > 0). No fragments were indexed from project files.
|
|
|
|
Context Inspect Shows Indexed Resources
|
|
[Documentation] Run ``project context inspect`` against a project with
|
|
... files and assert that the indexed resource count is > 0.
|
|
...
|
|
... **Expected bug behavior:** ``tier_metrics`` counters are
|
|
... all zero because no indexing occurs — the ACMS pipeline
|
|
... is disconnected from the CLI.
|
|
...
|
|
[Tags] tdd_bug tdd_bug_1028 tdd_issue tdd_issue_1028 E2E tdd_issue tdd_issue_4306 tdd_expected_fail
|
|
|
|
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
|
|
... msg=Prerequisite not met: suite setup did not complete
|
|
# Create project and configure context policy
|
|
Run CLI project create ${PROJECT_INSPECT}
|
|
Link Resource To Project ${PROJECT_INSPECT}
|
|
Run CLI ... project context set ${PROJECT_INSPECT}
|
|
... --view default
|
|
... --include-path **/*.py
|
|
# Run inspect and check for indexed resources
|
|
${result}= Run CLI
|
|
... project context inspect ${PROJECT_INSPECT}
|
|
... --format json
|
|
${inspect_json}= Extract JSON From Stdout ${result.stdout}
|
|
# Behavioral assertion: at least one tier should have fragments
|
|
${metrics}= Evaluate $inspect_json.get('tier_metrics', {})
|
|
${hot_count}= Evaluate int($metrics.get('hot_count', 0))
|
|
${warm_count}= Evaluate int($metrics.get('warm_count', 0))
|
|
${cold_count}= Evaluate int($metrics.get('cold_count', 0))
|
|
${total_indexed}= Evaluate ${hot_count} + ${warm_count} + ${cold_count}
|
|
Should Be True ${total_indexed} > 0
|
|
... msg=Bug #1028: total indexed fragments is ${total_indexed} (expected > 0). tier_metrics: hot=${hot_count}, warm=${warm_count}, cold=${cold_count}. ACMS indexing pipeline is not wired into CLI.
|
|
|
|
Budget Enforcement Excludes Oversized Files
|
|
[Documentation] Configure ``max_file_size`` policy, add a file exceeding
|
|
... that limit, run simulate, and assert that the oversized
|
|
... file is excluded while smaller files are indexed.
|
|
...
|
|
... **Expected bug behavior:** ``fragment_count`` is 0 because
|
|
... the indexing pipeline does not run at all — regardless of
|
|
... ``max_file_size`` configuration, no files are scanned.
|
|
...
|
|
[Tags] tdd_bug tdd_bug_1028 tdd_issue tdd_issue_1028 E2E tdd_issue tdd_issue_4306 tdd_expected_fail
|
|
|
|
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
|
|
... msg=Prerequisite not met: suite setup did not complete
|
|
# Create project with tight max_file_size (1024 bytes)
|
|
Run CLI project create ${PROJECT_BUDGET}
|
|
Link Resource To Project ${PROJECT_BUDGET}
|
|
Run CLI ... project context set ${PROJECT_BUDGET}
|
|
... --view default
|
|
... --include-path **/*.py
|
|
... --max-file-size 1024
|
|
... --max-total-size 8192
|
|
# Simulate should index small files but exclude large_file.py (>1KiB)
|
|
${result}= Run CLI
|
|
... project context simulate ${PROJECT_BUDGET}
|
|
... --format json
|
|
${sim_json}= Extract JSON From Stdout ${result.stdout}
|
|
# Behavioral assertion: at least the small files (main.py, utils.py,
|
|
# config.py) should be indexed — large_file.py should be excluded by
|
|
# max_file_size policy. If fragment_count > 0, some files were indexed
|
|
# (and budget enforcement partially works).
|
|
# TODO(bugfix/m5-acms-cli-indexing-pipeline-wiring): After the indexing
|
|
# pipeline is wired, add a second assertion verifying that large_file.py
|
|
# is absent from the fragment list (i.e. budget enforcement actually
|
|
# excludes oversized files, not just that *some* files are indexed).
|
|
${fragment_count}= Evaluate int($sim_json.get('fragment_count', 0))
|
|
Should Be True ${fragment_count} > 0
|
|
... msg=Bug #1028: fragment_count is ${fragment_count} (expected > 0). Budget enforcement cannot exclude oversized files because the indexing pipeline does not run at all.
|
|
|
|
Large Project Indexes Without Timeout
|
|
[Documentation] Create a synthetic 10,000+ file project, run
|
|
... ``project context simulate``, and assert completion
|
|
... within a reasonable timeout with non-empty results.
|
|
...
|
|
... **Expected bug behavior:** The simulate command completes
|
|
... but returns zero fragments because the indexing pipeline
|
|
... is not wired — the 10K files are never scanned.
|
|
...
|
|
[Tags] tdd_bug tdd_bug_1028 tdd_issue tdd_issue_1028 E2E tdd_issue tdd_issue_4306 tdd_expected_fail
|
|
|
|
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
|
|
... msg=Prerequisite not met: suite setup did not complete
|
|
# Generate 10,000 tiny .py files in a subdirectory
|
|
${scale_dir}= Set Variable ${WS}${/}scale_src
|
|
Create Directory ${scale_dir}
|
|
${script}= Catenate SEPARATOR=\n
|
|
... import os, sys
|
|
... d = sys.argv[1]
|
|
... for i in range(10000):
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}with open(os.path.join(d, f"mod_{i:05d}.py"), "w") as f:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}f.write(f"# module {i}\\ndef fn_{i}(): return {i}\\n")
|
|
${gen_script}= Set Variable ${WS}${/}_gen_10k.py
|
|
Create File ${gen_script} ${script}
|
|
${gen}= Run Process ${PYTHON} ${gen_script} ${scale_dir}
|
|
... timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${gen.rc} 0 msg=10K file generation failed
|
|
# NOTE: The 10K generated files are written to the filesystem but NOT
|
|
# committed to the workspace git repo. The resource is registered as
|
|
# git-checkout type. The bug-fix developer MUST evaluate whether the
|
|
# fix routes indexing through the git sandbox (which only exposes
|
|
# git-tracked content) or the filesystem. If the fix uses git-tracked
|
|
# content, uncomment the following lines to commit the generated files:
|
|
# Run Process git add scale_src/ cwd=${WS}
|
|
# Run Process git commit -m Add 10K test files cwd=${WS}
|
|
# The m5_acceptance.robot structural test has the same pattern —
|
|
# neither suite commits generated files.
|
|
Remove File ${gen_script}
|
|
# Create project with include path covering the 10K files
|
|
Run CLI project create ${PROJECT_SCALE}
|
|
Link Resource To Project ${PROJECT_SCALE}
|
|
Run CLI ... project context set ${PROJECT_SCALE}
|
|
... --view default
|
|
... --include-path scale_src/**/*.py
|
|
# Simulate must complete within 600s (10 minutes) and return data
|
|
${result}= Run CLI
|
|
... project context simulate ${PROJECT_SCALE}
|
|
... --format json timeout=600s
|
|
${sim_json}= Extract JSON From Stdout ${result.stdout}
|
|
# Behavioral assertion: with 10,000+ files, fragment_count must be
|
|
# non-zero if the indexing pipeline is operational.
|
|
${fragment_count}= Evaluate int($sim_json.get('fragment_count', 0))
|
|
Should Be True ${fragment_count} > 0
|
|
... msg=Bug #1028: fragment_count is ${fragment_count} (expected > 0). 10,000+ files were generated but none were indexed. ACMS indexing pipeline is not wired into CLI.
|