test(e2e): E2E acceptance criteria for M5 (v3.4.0) — ACMS v1 and context scaling #811

Merged
hurui200320 merged 2 commits from test/e2e-m5-acceptance into master 2026-03-19 06:53:06 +00:00

2 Commits

Author SHA1 Message Date
hurui200320 19964c75c3 Merge branch 'master' into test/e2e-m5-acceptance
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 19s
CI / quality (pull_request) Successful in 30s
CI / build (pull_request) Successful in 30s
CI / security (pull_request) Successful in 45s
CI / typecheck (pull_request) Successful in 46s
CI / unit_tests (pull_request) Successful in 3m8s
CI / integration_tests (pull_request) Successful in 3m34s
CI / docker (pull_request) Successful in 57s
CI / e2e_tests (pull_request) Successful in 5m24s
CI / coverage (pull_request) Successful in 6m56s
CI / benchmark-regression (pull_request) Failing after 31m11s
2026-03-19 06:45:20 +00:00
hurui200320 d878c04745 test(e2e): E2E acceptance criteria for M5 (v3.4.0) — ACMS v1 and context scaling
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 19s
CI / build (pull_request) Successful in 25s
CI / quality (pull_request) Successful in 30s
CI / typecheck (pull_request) Successful in 49s
CI / security (pull_request) Successful in 1m6s
CI / unit_tests (pull_request) Successful in 3m25s
CI / integration_tests (pull_request) Successful in 4m45s
CI / docker (pull_request) Successful in 1m0s
CI / e2e_tests (pull_request) Successful in 5m24s
CI / coverage (pull_request) Successful in 9m3s
CI / benchmark-regression (pull_request) Has been cancelled
Add robot/e2e/m5_acceptance.robot with 21 zero-mock E2E tests covering
all M5 acceptance criteria: context assembly (add/list/show/clear),
context scaling (10,000+ files without timeout), context policy
configuration (view-specific settings), budget enforcement
(max_file_size/max_total_size constraints), context analysis (ACMS
pipeline inspect/simulate), and plan execution with real LLM calls
(openai/gpt-4o-mini via plan use + plan resume).

Bug fixes discovered during E2E testing:

- Fix _save_policy_json commit bug: session.flush() -> session.commit()
  so policy changes persist after session.close() (project_context.py).
  Rollback wrapped in contextlib.suppress(Exception) to prevent masking
  the original commit exception.

- Add session_factory DI provider to Container as providers.Singleton so
  the four project context commands (set/show/inspect/simulate) resolve
  a real sessionmaker instead of hitting a missing attribute.  Return
  type annotated as sessionmaker[Session] using future annotations and
  TYPE_CHECKING guard (container.py).

- Propagate GEMINI_API_KEY in nox e2e_tests session (noxfile.py).

- Add Google/Gemini API key pattern (AIzaSy...) to secret redaction
  in redaction.py.

Review feedback addressed across eight review passes:

- Budget enforcement test verifies large_file.py exclusion by
  max_file_size constraint (not tautological token comparison).
- 10K scaling test validates simulation returns valid fragment_count
  (non-negative; non-zero deferred to full ACMS pipeline integration).
- Plan resume test parses JSON via Extract JSON From Stdout with
  TRY/EXCEPT — consistent with plan use test pattern.
- __import__("sqlalchemy") replaced with proper import in step defs.
- Context clear test verifies all three files removed (config/main/utils).
- Inspect test asserts tier_metrics and tier_budget are non-empty.
- _save_policy_json rollback wrapped in contextlib.suppress(Exception).
- Skip If No LLM Keys rewritten to use Evaluate with os.environ.get —
  keys never stored in any Robot variable.
- All assertion messages follow safe pattern (no stdout/stderr embedding).
- Review finding ID prefixes removed from all comments.
- Type annotations added to all 8 _build_session_factory step functions.
- Default view verification checks exclude-path __pycache__.
- Prerequisite skip guards added to sections 2-4 (Variable Should Exist).
- Plan ID extraction moved inside TRY block.

Results: 25/25 E2E (21 M5 + 2 smoke + 2 M1), 1536/1536 Robot
integration, 383/383 Behave features (10,926 scenarios), coverage 97%.

ISSUES CLOSED: #745
2026-03-19 06:31:05 +00:00