test(e2e): workflow example 18 — container with remote repo clone (trusted profile) #820

Merged
hurui200320 merged 1 commits from test/e2e-wf18-container-clone into master 2026-03-30 11:40:57 +00:00
Owner

Summary

E2E test for Specification Workflow Example 18: Container with Remote Repo Clone using the trusted automation profile. Tests the full container-instance workflow including --clone-into, plan-level execution environment with fallback priority, and the complete plan lifecycle.

Closes #764

ISSUES CLOSED: #764

Changes

E2E Test (robot/e2e/wf18_container_clone.robot)

  • Container-instance resource with --clone-into: Registers a container-instance resource with the --clone-into REPO_URL:CONTAINER_PATH flag per the spec (not git-checkout).
  • Two-step project creation: Uses project create then project link-resource to match the spec's demonstrated flow for Example 18.
  • Dynamic actor selection: Detects ANTHROPIC_API_KEY vs OPENAI_API_KEY and selects the appropriate actor (following m6_acceptance.robot pattern).
  • Plan use with execution environment: Passes --execution-environment container and --execution-env-priority fallback with --automation-profile trusted. Note: the spec shows --execution-environment cloud/build-env (a resource name), but the current CLI only supports enum values (host or container); resource-name-based resolution is future work.
  • Full plan lifecycle: Strategize → Execute → Diff → Apply → Status.
  • lifecycle-apply --yes: Uses --yes flag to skip the confirmation prompt in non-interactive (CI) execution, consistent with all other E2E tests.
  • Soft assertions for AC-4/5/6: Container/clone evidence (AC-4), execution environment resolution evidence (AC-5), and apply commit/push evidence (AC-6) use Run Keyword And Warn On Failure to handle LLM output non-determinism.
  • Negative assertions: Traceback and INTERNAL error checks on every step.
  • Positive assertion on link-resource: Verifies the resource name appears in the output.
  • Pre/post commit count: Captures commit count before and after apply; asserts count does not decrease (flexible — LLM may or may not generate file changes).
  • Post-lifecycle fixture verification: Verifies fixture files (deploy_manager.py, Dockerfile) still exist after the full lifecycle.
  • ULID regex: Uses Crockford base32 character set [0-9A-HJKMNP-TV-Z]{26}.
  • Unique suffix: UUID-based suffix on resource/project/action names for parallel CI safety (all names include ${RUN_SUFFIX}).
  • Force Tags E2E: Suite-level tag instead of per-test [Tags].
  • [Timeout] 20 minutes: Comfortable margin over cumulative step timeouts.
  • [Teardown]: Per-test teardown that logs completion.
  • Two-call pattern comment: Explains the sequential plan execute calls (Strategize then Execute).
  • Dockerfile fixture comment: Clarifies that the Dockerfile is fixture data only, never built by the test.
  • Skip If No LLM Keys: Graceful skip when no LLM API keys are available.
  • --format plain: All CLI commands use --format plain for consistency.

CLI Enhancement (src/cleveragents/cli/commands/resource.py)

  • New --clone-into flag on resource add command. Accepts REPO_URL:CONTAINER_PATH format, stores the value as a clone_into property on the resource.
  • Type restriction: --clone-into is only accepted for container-instance and devcontainer-instance resource types. Using it with other types emits a validation error.
  • Format validation: Validates that the --clone-into value contains a : separator with non-empty segments on both sides. Uses rsplit(":", 1) to handle URLs with ports.

Behave Tests (features/resource_cli_coverage.feature)

  • 4 new scenarios for --clone-into CLI coverage:
    1. resource add container-instance with --clone-into stores clone_into property successfully.
    2. --clone-into on non-container type (git-checkout) fails with validation error.
    3. --clone-into with invalid format (no colon separator) fails with validation error.
    4. --clone-into with missing path segment fails with validation error.

CHANGELOG (CHANGELOG.md)

  • Updated entry to clearly indicate --clone-into is a new CLI flag introduced in this PR, with input validation and type restriction.

Rebase and Fix Notes

  • Rebased onto latest master (commit 5ddc04bc), resolving merge conflicts in CHANGELOG.md and src/cleveragents/cli/commands/resource.py. Both the new --mount flag (from WF17/PR merged to master) and --clone-into flag (from this PR) are preserved.
  • Fixed broken E2E test: Added missing --yes flag to plan lifecycle-apply command call. The --yes flag was introduced to skip the confirmation prompt in non-interactive mode; all other E2E tests already use it. Without --yes, the command failed with rc=1 when run in CI (no terminal available for prompting).

Manual Verification

Prerequisites

  • ANTHROPIC_API_KEY or OPENAI_API_KEY environment variable set

Commands

nox -e e2e_tests   # runs all E2E tests including WF18

What to Look For

  • WF18 test passes (or skips gracefully if no LLM keys)
  • All other E2E tests still pass
  • nox -e lint, nox -e typecheck, nox -e unit_tests, nox -e integration_tests, nox -e coverage_report all pass
## Summary E2E test for Specification Workflow Example 18: Container with Remote Repo Clone using the `trusted` automation profile. Tests the full container-instance workflow including `--clone-into`, plan-level execution environment with `fallback` priority, and the complete plan lifecycle. Closes #764 ISSUES CLOSED: #764 ## Changes ### E2E Test (`robot/e2e/wf18_container_clone.robot`) - **Container-instance resource with `--clone-into`:** Registers a `container-instance` resource with the `--clone-into REPO_URL:CONTAINER_PATH` flag per the spec (not `git-checkout`). - **Two-step project creation:** Uses `project create` then `project link-resource` to match the spec's demonstrated flow for Example 18. - **Dynamic actor selection:** Detects `ANTHROPIC_API_KEY` vs `OPENAI_API_KEY` and selects the appropriate actor (following `m6_acceptance.robot` pattern). - **Plan use with execution environment:** Passes `--execution-environment container` and `--execution-env-priority fallback` with `--automation-profile trusted`. Note: the spec shows `--execution-environment cloud/build-env` (a resource name), but the current CLI only supports enum values (`host` or `container`); resource-name-based resolution is future work. - **Full plan lifecycle:** Strategize → Execute → Diff → Apply → Status. - **`lifecycle-apply --yes`:** Uses `--yes` flag to skip the confirmation prompt in non-interactive (CI) execution, consistent with all other E2E tests. - **Soft assertions for AC-4/5/6:** Container/clone evidence (AC-4), execution environment resolution evidence (AC-5), and apply commit/push evidence (AC-6) use `Run Keyword And Warn On Failure` to handle LLM output non-determinism. - **Negative assertions:** `Traceback` and `INTERNAL` error checks on every step. - **Positive assertion on link-resource:** Verifies the resource name appears in the output. - **Pre/post commit count:** Captures commit count before and after apply; asserts count does not decrease (flexible — LLM may or may not generate file changes). - **Post-lifecycle fixture verification:** Verifies fixture files (`deploy_manager.py`, `Dockerfile`) still exist after the full lifecycle. - **ULID regex:** Uses Crockford base32 character set `[0-9A-HJKMNP-TV-Z]{26}`. - **Unique suffix:** UUID-based suffix on resource/project/action names for parallel CI safety (all names include `${RUN_SUFFIX}`). - **`Force Tags E2E`:** Suite-level tag instead of per-test `[Tags]`. - **`[Timeout] 20 minutes`:** Comfortable margin over cumulative step timeouts. - **`[Teardown]`:** Per-test teardown that logs completion. - **Two-call pattern comment:** Explains the sequential `plan execute` calls (Strategize then Execute). - **Dockerfile fixture comment:** Clarifies that the Dockerfile is fixture data only, never built by the test. - **`Skip If No LLM Keys`:** Graceful skip when no LLM API keys are available. - **`--format plain`:** All CLI commands use `--format plain` for consistency. ### CLI Enhancement (`src/cleveragents/cli/commands/resource.py`) - **New `--clone-into` flag** on `resource add` command. Accepts `REPO_URL:CONTAINER_PATH` format, stores the value as a `clone_into` property on the resource. - **Type restriction:** `--clone-into` is only accepted for `container-instance` and `devcontainer-instance` resource types. Using it with other types emits a validation error. - **Format validation:** Validates that the `--clone-into` value contains a `:` separator with non-empty segments on both sides. Uses `rsplit(":", 1)` to handle URLs with ports. ### Behave Tests (`features/resource_cli_coverage.feature`) - **4 new scenarios** for `--clone-into` CLI coverage: 1. `resource add container-instance` with `--clone-into` stores `clone_into` property successfully. 2. `--clone-into` on non-container type (`git-checkout`) fails with validation error. 3. `--clone-into` with invalid format (no colon separator) fails with validation error. 4. `--clone-into` with missing path segment fails with validation error. ### CHANGELOG (`CHANGELOG.md`) - Updated entry to clearly indicate `--clone-into` is a new CLI flag introduced in this PR, with input validation and type restriction. ## Rebase and Fix Notes - **Rebased onto latest `master`** (commit `5ddc04bc`), resolving merge conflicts in `CHANGELOG.md` and `src/cleveragents/cli/commands/resource.py`. Both the new `--mount` flag (from WF17/PR merged to master) and `--clone-into` flag (from this PR) are preserved. - **Fixed broken E2E test:** Added missing `--yes` flag to `plan lifecycle-apply` command call. The `--yes` flag was introduced to skip the confirmation prompt in non-interactive mode; all other E2E tests already use it. Without `--yes`, the command failed with rc=1 when run in CI (no terminal available for prompting). ## Manual Verification ### Prerequisites - `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` environment variable set ### Commands ```bash nox -e e2e_tests # runs all E2E tests including WF18 ``` ### What to Look For - WF18 test passes (or skips gracefully if no LLM keys) - All other E2E tests still pass - `nox -e lint`, `nox -e typecheck`, `nox -e unit_tests`, `nox -e integration_tests`, `nox -e coverage_report` all pass
freemo added this to the v3.8.0 milestone 2026-03-13 17:17:06 +00:00
freemo added the
State
In Review
Type
Testing
labels 2026-03-13 17:17:06 +00:00
freemo force-pushed test/e2e-wf18-container-clone from 5505f1669d to 4e307bc57d 2026-03-13 17:28:57 +00:00 Compare
freemo force-pushed test/e2e-wf18-container-clone from 4e307bc57d to 602cfbfd2d 2026-03-13 17:47:00 +00:00 Compare
freemo force-pushed test/e2e-wf18-container-clone from 602cfbfd2d to d840408f49 2026-03-13 18:13:20 +00:00 Compare
freemo force-pushed test/e2e-wf18-container-clone from d840408f49 to 0eccc8f397 2026-03-13 18:25:25 +00:00 Compare
freemo force-pushed test/e2e-wf18-container-clone from 0eccc8f397 to 70d9d17532 2026-03-13 23:19:54 +00:00 Compare
freemo added the
Priority
Medium
label 2026-03-14 04:10:25 +00:00
Author
Owner

PM Review — Day 34

Status: Mergeable, 0 reviews, M9 (v3.8.0)
Author: @freemo

E2E test for WF18 (container with remote repo clone, trusted profile). M9 — lowest urgency.

Action Items

Who Action Deadline
Peer review deferred to M9 sprint
## PM Review — Day 34 **Status**: Mergeable, 0 reviews, M9 (v3.8.0) **Author**: @freemo E2E test for WF18 (container with remote repo clone, trusted profile). M9 — lowest urgency. ### Action Items | Who | Action | Deadline | |-----|--------|----------| | Peer review deferred to M9 sprint | | |
freemo added a new dependency 2026-03-16 02:42:20 +00:00
freemo added a new dependency 2026-03-16 02:42:21 +00:00
Author
Owner

PM Status — Day 36 (2026-03-16)

Day 34 review assignment deadline check. This PR has 0 reviewer activity after 2 days.

Priority note: M3 PRs take precedence. Reviewers should complete M3 reviews first, then address M4+ PRs in milestone order.

Assigned reviewer: Please acknowledge and provide an ETA for your review, or flag if reassignment is needed.

## PM Status — Day 36 (2026-03-16) Day 34 review assignment deadline check. This PR has 0 reviewer activity after 2 days. **Priority note**: M3 PRs take precedence. Reviewers should complete M3 reviews first, then address M4+ PRs in milestone order. **Assigned reviewer**: Please acknowledge and provide an ETA for your review, or flag if reassignment is needed.
hurui200320 was assigned by freemo 2026-03-16 22:24:15 +00:00
Author
Owner

@hurui200320 I am going to have you take over this PR, it is mostly completed but is waiting on #628 and #966 One is yours and one is Brent's. Please be sure to get this PR and the two blocking PRs I listed in asap, thanks.

@hurui200320 I am going to have you take over this PR, it is mostly completed but is waiting on https://git.cleverthis.com/cleveragents/cleveragents-core/issues/628 and https://git.cleverthis.com/cleveragents/cleveragents-core/issues/966 One is yours and one is Brent's. Please be sure to get this PR and the two blocking PRs I listed in asap, thanks.
Author
Owner

PM Status — Day 37

Reviewers assigned. This PR needs at least 2 approving reviews per CONTRIBUTING.md before merge.

Author: Please ensure this PR is rebased on latest master and all quality gates pass before requesting merge.


PM status — Day 37

## PM Status — Day 37 Reviewers assigned. This PR needs at least 2 approving reviews per `CONTRIBUTING.md` before merge. **Author**: Please ensure this PR is rebased on latest `master` and all quality gates pass before requesting merge. --- *PM status — Day 37*
freemo requested review from hamza.khyari 2026-03-17 18:34:19 +00:00
hurui200320 force-pushed test/e2e-wf18-container-clone from 70d9d17532 to 310ae288f7 2026-03-18 08:35:48 +00:00 Compare
Author
Owner

Code Review — PR #820

(Cannot submit formal approval — self-authored PR.)

E2E test for WF18. Well-structured with proper labels, milestone, and issue linkage. No issues found.

## Code Review — PR #820 *(Cannot submit formal approval — self-authored PR.)* E2E test for WF18. Well-structured with proper labels, milestone, and issue linkage. No issues found.
freemo requested review from brent.edwards 2026-03-19 05:20:03 +00:00
freemo requested review from CoreRasurae 2026-03-19 05:20:03 +00:00
hurui200320 force-pushed test/e2e-wf18-container-clone from 310ae288f7 to 5e83950533 2026-03-20 07:59:39 +00:00 Compare
hurui200320 force-pushed test/e2e-wf18-container-clone from 5e83950533 to c658b928c3 2026-03-20 08:40:26 +00:00 Compare
hurui200320 force-pushed test/e2e-wf18-container-clone from c658b928c3 to 4f12abdf20 2026-03-20 09:38:44 +00:00 Compare
hurui200320 force-pushed test/e2e-wf18-container-clone from 4f12abdf20 to 13a51b1494 2026-03-23 04:10:55 +00:00 Compare
hurui200320 force-pushed test/e2e-wf18-container-clone from 13a51b1494 to be37ef3dcb 2026-03-24 05:40:21 +00:00 Compare
hurui200320 force-pushed test/e2e-wf18-container-clone from be37ef3dcb to 92cb31306d 2026-03-27 10:00:15 +00:00 Compare
hurui200320 force-pushed test/e2e-wf18-container-clone from 92cb31306d to 7f5c376a64 2026-03-30 09:54:54 +00:00 Compare
hurui200320 force-pushed test/e2e-wf18-container-clone from 7f5c376a64 to 3b47eccb43 2026-03-30 10:31:17 +00:00 Compare
hurui200320 force-pushed test/e2e-wf18-container-clone from 3b47eccb43 to 1613d6f4ad 2026-03-30 10:33:09 +00:00 Compare
hurui200320 force-pushed test/e2e-wf18-container-clone from 1613d6f4ad to 9cace15d6e 2026-03-30 11:17:52 +00:00 Compare
hurui200320 scheduled this pull request to auto merge when all checks succeed 2026-03-30 11:18:15 +00:00
hurui200320 merged commit 9cace15d6e into master 2026-03-30 11:40:57 +00:00
hurui200320 deleted branch test/e2e-wf18-container-clone 2026-03-30 11:40:57 +00:00
Sign in to join this conversation.
2 Participants
Notifications
Due Date
No due date set.
Reference: cleveragents/cleveragents-core#820