TEST-INFRA: [ci-pipeline-design] Run E2E tests in a secure, isolated environment #1962

Open
opened 2026-04-03 00:24:27 +00:00 by freemo · 1 comment
Owner

Metadata

  • Branch: task/e2e-tests-secure-isolated-environment
  • Commit Message: feat(ci): run E2E tests in a secure, isolated environment
  • Milestone: v3.8.0
  • Parent Epic: #1678

Background and Context

The E2E test suite (robot/) currently executes directly on the shared CI runner without any environment isolation or security hardening. This creates two risks:

  1. Security: E2E tests exercise real agent execution paths — including tool calls, file system access, and potentially network I/O — on the same runner that handles lint, build, and other jobs. A compromised or misbehaving test could affect the runner's state or leak sensitive environment variables.
  2. Reproducibility: Without isolation, E2E tests can be polluted by artefacts left by previous jobs (e.g., stale .nox/ directories, leftover temp files, or environment variable leakage), causing non-deterministic failures.

Per the specification's quality mandate and the CI hardening effort tracked in Epic #1678, E2E tests must run in a dedicated, sandboxed environment that is torn down after each run.

Expected Behaviour

The e2e_tests CI job should:

  1. Spin up a dedicated, ephemeral container or sandbox (e.g., a Docker-in-Docker service, a restricted network namespace, or a dedicated runner label) for each run.
  2. Mount only the minimum required secrets and environment variables needed by the E2E suite — no CI tokens, registry credentials, or unrelated secrets should be visible inside the sandbox.
  3. Enforce a strict timeout-minutes on the job to prevent runaway tests from blocking shared runners.
  4. Tear down the sandbox completely after the job completes (pass or fail), leaving no artefacts on the host runner.
  5. Report test results (Robot Framework XML/HTML output) as CI artefacts for post-run inspection.

Acceptance Criteria

  • The e2e_tests job in .forgejo/workflows/ci.yml runs inside an isolated container or restricted environment (not directly on the shared runner host)
  • Only the minimum required environment variables and secrets are injected into the E2E sandbox
  • The job has an explicit timeout-minutes value
  • The sandbox is fully torn down after the job completes
  • Robot Framework XML/HTML output is uploaded as a CI artefact
  • features/ci_workflow_validation.feature is updated to assert the isolation and artefact-upload steps are present
  • All nox stages pass after the changes
  • Coverage ≥ 97% is maintained

Subtasks

  • Audit the current e2e_tests job in .forgejo/workflows/ci.yml to identify all secrets and environment variables it consumes
  • Design the isolation strategy (Docker-in-Docker service container, restricted runner label, or network namespace) and document the decision in a comment in the workflow file
  • Update the e2e_tests job to run inside the chosen isolated environment
  • Restrict secret/env injection to only what the E2E suite requires; remove all unneeded variables from the job scope
  • Add or verify an explicit timeout-minutes on the e2e_tests job
  • Add an upload-artifact step to publish Robot Framework XML/HTML output after the job completes (even on failure)
  • Update features/ci_workflow_validation.feature with Behave scenarios asserting isolation, secret scoping, timeout, and artefact upload are present
  • Run nox (all default sessions) locally and fix any errors
  • Verify coverage ≥ 97% via nox -s coverage_report

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly (feat(ci): run E2E tests in a secure, isolated environment), followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly (task/e2e-tests-secure-isolated-environment).
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
  • All nox stages pass.
  • Coverage ≥ 97%

Automated by CleverAgents Bot
Supervisor: Unknown | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/e2e-tests-secure-isolated-environment` - **Commit Message**: `feat(ci): run E2E tests in a secure, isolated environment` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The E2E test suite (`robot/`) currently executes directly on the shared CI runner without any environment isolation or security hardening. This creates two risks: 1. **Security**: E2E tests exercise real agent execution paths — including tool calls, file system access, and potentially network I/O — on the same runner that handles lint, build, and other jobs. A compromised or misbehaving test could affect the runner's state or leak sensitive environment variables. 2. **Reproducibility**: Without isolation, E2E tests can be polluted by artefacts left by previous jobs (e.g., stale `.nox/` directories, leftover temp files, or environment variable leakage), causing non-deterministic failures. Per the specification's quality mandate and the CI hardening effort tracked in Epic #1678, E2E tests must run in a dedicated, sandboxed environment that is torn down after each run. ## Expected Behaviour The `e2e_tests` CI job should: 1. Spin up a dedicated, ephemeral container or sandbox (e.g., a Docker-in-Docker service, a restricted network namespace, or a dedicated runner label) for each run. 2. Mount only the minimum required secrets and environment variables needed by the E2E suite — no CI tokens, registry credentials, or unrelated secrets should be visible inside the sandbox. 3. Enforce a strict `timeout-minutes` on the job to prevent runaway tests from blocking shared runners. 4. Tear down the sandbox completely after the job completes (pass or fail), leaving no artefacts on the host runner. 5. Report test results (Robot Framework XML/HTML output) as CI artefacts for post-run inspection. ## Acceptance Criteria - [ ] The `e2e_tests` job in `.forgejo/workflows/ci.yml` runs inside an isolated container or restricted environment (not directly on the shared runner host) - [ ] Only the minimum required environment variables and secrets are injected into the E2E sandbox - [ ] The job has an explicit `timeout-minutes` value - [ ] The sandbox is fully torn down after the job completes - [ ] Robot Framework XML/HTML output is uploaded as a CI artefact - [ ] `features/ci_workflow_validation.feature` is updated to assert the isolation and artefact-upload steps are present - [ ] All nox stages pass after the changes - [ ] Coverage ≥ 97% is maintained ## Subtasks - [ ] Audit the current `e2e_tests` job in `.forgejo/workflows/ci.yml` to identify all secrets and environment variables it consumes - [ ] Design the isolation strategy (Docker-in-Docker service container, restricted runner label, or network namespace) and document the decision in a comment in the workflow file - [ ] Update the `e2e_tests` job to run inside the chosen isolated environment - [ ] Restrict secret/env injection to only what the E2E suite requires; remove all unneeded variables from the job scope - [ ] Add or verify an explicit `timeout-minutes` on the `e2e_tests` job - [ ] Add an `upload-artifact` step to publish Robot Framework XML/HTML output after the job completes (even on failure) - [ ] Update `features/ci_workflow_validation.feature` with Behave scenarios asserting isolation, secret scoping, timeout, and artefact upload are present - [ ] Run `nox` (all default sessions) locally and fix any errors - [ ] Verify coverage ≥ 97% via `nox -s coverage_report` ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly (`feat(ci): run E2E tests in a secure, isolated environment`), followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly (`task/e2e-tests-secure-isolated-environment`). - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage ≥ 97% --- **Automated by CleverAgents Bot** Supervisor: Unknown | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-03 00:24:32 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • MoSCoW: MoSCoW/Could Have — CI/test infrastructure improvement.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: MoSCoW/Could Have — CI/test infrastructure improvement. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
freemo removed this from the v3.8.0 milestone 2026-04-07 01:31:17 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#1962
No description provided.