TEST-INFRA: [ci-execution-time] Implement a strategy for running E2E tests #2235

New issue

Open

opened 2026-04-03 09:44:06 +00:00 by freemo · 0 comments

freemo commented

2026-04-03 09:44:06 +00:00

Owner

Metadata

Branch: chore/ci-execution-time-e2e-test-strategy
Commit Message: chore(ci): implement secure CI strategy for running E2E tests with real LLM API keys
Milestone: v3.8.0
Parent Epic: #1678

Background and Context

The e2e_tests nox session is not run by default because it requires real LLM API keys. Without a dedicated CI strategy, end-to-end functionality of the application is never automatically verified against live LLM providers. This creates a blind spot where integration regressions between the application and real LLM APIs can go undetected until they surface in production or manual testing.

Expected Behavior

A secure, restricted CI environment exists with access to the necessary LLM API keys. A scheduled CI job runs the e2e_tests nox session regularly (e.g., nightly or weekly), and results are reported to the development team.

Acceptance Criteria

A new, restricted CI environment is created with access to the necessary API keys.
A new CI job is created to run the e2e_tests nox session in the restricted environment.
The job is scheduled to run on a regular basis (e.g., nightly or weekly).
The results of the E2E tests are reported to the development team.

Supporting Information

Related Epic: #1678 — CI Execution Time Optimization
The e2e_tests nox session requires real LLM API keys and is excluded from the default CI loop.
API keys must be stored as CI secrets and never exposed in logs or artifacts.

Subtasks

Create a new, restricted CI environment with access to the necessary API keys
Create a new CI job that runs the e2e_tests nox session in the restricted environment
Schedule this job to run on a regular basis (e.g., nightly or weekly)
Ensure that the results of the E2E tests are reported to the development team
Tests (Behave): Verify any CI configuration changes are covered by scenario tests
Tests (Robot): Confirm E2E test execution is validated end-to-end
Verify coverage >= 97% via nox -s coverage_report
Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A new, restricted CI environment is created.
A new CI job is created to run the e2e_tests nox session.
The job is scheduled to run on a regular basis.
The test results are reported to the development team.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
All nox stages pass.
Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `chore/ci-execution-time-e2e-test-strategy` - **Commit Message**: `chore(ci): implement secure CI strategy for running E2E tests with real LLM API keys` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The `e2e_tests` nox session is not run by default because it requires real LLM API keys. Without a dedicated CI strategy, end-to-end functionality of the application is never automatically verified against live LLM providers. This creates a blind spot where integration regressions between the application and real LLM APIs can go undetected until they surface in production or manual testing. ## Expected Behavior A secure, restricted CI environment exists with access to the necessary LLM API keys. A scheduled CI job runs the `e2e_tests` nox session regularly (e.g., nightly or weekly), and results are reported to the development team. ## Acceptance Criteria - A new, restricted CI environment is created with access to the necessary API keys. - A new CI job is created to run the `e2e_tests` nox session in the restricted environment. - The job is scheduled to run on a regular basis (e.g., nightly or weekly). - The results of the E2E tests are reported to the development team. ## Supporting Information - Related Epic: #1678 — CI Execution Time Optimization - The `e2e_tests` nox session requires real LLM API keys and is excluded from the default CI loop. - API keys must be stored as CI secrets and never exposed in logs or artifacts. ## Subtasks - [ ] Create a new, restricted CI environment with access to the necessary API keys - [ ] Create a new CI job that runs the `e2e_tests` nox session in the restricted environment - [ ] Schedule this job to run on a regular basis (e.g., nightly or weekly) - [ ] Ensure that the results of the E2E tests are reported to the development team - [ ] Tests (Behave): Verify any CI configuration changes are covered by scenario tests - [ ] Tests (Robot): Confirm E2E test execution is validated end-to-end - [ ] Verify coverage >= 97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A new, restricted CI environment is created. - A new CI job is created to run the `e2e_tests` nox session. - The job is scheduled to run on a regular basis. - The test results are reported to the development team. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

freemo added this to the v3.8.0 milestone

2026-04-03 09:44:11 +00:00

freemo added a new dependency

2026-04-03 10:04:34 +00:00

#1678 Epic: CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing

freemo removed this from the v3.8.0 milestone

2026-04-07 01:13:28 +00:00