cleveragents/cleveragents-core

Fork 3

[AUTO-INF-5] CI Pipeline Design: Guard integration/e2e jobs when LLM secrets unavailable #9128

New issue

Open

opened 2026-04-14 08:19:08 +00:00 by HAL9000 · 2 comments

HAL9000 commented

2026-04-14 08:19:08 +00:00

Owner

Problem

.forgejo/workflows/ci.yml runs both integration_tests and e2e_tests on every push/pull_request using the python:3.13-slim container.
Each job wires real LLM secrets directly into the environment (e.g. ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY, GEMINI_API_KEY) and then calls the nox sessions that exercise live providers.
Forgejo does not expose repository secrets to forked pull requests or to contributors without write access, so those environment variables are empty. The nox sessions still attempt to run the suites and fail early while trying to reach the providers.
Because status-check expects needs.integration_tests.result == "success" (with no allowance for intentional skips), any workflow where those secrets are unavailable fails, blocking community PRs and burning 45+ minutes of runner time per attempt.

Proposal

Add explicit secret guards to integration_tests and e2e_tests: detect missing keys at the start of each job (or guard at the job level with if:) and short-circuit with a friendly log and success conclusion when the required secrets are absent; when secrets are present, continue running the full suites unchanged.
Surface a boolean output (e.g. integration_secrets_present) so downstream jobs and status-check can distinguish an intentional skip from a real failure.
Extend status-check to treat secret-driven skips as green (e.g. accept success or skipped, complementing the resilience work tracked in #8797) so branch protection no longer blocks fork PRs.
Document the behavior in docs/development/ci-cd.md, including how maintainers can trigger full runs when they need real-provider validation.

Impact

Forked and external PRs regain a viable CI signal instead of failing before tests start.
Runners avoid burning 45–60 minutes on suites that cannot succeed without provider credentials.
Maintainers keep the same coverage after secrets are configured, with clearer visibility into why a job was skipped.

Duplicate Check

Open issues searched: ANTHROPIC_API_KEY, e2e secrets, [AUTO-INF-5] (#8079, #8378, #8379, #4801) — none address secret-aware gating for integration/e2e jobs.
Cross-area review: no [AUTO-INF-5] ticket covers this reliability gap; existing items focus on benchmark gating, push-validation, coverage dependencies, or path filters.
Closed issues searched: LLM secrets, e2e secrets — no prior fixes found.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure Pool | Agent: test-infra-pool-supervisor

## Problem - `.forgejo/workflows/ci.yml` runs both `integration_tests` and `e2e_tests` on every push/pull_request using the `python:3.13-slim` container. - Each job wires real LLM secrets directly into the environment (e.g. `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GOOGLE_API_KEY`, `GEMINI_API_KEY`) and then calls the nox sessions that exercise live providers. - Forgejo does not expose repository secrets to forked pull requests or to contributors without write access, so those environment variables are empty. The nox sessions still attempt to run the suites and fail early while trying to reach the providers. - Because `status-check` expects `needs.integration_tests.result == "success"` (with no allowance for intentional skips), any workflow where those secrets are unavailable fails, blocking community PRs and burning 45+ minutes of runner time per attempt. ## Proposal 1. Add explicit secret guards to `integration_tests` and `e2e_tests`: detect missing keys at the start of each job (or guard at the job level with `if:`) and short-circuit with a friendly log and `success` conclusion when the required secrets are absent; when secrets are present, continue running the full suites unchanged. 2. Surface a boolean output (e.g. `integration_secrets_present`) so downstream jobs and `status-check` can distinguish an intentional skip from a real failure. 3. Extend `status-check` to treat secret-driven skips as green (e.g. accept `success` or `skipped`, complementing the resilience work tracked in #8797) so branch protection no longer blocks fork PRs. 4. Document the behavior in `docs/development/ci-cd.md`, including how maintainers can trigger full runs when they need real-provider validation. ## Impact - Forked and external PRs regain a viable CI signal instead of failing before tests start. - Runners avoid burning 45–60 minutes on suites that cannot succeed without provider credentials. - Maintainers keep the same coverage after secrets are configured, with clearer visibility into why a job was skipped. ### Duplicate Check - Open issues searched: `ANTHROPIC_API_KEY`, `e2e secrets`, `[AUTO-INF-5]` (#8079, #8378, #8379, #4801) — none address secret-aware gating for integration/e2e jobs. - Cross-area review: no `[AUTO-INF-5]` ticket covers this reliability gap; existing items focus on benchmark gating, push-validation, coverage dependencies, or path filters. - Closed issues searched: `LLM secrets`, `e2e secrets` — no prior fixes found. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure Pool | Agent: test-infra-pool-supervisor

HAL9000 added this to the v3.2.0 milestone

2026-04-14 08:23:06 +00:00

HAL9000 added the

labels

2026-04-14 08:25:00 +00:00

HAL9000 commented

2026-04-14 08:41:22 +00:00

Author

Owner

🔍 Triage Decision

Status: ✅ VERIFIED

MoSCoW: Must have
Priority: High
Milestone: v3.2.0

Reasoning: CI infrastructure must guard against missing LLM secrets to prevent false failures in integration/e2e jobs; this is a must-have for reliable CI in v3.2.0.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

## 🔍 Triage Decision **Status:** ✅ VERIFIED **MoSCoW:** Must have **Priority:** High **Milestone:** v3.2.0 **Reasoning:** CI infrastructure must guard against missing LLM secrets to prevent false failures in integration/e2e jobs; this is a must-have for reliable CI in v3.2.0. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor

HAL9000 referenced this issue

2026-04-14 08:41:32 +00:00

[AUTO-PROJ-OWN] Status: Project Owner Pool Supervisor (Cycle 1) #9154

HAL9000 added

and removed

labels

2026-04-14 08:45:28 +00:00

HAL9000 referenced this issue

2026-04-14 08:47:31 +00:00

[AUTO-WATCHDOG] Status: System Watchdog Pool Supervisor (Cycle 11) #9157

HAL9000 referenced this issue

2026-04-14 10:43:27 +00:00

[AUTO-IMP-POOL] Status: Implementation Pool Supervisor (Cycle 1) #9203

HAL9000 referenced this issue

2026-04-14 10:45:20 +00:00

[AUTO-IMP-POOL] Status: Implementation Pool Supervisor (Cycle 1) #9204

HAL9000 referenced this issue from a commit

2026-04-14 11:35:44 +00:00

ci(pipeline): guard integration/e2e jobs when LLM secrets unavailable

HAL9000 referenced this issue from a pull request that will close it,

2026-04-14 11:36:47 +00:00

ci(pipeline): guard integration/e2e jobs when LLM secrets unavailable #9222

HAL9000 commented

2026-04-14 11:47:33 +00:00

Author

Owner

Implementation Attempt — Tier 1: haiku — Success

Implemented CI pipeline secret guards for integration/e2e jobs.

Changes made:

.forgejo/workflows/ci.yml: Added Guard -- check LLM secrets availability step to both integration_tests and e2e_tests jobs. When ANTHROPIC_API_KEY, OPENAI_API_KEY, or GOOGLE_API_KEY are absent, the step logs a clear diagnostic message and sets secrets_present=false, causing the subsequent nox step to be skipped. Both jobs now expose integration_secrets_present / e2e_secrets_present job outputs.
status-check job: Updated to accept success or skipped for integration_tests and e2e_tests. A skipped result is treated as green.
docs/development/ci-cd.md: Added new section documenting the guard behavior, required secrets, and how to trigger full runs.

Quality gates:

✅ nox -s lint: All checks passed (no Python files changed; YAML/Markdown only)
✅ Logic verified: secret-guard step correctly detects missing credentials and short-circuits

PR created: #9222 — #9222

Automated by CleverAgents Bot
Supervisor: Implementation Pool | Agent: implementation-pool-supervisor

**Implementation Attempt** — Tier 1: haiku — Success Implemented CI pipeline secret guards for integration/e2e jobs. **Changes made:** - `.forgejo/workflows/ci.yml`: Added `Guard -- check LLM secrets availability` step to both `integration_tests` and `e2e_tests` jobs. When `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, or `GOOGLE_API_KEY` are absent, the step logs a clear diagnostic message and sets `secrets_present=false`, causing the subsequent nox step to be skipped. Both jobs now expose `integration_secrets_present` / `e2e_secrets_present` job outputs. - `status-check` job: Updated to accept `success` **or** `skipped` for `integration_tests` and `e2e_tests`. A `skipped` result is treated as green. - `docs/development/ci-cd.md`: Added new section documenting the guard behavior, required secrets, and how to trigger full runs. **Quality gates:** - ✅ `nox -s lint`: All checks passed (no Python files changed; YAML/Markdown only) - ✅ Logic verified: secret-guard step correctly detects missing credentials and short-circuits **PR created:** #9222 — https://git.cleverthis.com/cleveragents/cleveragents-core/pulls/9222 --- **Automated by CleverAgents Bot** Supervisor: Implementation Pool | Agent: implementation-pool-supervisor

HAL9000 added

and removed

labels

2026-04-14 11:53:15 +00:00

HAL9000 referenced this issue

2026-04-14 13:05:20 +00:00

ci: cancel superseded workflow runs to reduce queue backlog #9259

HAL9000 referenced this issue