Update the nightly workflow quality gates script invocation to use
--coverage-min 97 instead of --coverage-min 85, aligning it with the
project's required COVERAGE_THRESHOLD = 97 defined in noxfile.py.
Also update the consolidated_config.feature BDD scenario to assert
the nightly workflow uses at least 97% (not 85%) to properly enforce
the coverage threshold requirement.
ISSUES CLOSED: #2769
- Replaced the e2e_tests nox session runner from sequential robot to the parallel pabot runner to shrink CI push times.
- Added _split_pabot_args and _pabot_parallel_args support to the e2e_tests session, mirroring the pattern used by integration_tests for consistent parallel execution control.
- CI workflow: added TEST_PROCESSES: "4" environment variable to the e2e_tests job to run four parallel workers. Rationale: E2E suites are IO-bound (LLM API calls) rather than CPU-bound, so increasing parallelism reduces wall-clock time without CPU contention.
- Updated the e2e_tests docstring to document parallelism control via the TEST_PROCESSES environment variable or the --processes positional argument.
- Pre-compiled bytecode comment updated to explain thundering-herd prevention for parallel workers.
- Template DB comment updated to explain the criticality of proper initialization for parallel pabot execution.
Key design decisions:
- Used pabot (robotframework-pabot>=4.0.0), a project-provided dependency, instead of introducing new tooling.
- Set TEST_PROCESSES=4 in CI (instead of the default min(cpu,2)) because E2E tasks are IO-bound and benefit from higher concurrency without CPU contention.
- Suite-level isolation via E2E Suite Setup (separate CLEVERAGENTS_HOME per suite) ensures workers do not share database state, enabling safe parallelism.
- Followed the exact same pattern as integration_tests for consistency and predictability.
Modules/components affected:
- e2e_tests nox session (parallelization logic and arg parsing)
- CI workflow (TEST_PROCESSES environment variable)
- E2E session docstring and related comments (documentation of parallelism and initialization)
- Inline comments for pre-compiled bytecode and template database initialization to reflect parallel execution considerations
ISSUES CLOSED: #1860
All 8 nox-running CI jobs in .forgejo/workflows/ci.yml now capture
stdout+stderr to build/nox-<job>-output.log via `2>&1 | tee` and upload
the log as a named Forgejo artifact (if: always(), retention-days: 30).
Artifact names follow the pattern ci-logs-<job>:
ci-logs-lint, ci-logs-typecheck, ci-logs-security, ci-logs-quality,
ci-logs-unit-tests, ci-logs-integration-tests, ci-logs-e2e-tests,
ci-logs-coverage
Seven agent definitions updated with a CI Log Artifacts section:
ca-pr-checker.md: artifact table + curl download instructions; Step 2
now downloads the relevant artifact before dispatching fix subagents.
ca-lint-fixer.md, ca-typecheck-fixer.md, ca-unit-test-runner.md,
ca-integration-test-runner.md, ca-coverage-checker.md,
ca-pr-self-reviewer.md: each receives a section explaining which
artifact corresponds to its domain and how to use it.
Design notes:
- tee (not redirect) preserves output in CI job logs AND captures to file
- if: always() ensures artifacts are available even when the job fails
- Multi-session jobs (lint, security) use tee -a to append to one file
- Existing coverage-reports artifact preserved alongside ci-logs-coverage
ISSUES CLOSED: #2750
- Add missing 'nox -s format -- --check' step (parity with ci.yml)
- Add NOX_DEFAULT_VENV_BACKEND: uv to global env and each nox step
- Fix quality gates script invocation (was using invalid nox session '3.13')
- Fix report artifact paths to match actual nox session output locations
(build/coverage.json and build/bandit-report.json instead of build/reports/)
- Add mkdir for build/reports before writing quality-trend.json
ISSUES CLOSED: #1537
Root cause: actions/checkout@v4 was not configured with explicit write
credentials (token + persist-credentials), and no git user identity
(user.name/user.email) was set. Both are required for any git push
operation in Forgejo Actions.
Changes:
- release.yml create-release job: add token: secrets.FORGEJO_TOKEN and
fetch-depth: 0 to checkout; add 'Configure git identity for push
operations' step using HTTPS credential store; add 'Smoke-test push
access' step that validates write permission via Forgejo API before
any push attempt
- ci.yml: add push-validation job that validates push credentials on
every CI run using FORGEJO_TOKEN, including credential helper
verification and API-based write permission check; add push-validation
to status-check needs and result reporting
- docs/development/ci-cd.md: add FORGEJO_TOKEN, FORGEJO_URL, and
CONTAINER_REGISTRY* secrets to the secrets table; add 'Repository
Push Authentication' section documenting root cause, fix pattern,
smoke-test step, setup instructions, and security notes; add
push-validation to CI job dependency graph and quality gates table
Design decisions:
- HTTPS token authentication (not SSH deploy keys) -- simpler to manage
- ~/.git-credentials with chmod 600 for ephemeral, secure storage
- Smoke-test validates write permission via API before push attempts
- push-validation job is independent (no needs) -- runs in parallel
- No hardcoded credentials -- all secrets via Forgejo Secrets
ISSUES CLOSED: #1541
Replace per-job cache key prefixes (uv-lint-, uv-typecheck-, uv-security-,
uv-quality-, uv-tests-, uv-coverage-) with a single shared key
uv-${{ hashFiles('pyproject.toml') }} across all jobs in ci.yml.
Also add a missing uv cache step to the build job and add uv caching
to nightly-quality.yml, which previously had no cache at all.
All jobs now share the same cache entry when pyproject.toml is unchanged,
eliminating redundant package downloads and reducing CI wall-clock time.
ISSUES CLOSED: #1535
The lint, typecheck, security, and quality static analysis jobs already
run in parallel (no needs dependencies between them). However, downstream
jobs only gated on a subset of these jobs, allowing the pipeline to
proceed even when some static analysis checks failed.
Updated needs dependencies for downstream jobs:
- coverage: [lint, typecheck] → [lint, typecheck, security, quality]
- benchmark-regression: [lint, typecheck] → [lint, typecheck, security, quality]
- docker: [lint, typecheck, unit_tests, security] → [lint, typecheck, security, quality, unit_tests]
All four static analysis jobs (lint, typecheck, security, quality) now
run in parallel and all downstream jobs correctly gate on the full set,
ensuring no job proceeds unless all static analysis checks pass.
ISSUES CLOSED: #1536
Add kubeconform manifest validation to the existing helm CI job.
The helm job already had helm lint and helm template steps (added in
#1085). This commit completes the optional acceptance criterion from
#1089 by adding kubeconform v0.7.0 to validate rendered manifests
against the Kubernetes 1.29.0 schema in strict mode.
Changes:
- Install kubeconform v0.7.0 in the helm CI job
- Add 'Validate rendered manifests with kubeconform' step after
helm template smoke render (strict mode, ignore-missing-schemas,
kubernetes-version 1.29.0)
- Add 5 BDD scenarios to ci_workflow_validation.feature covering:
- helm job existence
- helm lint step
- helm template step
- kubeconform validation step
- status-check dependency on helm job
ISSUES CLOSED: #1089
## Summary
This PR adds a Behave TDD bug-capture test that proves bug #1039 exists: the specification references `validations/unit-tests.yaml` as an example validation configuration file (Workflow Examples, Example 1, Step 1) but the file does not exist in the project's `examples/validations/` directory.
## Changes
- **New feature file:** `features/tdd_missing_validation_unit_tests_yaml.feature` — four scenarios tagged `@tdd_expected_fail @tdd_bug @tdd_bug_1039` that verify:
1. The file `examples/validations/unit-tests.yaml` exists
2. The YAML config contains `name: local/unit-tests`
3. The validation mode is `required`
4. The description field is non-empty
- **New step file:** `features/steps/tdd_missing_validation_unit_tests_yaml_steps.py` — step implementations that locate the project root and check file existence/content
All four scenarios currently fail with `AssertionError` (the file doesn't exist, proving the bug exists). The `@tdd_expected_fail` tag inverts the results so the test suite passes CI.
## Robot Integration Test
N/A — this bug is purely about a missing example file referenced by the specification. There is no integration-level behavior to test.
## Quality Gate Results
| Gate | Result |
|------|--------|
| `nox -s lint` | passed |
| `nox -s typecheck` | passed (0 errors) |
| `nox -s unit_tests` | 462 features, 12234 scenarios, 0 failures |
| `nox -s integration_tests` | 1672 tests, 0 failures |
| `nox -s e2e_tests` | 37 tests, 0 failures |
| `nox -s coverage_report` | 98% (>= 97% threshold) |
Closes#1103
Reviewed-on: cleveragents/cleveragents-core#1132
Reviewed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com>
Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
## Summary
This PR adds Kubernetes Helm deployment support for the CleverAgents server.
Closes#928
### What this PR includes
- Helm chart under `k8s/` with Deployment, Service, optional Ingress, ConfigMap,
ServiceAccount, Secrets, NOTES, and optional Redis subchart configuration.
- Multi-stage `Dockerfile.server` for server runtime deployment.
- Deployment-focused docs in `k8s/README.md`.
- Behave + Robot + benchmark coverage for chart/deployment wiring.
### Review fixes applied (cycle 11 — hurui200320 review #2687)
**Critical fix:**
1. **CI SHA256 checksum verification fixed** — All 3 Helm install blocks (`unit_tests`, `integration_tests`, `helm` jobs) now save the tarball using its original filename (`helm-v3.16.4-linux-amd64.tar.gz`) instead of `helm.tgz`, so the `.sha256sum` file can correctly locate and verify it. This was causing all 3 CI Helm jobs to fail with "No such file or directory".
**Major fixes (test coverage gaps):**
2. **405 Allow header now tested** — The existing "POST to known path returns method not allowed" scenario now asserts that the `Allow: GET` header is present. Added a second 405 scenario testing `POST /live` for broader `_KNOWN_PATHS` coverage (review item #11).
3. **Security-hardening headers now tested** — New scenario "HTTP responses include security-hardening headers" verifies `content-length`, `x-content-type-options: nosniff`, and `cache-control: no-store` headers are present on HTTP responses.
4. **Lifespan warning logging now tested** — New scenario "Unrecognised lifespan message type logs warning and continues" queues `lifespan.startup` → `lifespan.bogus` → `lifespan.shutdown` and verifies: (a) the app completes the lifespan cycle cleanly, (b) a warning is logged mentioning the unrecognised type.
### Review fixes applied (cycle 10)
**Critical/Major fixes (from hurui200320's prior REQUEST_CHANGES):**
1. **Rebased branch onto master** — Removed merge commit per CONTRIBUTING.md rebase-only policy. Clean linear history restored.
2. **Fixed commit message body** — Replaced literal `\n` sequences with actual newlines. `ISSUES CLOSED: #928` footer is now on its own line after a blank separator.
3. **Moved uvicorn import to module top level** — `from uvicorn import run as uvicorn_run` is now at the top of `src/cleveragents/cli/commands/server.py` per Import Guidelines. Updated test mock target from `uvicorn.run` to `cleveragents.cli.commands.server.uvicorn_run`.
4. **Added SHA256 checksum verification** — All three Helm CLI install blocks in `.forgejo/workflows/ci.yml` now download and verify `helm.sha256sum` before extracting the binary.
**Minor fixes:**
5. **Added `Dockerfile.server` build to CI** — New "Build Docker image (Server)" step in the `docker` job validates the server Dockerfile.
6. **ASGI 405 Method Not Allowed** — Known paths (`/`, `/live`, `/ready`, `/health`) now return 405 with `Allow: GET` header for non-GET methods, per RFC 9110 §15.5.6. Added `_KNOWN_PATHS` frozenset.
7. **WebSocket close protocol fix** — App now calls `await receive()` to consume the `websocket.connect` event before closing. Changed close code from 1000 (Normal Closure) to 1008 (Policy Violation).
8. **Lifespan handler logging** — Unrecognised lifespan message types are now logged as warnings instead of silently consumed.
9. **Security-hardening headers** — `_send_response` now includes `content-length`, `x-content-type-options: nosniff`, and `cache-control: no-store` on all HTTP responses.
10. **`.dockerignore` credential patterns** — Added `*.pem`, `*.key`, `*.p12`, `*.pfx`, `credentials*.json`.
11. **`--log-level` validation** — Constrained to `click.Choice(["critical", "error", "warning", "info", "debug", "trace"])` for clean CLI validation errors.
12. **Reverted unrelated semgrep pre-commit change** — `pass_filenames` and `entry` restored to original values per atomic commit hygiene.
13. **Removed unused `ReceiveCallable` type alias** and `Callable`/`Awaitable` imports from `asgi_app_steps.py`.
14. **Fixed redundant `shutil.which("helm")` check** — `_skip_if_helm_missing` now returns `bool` to eliminate the duplicate check in `_render_chart`.
15. **Improved test deque error handling** — Lifespan test receive mock now raises descriptive `AssertionError` instead of opaque `IndexError`.
16. **Scope type dispatch** — Changed `if/if/if` to `if/elif/elif` for mutually exclusive ASGI scope types.
17. **Dockerfile.server base image** — Standardised to `python:3.13-slim` (floating minor) consistent with CLI Dockerfile.
18. **Dockerfile layer caching** — Split `uv pip install build` and `python -m build` into separate `RUN` instructions.
19. **Removed extraneous double blank line** in Dockerfile.server.
### Deferred items (acknowledged, not in scope)
- PodDisruptionBudget, HorizontalPodAutoscaler, NetworkPolicy — Follow-up for production hardening.
- `appVersion: "1.0.0"` placeholder — Needs tracking issue for release versioning alignment.
- Readiness probe with downstream dependency checks — Documented limitation.
- Cross-system test for probe paths matching ASGI routes — Test enhancement.
- Improved benchmarks (helm template timing vs PyYAML parsing) — Benchmark quality improvement.
- CI DRY violation (Helm install 3×) — Code quality improvement, consider composite action.
- File length limits exceeded (`k8s_helm_chart_steps.py` 551 lines, `helper_k8s_helm_chart.py` 678 lines) — Non-blocking, can be split in follow-up.
- `runAsGroup: 1000` in pod security context — Defense-in-depth improvement.
- HEAD method support on known paths — RFC compliance, does not affect K8s probes.
- `click.Choice` log-level validation via CLI runner test — Test gap.
### Scope note: status-check CI gate
The `status-check` job now includes `integration_tests`, `e2e_tests`, and `helm` in its `needs` list. The `helm` job is new in this PR. The `integration_tests` and `e2e_tests` additions fix previously-missing gate checks — included here since this PR modifies both of those jobs to install Helm.
### Quality gates
- `nox -e lint` ✅
- `nox -e typecheck` ✅
- `nox -e unit_tests` ✅ (12,321 scenarios passed, 4 skipped)
- `nox -e integration_tests` — 3 pre-existing failures in unrelated areas (plan correction, resource types)
- `nox -e e2e_tests` — pre-existing failures (LLM API keys not available in local env)
- `nox -e coverage_report` ✅ (**97.7%**)
Reviewed-on: cleveragents/cleveragents-core#1085
Reviewed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com>
Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Updated CI pipeline to inject ANTHROPIC_API_KEY and OPENAI_API_KEY
secrets as environment variables during Robot Framework integration
test execution. Added CI setup documentation.
ISSUES CLOSED: #701
Added dedicated E2E test infrastructure completely separate from the existing
integration test suite. E2E tests use zero mocking — they exercise the real
CleverAgents CLI with real LLM API keys.
Key changes:
- New e2e_tests nox session running Robot Framework with --include E2E tag
filter against robot/e2e/ directory. Uses sequential robot (not pabot)
since E2E tests hit real API endpoints with rate limits. Propagates
ANTHROPIC_API_KEY, OPENAI_API_KEY, and GOOGLE_API_KEY from environment.
Output goes to build/reports/robot-e2e/ to avoid artifact collisions.
- Existing integration_tests session now excludes E2E-tagged tests via
--exclude E2E on the pabot invocation.
- New robot/e2e/common_e2e.resource provides shared E2E keywords: suite
setup/teardown with per-suite isolation (no mock AI), graceful skip when
LLM API keys are absent, CLI runner keyword, flexible output assertions,
and temporary git repo fixture creation.
- Minimal smoke test (robot/e2e/smoke_test.robot) validates the harness by
running agents --version and agents --help. Does not require LLM keys.
- Dedicated e2e_tests CI job in .forgejo/workflows/ci.yml injects LLM API
keys from Forgejo secrets. Runs independently (no needs dependencies)
and does not block regular CI.
- e2e_tests is deliberately NOT in the default nox sessions list since it
requires real API keys not present in all environments.
ISSUES CLOSED: #740
Replace coverage.py (sys.settrace-based) with slipcover (bytecode-based
instrumentation) for significantly faster coverage collection:
- Each behave-parallel worker runs under slipcover, producing per-feature
JSON coverage files with unique UUIDs to avoid write contention.
- After all workers finish, slipcover --merge combines per-worker data
into a single build/coverage.json report.
- XML report generated via slipcover --merge --xml for CI tooling.
- Terminal report with --fail-under=97 threshold enforcement.
- Robust JSON key-fallback logic handles both slipcover and coverage.py
output formats.
- CI workflow (ci.yml, nightly-quality.yml) updated with defensive key
lookup instead of hardcoded coverage.py format.
- Documentation updated to reflect slipcover as the coverage tool.
- CHANGELOG.md updated.
ISSUES CLOSED: #482
freemo
brent.edwards
CoreRasurae
mngrif
