All 8 nox-running CI jobs in .forgejo/workflows/ci.yml now capture
stdout+stderr to build/nox-<job>-output.log via `2>&1 | tee` and upload
the log as a named Forgejo artifact (if: always(), retention-days: 30).
Artifact names follow the pattern ci-logs-<job>:
ci-logs-lint, ci-logs-typecheck, ci-logs-security, ci-logs-quality,
ci-logs-unit-tests, ci-logs-integration-tests, ci-logs-e2e-tests,
ci-logs-coverage
Seven agent definitions updated with a CI Log Artifacts section:
ca-pr-checker.md: artifact table + curl download instructions; Step 2
now downloads the relevant artifact before dispatching fix subagents.
ca-lint-fixer.md, ca-typecheck-fixer.md, ca-unit-test-runner.md,
ca-integration-test-runner.md, ca-coverage-checker.md,
ca-pr-self-reviewer.md: each receives a section explaining which
artifact corresponds to its domain and how to use it.
Design notes:
- tee (not redirect) preserves output in CI job logs AND captures to file
- if: always() ensures artifacts are available even when the job fails
- Multi-session jobs (lint, security) use tee -a to append to one file
- Existing coverage-reports artifact preserved alongside ci-logs-coverage
ISSUES CLOSED: #2750
- Add missing 'nox -s format -- --check' step (parity with ci.yml)
- Add NOX_DEFAULT_VENV_BACKEND: uv to global env and each nox step
- Fix quality gates script invocation (was using invalid nox session '3.13')
- Fix report artifact paths to match actual nox session output locations
(build/coverage.json and build/bandit-report.json instead of build/reports/)
- Add mkdir for build/reports before writing quality-trend.json
ISSUES CLOSED: #1537
Replace per-job cache key prefixes (uv-lint-, uv-typecheck-, uv-security-,
uv-quality-, uv-tests-, uv-coverage-) with a single shared key
uv-${{ hashFiles('pyproject.toml') }} across all jobs in ci.yml.
Also add a missing uv cache step to the build job and add uv caching
to nightly-quality.yml, which previously had no cache at all.
All jobs now share the same cache entry when pyproject.toml is unchanged,
eliminating redundant package downloads and reducing CI wall-clock time.
ISSUES CLOSED: #1535
The lint, typecheck, security, and quality static analysis jobs already
run in parallel (no needs dependencies between them). However, downstream
jobs only gated on a subset of these jobs, allowing the pipeline to
proceed even when some static analysis checks failed.
Updated needs dependencies for downstream jobs:
- coverage: [lint, typecheck] → [lint, typecheck, security, quality]
- benchmark-regression: [lint, typecheck] → [lint, typecheck, security, quality]
- docker: [lint, typecheck, unit_tests, security] → [lint, typecheck, security, quality, unit_tests]
All four static analysis jobs (lint, typecheck, security, quality) now
run in parallel and all downstream jobs correctly gate on the full set,
ensuring no job proceeds unless all static analysis checks pass.
ISSUES CLOSED: #1536
Add kubeconform manifest validation to the existing helm CI job.
The helm job already had helm lint and helm template steps (added in
#1085). This commit completes the optional acceptance criterion from
#1089 by adding kubeconform v0.7.0 to validate rendered manifests
against the Kubernetes 1.29.0 schema in strict mode.
Changes:
- Install kubeconform v0.7.0 in the helm CI job
- Add 'Validate rendered manifests with kubeconform' step after
helm template smoke render (strict mode, ignore-missing-schemas,
kubernetes-version 1.29.0)
- Add 5 BDD scenarios to ci_workflow_validation.feature covering:
- helm job existence
- helm lint step
- helm template step
- kubeconform validation step
- status-check dependency on helm job
ISSUES CLOSED: #1089
## Summary
This PR adds a Behave TDD bug-capture test that proves bug #1039 exists: the specification references `validations/unit-tests.yaml` as an example validation configuration file (Workflow Examples, Example 1, Step 1) but the file does not exist in the project's `examples/validations/` directory.
## Changes
- **New feature file:** `features/tdd_missing_validation_unit_tests_yaml.feature` — four scenarios tagged `@tdd_expected_fail @tdd_bug @tdd_bug_1039` that verify:
1. The file `examples/validations/unit-tests.yaml` exists
2. The YAML config contains `name: local/unit-tests`
3. The validation mode is `required`
4. The description field is non-empty
- **New step file:** `features/steps/tdd_missing_validation_unit_tests_yaml_steps.py` — step implementations that locate the project root and check file existence/content
All four scenarios currently fail with `AssertionError` (the file doesn't exist, proving the bug exists). The `@tdd_expected_fail` tag inverts the results so the test suite passes CI.
## Robot Integration Test
N/A — this bug is purely about a missing example file referenced by the specification. There is no integration-level behavior to test.
## Quality Gate Results
| Gate | Result |
|------|--------|
| `nox -s lint` | passed |
| `nox -s typecheck` | passed (0 errors) |
| `nox -s unit_tests` | 462 features, 12234 scenarios, 0 failures |
| `nox -s integration_tests` | 1672 tests, 0 failures |
| `nox -s e2e_tests` | 37 tests, 0 failures |
| `nox -s coverage_report` | 98% (>= 97% threshold) |
Closes#1103
Reviewed-on: cleveragents/cleveragents-core#1132
Reviewed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com>
Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
## Summary
This PR adds Kubernetes Helm deployment support for the CleverAgents server.
Closes#928
### What this PR includes
- Helm chart under `k8s/` with Deployment, Service, optional Ingress, ConfigMap,
ServiceAccount, Secrets, NOTES, and optional Redis subchart configuration.
- Multi-stage `Dockerfile.server` for server runtime deployment.
- Deployment-focused docs in `k8s/README.md`.
- Behave + Robot + benchmark coverage for chart/deployment wiring.
### Review fixes applied (cycle 11 — hurui200320 review #2687)
**Critical fix:**
1. **CI SHA256 checksum verification fixed** — All 3 Helm install blocks (`unit_tests`, `integration_tests`, `helm` jobs) now save the tarball using its original filename (`helm-v3.16.4-linux-amd64.tar.gz`) instead of `helm.tgz`, so the `.sha256sum` file can correctly locate and verify it. This was causing all 3 CI Helm jobs to fail with "No such file or directory".
**Major fixes (test coverage gaps):**
2. **405 Allow header now tested** — The existing "POST to known path returns method not allowed" scenario now asserts that the `Allow: GET` header is present. Added a second 405 scenario testing `POST /live` for broader `_KNOWN_PATHS` coverage (review item #11).
3. **Security-hardening headers now tested** — New scenario "HTTP responses include security-hardening headers" verifies `content-length`, `x-content-type-options: nosniff`, and `cache-control: no-store` headers are present on HTTP responses.
4. **Lifespan warning logging now tested** — New scenario "Unrecognised lifespan message type logs warning and continues" queues `lifespan.startup` → `lifespan.bogus` → `lifespan.shutdown` and verifies: (a) the app completes the lifespan cycle cleanly, (b) a warning is logged mentioning the unrecognised type.
### Review fixes applied (cycle 10)
**Critical/Major fixes (from hurui200320's prior REQUEST_CHANGES):**
1. **Rebased branch onto master** — Removed merge commit per CONTRIBUTING.md rebase-only policy. Clean linear history restored.
2. **Fixed commit message body** — Replaced literal `\n` sequences with actual newlines. `ISSUES CLOSED: #928` footer is now on its own line after a blank separator.
3. **Moved uvicorn import to module top level** — `from uvicorn import run as uvicorn_run` is now at the top of `src/cleveragents/cli/commands/server.py` per Import Guidelines. Updated test mock target from `uvicorn.run` to `cleveragents.cli.commands.server.uvicorn_run`.
4. **Added SHA256 checksum verification** — All three Helm CLI install blocks in `.forgejo/workflows/ci.yml` now download and verify `helm.sha256sum` before extracting the binary.
**Minor fixes:**
5. **Added `Dockerfile.server` build to CI** — New "Build Docker image (Server)" step in the `docker` job validates the server Dockerfile.
6. **ASGI 405 Method Not Allowed** — Known paths (`/`, `/live`, `/ready`, `/health`) now return 405 with `Allow: GET` header for non-GET methods, per RFC 9110 §15.5.6. Added `_KNOWN_PATHS` frozenset.
7. **WebSocket close protocol fix** — App now calls `await receive()` to consume the `websocket.connect` event before closing. Changed close code from 1000 (Normal Closure) to 1008 (Policy Violation).
8. **Lifespan handler logging** — Unrecognised lifespan message types are now logged as warnings instead of silently consumed.
9. **Security-hardening headers** — `_send_response` now includes `content-length`, `x-content-type-options: nosniff`, and `cache-control: no-store` on all HTTP responses.
10. **`.dockerignore` credential patterns** — Added `*.pem`, `*.key`, `*.p12`, `*.pfx`, `credentials*.json`.
11. **`--log-level` validation** — Constrained to `click.Choice(["critical", "error", "warning", "info", "debug", "trace"])` for clean CLI validation errors.
12. **Reverted unrelated semgrep pre-commit change** — `pass_filenames` and `entry` restored to original values per atomic commit hygiene.
13. **Removed unused `ReceiveCallable` type alias** and `Callable`/`Awaitable` imports from `asgi_app_steps.py`.
14. **Fixed redundant `shutil.which("helm")` check** — `_skip_if_helm_missing` now returns `bool` to eliminate the duplicate check in `_render_chart`.
15. **Improved test deque error handling** — Lifespan test receive mock now raises descriptive `AssertionError` instead of opaque `IndexError`.
16. **Scope type dispatch** — Changed `if/if/if` to `if/elif/elif` for mutually exclusive ASGI scope types.
17. **Dockerfile.server base image** — Standardised to `python:3.13-slim` (floating minor) consistent with CLI Dockerfile.
18. **Dockerfile layer caching** — Split `uv pip install build` and `python -m build` into separate `RUN` instructions.
19. **Removed extraneous double blank line** in Dockerfile.server.
### Deferred items (acknowledged, not in scope)
- PodDisruptionBudget, HorizontalPodAutoscaler, NetworkPolicy — Follow-up for production hardening.
- `appVersion: "1.0.0"` placeholder — Needs tracking issue for release versioning alignment.
- Readiness probe with downstream dependency checks — Documented limitation.
- Cross-system test for probe paths matching ASGI routes — Test enhancement.
- Improved benchmarks (helm template timing vs PyYAML parsing) — Benchmark quality improvement.
- CI DRY violation (Helm install 3×) — Code quality improvement, consider composite action.
- File length limits exceeded (`k8s_helm_chart_steps.py` 551 lines, `helper_k8s_helm_chart.py` 678 lines) — Non-blocking, can be split in follow-up.
- `runAsGroup: 1000` in pod security context — Defense-in-depth improvement.
- HEAD method support on known paths — RFC compliance, does not affect K8s probes.
- `click.Choice` log-level validation via CLI runner test — Test gap.
### Scope note: status-check CI gate
The `status-check` job now includes `integration_tests`, `e2e_tests`, and `helm` in its `needs` list. The `helm` job is new in this PR. The `integration_tests` and `e2e_tests` additions fix previously-missing gate checks — included here since this PR modifies both of those jobs to install Helm.
### Quality gates
- `nox -e lint` ✅
- `nox -e typecheck` ✅
- `nox -e unit_tests` ✅ (12,321 scenarios passed, 4 skipped)
- `nox -e integration_tests` — 3 pre-existing failures in unrelated areas (plan correction, resource types)
- `nox -e e2e_tests` — pre-existing failures (LLM API keys not available in local env)
- `nox -e coverage_report` ✅ (**97.7%**)
Reviewed-on: cleveragents/cleveragents-core#1085
Reviewed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com>
Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Updated CI pipeline to inject ANTHROPIC_API_KEY and OPENAI_API_KEY
secrets as environment variables during Robot Framework integration
test execution. Added CI setup documentation.
ISSUES CLOSED: #701
Added dedicated E2E test infrastructure completely separate from the existing
integration test suite. E2E tests use zero mocking — they exercise the real
CleverAgents CLI with real LLM API keys.
Key changes:
- New e2e_tests nox session running Robot Framework with --include E2E tag
filter against robot/e2e/ directory. Uses sequential robot (not pabot)
since E2E tests hit real API endpoints with rate limits. Propagates
ANTHROPIC_API_KEY, OPENAI_API_KEY, and GOOGLE_API_KEY from environment.
Output goes to build/reports/robot-e2e/ to avoid artifact collisions.
- Existing integration_tests session now excludes E2E-tagged tests via
--exclude E2E on the pabot invocation.
- New robot/e2e/common_e2e.resource provides shared E2E keywords: suite
setup/teardown with per-suite isolation (no mock AI), graceful skip when
LLM API keys are absent, CLI runner keyword, flexible output assertions,
and temporary git repo fixture creation.
- Minimal smoke test (robot/e2e/smoke_test.robot) validates the harness by
running agents --version and agents --help. Does not require LLM keys.
- Dedicated e2e_tests CI job in .forgejo/workflows/ci.yml injects LLM API
keys from Forgejo secrets. Runs independently (no needs dependencies)
and does not block regular CI.
- e2e_tests is deliberately NOT in the default nox sessions list since it
requires real API keys not present in all environments.
ISSUES CLOSED: #740
Replace coverage.py (sys.settrace-based) with slipcover (bytecode-based
instrumentation) for significantly faster coverage collection:
- Each behave-parallel worker runs under slipcover, producing per-feature
JSON coverage files with unique UUIDs to avoid write contention.
- After all workers finish, slipcover --merge combines per-worker data
into a single build/coverage.json report.
- XML report generated via slipcover --merge --xml for CI tooling.
- Terminal report with --fail-under=97 threshold enforcement.
- Robust JSON key-fallback logic handles both slipcover and coverage.py
output formats.
- CI workflow (ci.yml, nightly-quality.yml) updated with defensive key
lookup instead of hardcoded coverage.py format.
- Documentation updated to reflect slipcover as the coverage tool.
- CHANGELOG.md updated.
ISSUES CLOSED: #482
- Rewrite .forgejo/workflows/ci.yml to route all jobs through nox sessions
- Fix coverage_report nox session: serial behave mode replaces broken parallel
mode (22% -> 97% accuracy), raise fail-under from 85% to 97%
- Pass posargs through format nox session for CI --check support
- Add 11 CI workflow validation scenarios (Behave) + Robot smoke test + ASV bench
- Add 108 new Behave scenarios covering 6 largest coverage gaps to reach 97%:
yaml_template_engine, actor/config, actor/registry, message_router,
context_analysis, context_service
- Update docs/development/ci-cd.md with nox-based CI docs and 97% threshold
- Restore implementation_plan.md verbose style, check off completed CI tasks
Verified: 1673 scenarios pass, 97% coverage, lint clean, typecheck clean
Forgejo act_runner service container networking fails with 'network not
found'. Use docker:dind directly with --privileged and start dockerd
inline to avoid the services networking issue.
- Add DinD service container and docker:cli image so docker CLI is available
- Fix Dockerfile: remove missing uv.lock, fix build order (copy source before build)
- Remove helm job (k8s/ directory does not exist yet)
freemo
brent.edwards
CoreRasurae
mngrif
