Add kubeconform manifest validation to the existing helm CI job.
The helm job already had helm lint and helm template steps (added in
#1085). This commit completes the optional acceptance criterion from
#1089 by adding kubeconform v0.7.0 to validate rendered manifests
against the Kubernetes 1.29.0 schema in strict mode.
Changes:
- Install kubeconform v0.7.0 in the helm CI job
- Add 'Validate rendered manifests with kubeconform' step after
helm template smoke render (strict mode, ignore-missing-schemas,
kubernetes-version 1.29.0)
- Add 5 BDD scenarios to ci_workflow_validation.feature covering:
- helm job existence
- helm lint step
- helm template step
- kubeconform validation step
- status-check dependency on helm job
ISSUES CLOSED: #1089
## Summary
This PR adds a Behave TDD bug-capture test that proves bug #1039 exists: the specification references `validations/unit-tests.yaml` as an example validation configuration file (Workflow Examples, Example 1, Step 1) but the file does not exist in the project's `examples/validations/` directory.
## Changes
- **New feature file:** `features/tdd_missing_validation_unit_tests_yaml.feature` — four scenarios tagged `@tdd_expected_fail @tdd_bug @tdd_bug_1039` that verify:
1. The file `examples/validations/unit-tests.yaml` exists
2. The YAML config contains `name: local/unit-tests`
3. The validation mode is `required`
4. The description field is non-empty
- **New step file:** `features/steps/tdd_missing_validation_unit_tests_yaml_steps.py` — step implementations that locate the project root and check file existence/content
All four scenarios currently fail with `AssertionError` (the file doesn't exist, proving the bug exists). The `@tdd_expected_fail` tag inverts the results so the test suite passes CI.
## Robot Integration Test
N/A — this bug is purely about a missing example file referenced by the specification. There is no integration-level behavior to test.
## Quality Gate Results
| Gate | Result |
|------|--------|
| `nox -s lint` | passed |
| `nox -s typecheck` | passed (0 errors) |
| `nox -s unit_tests` | 462 features, 12234 scenarios, 0 failures |
| `nox -s integration_tests` | 1672 tests, 0 failures |
| `nox -s e2e_tests` | 37 tests, 0 failures |
| `nox -s coverage_report` | 98% (>= 97% threshold) |
Closes#1103
Reviewed-on: cleveragents/cleveragents-core#1132
Reviewed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com>
Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
## Summary
This PR adds Kubernetes Helm deployment support for the CleverAgents server.
Closes#928
### What this PR includes
- Helm chart under `k8s/` with Deployment, Service, optional Ingress, ConfigMap,
ServiceAccount, Secrets, NOTES, and optional Redis subchart configuration.
- Multi-stage `Dockerfile.server` for server runtime deployment.
- Deployment-focused docs in `k8s/README.md`.
- Behave + Robot + benchmark coverage for chart/deployment wiring.
### Review fixes applied (cycle 11 — hurui200320 review #2687)
**Critical fix:**
1. **CI SHA256 checksum verification fixed** — All 3 Helm install blocks (`unit_tests`, `integration_tests`, `helm` jobs) now save the tarball using its original filename (`helm-v3.16.4-linux-amd64.tar.gz`) instead of `helm.tgz`, so the `.sha256sum` file can correctly locate and verify it. This was causing all 3 CI Helm jobs to fail with "No such file or directory".
**Major fixes (test coverage gaps):**
2. **405 Allow header now tested** — The existing "POST to known path returns method not allowed" scenario now asserts that the `Allow: GET` header is present. Added a second 405 scenario testing `POST /live` for broader `_KNOWN_PATHS` coverage (review item #11).
3. **Security-hardening headers now tested** — New scenario "HTTP responses include security-hardening headers" verifies `content-length`, `x-content-type-options: nosniff`, and `cache-control: no-store` headers are present on HTTP responses.
4. **Lifespan warning logging now tested** — New scenario "Unrecognised lifespan message type logs warning and continues" queues `lifespan.startup` → `lifespan.bogus` → `lifespan.shutdown` and verifies: (a) the app completes the lifespan cycle cleanly, (b) a warning is logged mentioning the unrecognised type.
### Review fixes applied (cycle 10)
**Critical/Major fixes (from hurui200320's prior REQUEST_CHANGES):**
1. **Rebased branch onto master** — Removed merge commit per CONTRIBUTING.md rebase-only policy. Clean linear history restored.
2. **Fixed commit message body** — Replaced literal `\n` sequences with actual newlines. `ISSUES CLOSED: #928` footer is now on its own line after a blank separator.
3. **Moved uvicorn import to module top level** — `from uvicorn import run as uvicorn_run` is now at the top of `src/cleveragents/cli/commands/server.py` per Import Guidelines. Updated test mock target from `uvicorn.run` to `cleveragents.cli.commands.server.uvicorn_run`.
4. **Added SHA256 checksum verification** — All three Helm CLI install blocks in `.forgejo/workflows/ci.yml` now download and verify `helm.sha256sum` before extracting the binary.
**Minor fixes:**
5. **Added `Dockerfile.server` build to CI** — New "Build Docker image (Server)" step in the `docker` job validates the server Dockerfile.
6. **ASGI 405 Method Not Allowed** — Known paths (`/`, `/live`, `/ready`, `/health`) now return 405 with `Allow: GET` header for non-GET methods, per RFC 9110 §15.5.6. Added `_KNOWN_PATHS` frozenset.
7. **WebSocket close protocol fix** — App now calls `await receive()` to consume the `websocket.connect` event before closing. Changed close code from 1000 (Normal Closure) to 1008 (Policy Violation).
8. **Lifespan handler logging** — Unrecognised lifespan message types are now logged as warnings instead of silently consumed.
9. **Security-hardening headers** — `_send_response` now includes `content-length`, `x-content-type-options: nosniff`, and `cache-control: no-store` on all HTTP responses.
10. **`.dockerignore` credential patterns** — Added `*.pem`, `*.key`, `*.p12`, `*.pfx`, `credentials*.json`.
11. **`--log-level` validation** — Constrained to `click.Choice(["critical", "error", "warning", "info", "debug", "trace"])` for clean CLI validation errors.
12. **Reverted unrelated semgrep pre-commit change** — `pass_filenames` and `entry` restored to original values per atomic commit hygiene.
13. **Removed unused `ReceiveCallable` type alias** and `Callable`/`Awaitable` imports from `asgi_app_steps.py`.
14. **Fixed redundant `shutil.which("helm")` check** — `_skip_if_helm_missing` now returns `bool` to eliminate the duplicate check in `_render_chart`.
15. **Improved test deque error handling** — Lifespan test receive mock now raises descriptive `AssertionError` instead of opaque `IndexError`.
16. **Scope type dispatch** — Changed `if/if/if` to `if/elif/elif` for mutually exclusive ASGI scope types.
17. **Dockerfile.server base image** — Standardised to `python:3.13-slim` (floating minor) consistent with CLI Dockerfile.
18. **Dockerfile layer caching** — Split `uv pip install build` and `python -m build` into separate `RUN` instructions.
19. **Removed extraneous double blank line** in Dockerfile.server.
### Deferred items (acknowledged, not in scope)
- PodDisruptionBudget, HorizontalPodAutoscaler, NetworkPolicy — Follow-up for production hardening.
- `appVersion: "1.0.0"` placeholder — Needs tracking issue for release versioning alignment.
- Readiness probe with downstream dependency checks — Documented limitation.
- Cross-system test for probe paths matching ASGI routes — Test enhancement.
- Improved benchmarks (helm template timing vs PyYAML parsing) — Benchmark quality improvement.
- CI DRY violation (Helm install 3×) — Code quality improvement, consider composite action.
- File length limits exceeded (`k8s_helm_chart_steps.py` 551 lines, `helper_k8s_helm_chart.py` 678 lines) — Non-blocking, can be split in follow-up.
- `runAsGroup: 1000` in pod security context — Defense-in-depth improvement.
- HEAD method support on known paths — RFC compliance, does not affect K8s probes.
- `click.Choice` log-level validation via CLI runner test — Test gap.
### Scope note: status-check CI gate
The `status-check` job now includes `integration_tests`, `e2e_tests`, and `helm` in its `needs` list. The `helm` job is new in this PR. The `integration_tests` and `e2e_tests` additions fix previously-missing gate checks — included here since this PR modifies both of those jobs to install Helm.
### Quality gates
- `nox -e lint` ✅
- `nox -e typecheck` ✅
- `nox -e unit_tests` ✅ (12,321 scenarios passed, 4 skipped)
- `nox -e integration_tests` — 3 pre-existing failures in unrelated areas (plan correction, resource types)
- `nox -e e2e_tests` — pre-existing failures (LLM API keys not available in local env)
- `nox -e coverage_report` ✅ (**97.7%**)
Reviewed-on: cleveragents/cleveragents-core#1085
Reviewed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com>
Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Updated CI pipeline to inject ANTHROPIC_API_KEY and OPENAI_API_KEY
secrets as environment variables during Robot Framework integration
test execution. Added CI setup documentation.
ISSUES CLOSED: #701
Added dedicated E2E test infrastructure completely separate from the existing
integration test suite. E2E tests use zero mocking — they exercise the real
CleverAgents CLI with real LLM API keys.
Key changes:
- New e2e_tests nox session running Robot Framework with --include E2E tag
filter against robot/e2e/ directory. Uses sequential robot (not pabot)
since E2E tests hit real API endpoints with rate limits. Propagates
ANTHROPIC_API_KEY, OPENAI_API_KEY, and GOOGLE_API_KEY from environment.
Output goes to build/reports/robot-e2e/ to avoid artifact collisions.
- Existing integration_tests session now excludes E2E-tagged tests via
--exclude E2E on the pabot invocation.
- New robot/e2e/common_e2e.resource provides shared E2E keywords: suite
setup/teardown with per-suite isolation (no mock AI), graceful skip when
LLM API keys are absent, CLI runner keyword, flexible output assertions,
and temporary git repo fixture creation.
- Minimal smoke test (robot/e2e/smoke_test.robot) validates the harness by
running agents --version and agents --help. Does not require LLM keys.
- Dedicated e2e_tests CI job in .forgejo/workflows/ci.yml injects LLM API
keys from Forgejo secrets. Runs independently (no needs dependencies)
and does not block regular CI.
- e2e_tests is deliberately NOT in the default nox sessions list since it
requires real API keys not present in all environments.
ISSUES CLOSED: #740
Replace coverage.py (sys.settrace-based) with slipcover (bytecode-based
instrumentation) for significantly faster coverage collection:
- Each behave-parallel worker runs under slipcover, producing per-feature
JSON coverage files with unique UUIDs to avoid write contention.
- After all workers finish, slipcover --merge combines per-worker data
into a single build/coverage.json report.
- XML report generated via slipcover --merge --xml for CI tooling.
- Terminal report with --fail-under=97 threshold enforcement.
- Robust JSON key-fallback logic handles both slipcover and coverage.py
output formats.
- CI workflow (ci.yml, nightly-quality.yml) updated with defensive key
lookup instead of hardcoded coverage.py format.
- Documentation updated to reflect slipcover as the coverage tool.
- CHANGELOG.md updated.
ISSUES CLOSED: #482
- Rewrite .forgejo/workflows/ci.yml to route all jobs through nox sessions
- Fix coverage_report nox session: serial behave mode replaces broken parallel
mode (22% -> 97% accuracy), raise fail-under from 85% to 97%
- Pass posargs through format nox session for CI --check support
- Add 11 CI workflow validation scenarios (Behave) + Robot smoke test + ASV bench
- Add 108 new Behave scenarios covering 6 largest coverage gaps to reach 97%:
yaml_template_engine, actor/config, actor/registry, message_router,
context_analysis, context_service
- Update docs/development/ci-cd.md with nox-based CI docs and 97% threshold
- Restore implementation_plan.md verbose style, check off completed CI tasks
Verified: 1673 scenarios pass, 97% coverage, lint clean, typecheck clean
Forgejo act_runner service container networking fails with 'network not
found'. Use docker:dind directly with --privileged and start dockerd
inline to avoid the services networking issue.
- Add DinD service container and docker:cli image so docker CLI is available
- Fix Dockerfile: remove missing uv.lock, fix build order (copy source before build)
- Remove helm job (k8s/ directory does not exist yet)
Forgejo shows the following error: ::error::@actions/artifact v2.0.0+, upload-artifact@v4+ and download-artifact@v4+ are not currently supported on GHES.
freemo
brent.edwards
CoreRasurae
mngrif
