forked from cleveragents/cleveragents-core
brent.edwards
b51df2ee0f
## Summary This PR adds Kubernetes Helm deployment support for the CleverAgents server. Closes #928 ### What this PR includes - Helm chart under `k8s/` with Deployment, Service, optional Ingress, ConfigMap, ServiceAccount, Secrets, NOTES, and optional Redis subchart configuration. - Multi-stage `Dockerfile.server` for server runtime deployment. - Deployment-focused docs in `k8s/README.md`. - Behave + Robot + benchmark coverage for chart/deployment wiring. ### Review fixes applied (cycle 11 — hurui200320 review #2687) **Critical fix:** 1. **CI SHA256 checksum verification fixed** — All 3 Helm install blocks (`unit_tests`, `integration_tests`, `helm` jobs) now save the tarball using its original filename (`helm-v3.16.4-linux-amd64.tar.gz`) instead of `helm.tgz`, so the `.sha256sum` file can correctly locate and verify it. This was causing all 3 CI Helm jobs to fail with "No such file or directory". **Major fixes (test coverage gaps):** 2. **405 Allow header now tested** — The existing "POST to known path returns method not allowed" scenario now asserts that the `Allow: GET` header is present. Added a second 405 scenario testing `POST /live` for broader `_KNOWN_PATHS` coverage (review item #11). 3. **Security-hardening headers now tested** — New scenario "HTTP responses include security-hardening headers" verifies `content-length`, `x-content-type-options: nosniff`, and `cache-control: no-store` headers are present on HTTP responses. 4. **Lifespan warning logging now tested** — New scenario "Unrecognised lifespan message type logs warning and continues" queues `lifespan.startup` → `lifespan.bogus` → `lifespan.shutdown` and verifies: (a) the app completes the lifespan cycle cleanly, (b) a warning is logged mentioning the unrecognised type. ### Review fixes applied (cycle 10) **Critical/Major fixes (from hurui200320's prior REQUEST_CHANGES):** 1. **Rebased branch onto master** — Removed merge commit per CONTRIBUTING.md rebase-only policy. Clean linear history restored. 2. **Fixed commit message body** — Replaced literal `\n` sequences with actual newlines. `ISSUES CLOSED: #928` footer is now on its own line after a blank separator. 3. **Moved uvicorn import to module top level** — `from uvicorn import run as uvicorn_run` is now at the top of `src/cleveragents/cli/commands/server.py` per Import Guidelines. Updated test mock target from `uvicorn.run` to `cleveragents.cli.commands.server.uvicorn_run`. 4. **Added SHA256 checksum verification** — All three Helm CLI install blocks in `.forgejo/workflows/ci.yml` now download and verify `helm.sha256sum` before extracting the binary. **Minor fixes:** 5. **Added `Dockerfile.server` build to CI** — New "Build Docker image (Server)" step in the `docker` job validates the server Dockerfile. 6. **ASGI 405 Method Not Allowed** — Known paths (`/`, `/live`, `/ready`, `/health`) now return 405 with `Allow: GET` header for non-GET methods, per RFC 9110 §15.5.6. Added `_KNOWN_PATHS` frozenset. 7. **WebSocket close protocol fix** — App now calls `await receive()` to consume the `websocket.connect` event before closing. Changed close code from 1000 (Normal Closure) to 1008 (Policy Violation). 8. **Lifespan handler logging** — Unrecognised lifespan message types are now logged as warnings instead of silently consumed. 9. **Security-hardening headers** — `_send_response` now includes `content-length`, `x-content-type-options: nosniff`, and `cache-control: no-store` on all HTTP responses. 10. **`.dockerignore` credential patterns** — Added `*.pem`, `*.key`, `*.p12`, `*.pfx`, `credentials*.json`. 11. **`--log-level` validation** — Constrained to `click.Choice(["critical", "error", "warning", "info", "debug", "trace"])` for clean CLI validation errors. 12. **Reverted unrelated semgrep pre-commit change** — `pass_filenames` and `entry` restored to original values per atomic commit hygiene. 13. **Removed unused `ReceiveCallable` type alias** and `Callable`/`Awaitable` imports from `asgi_app_steps.py`. 14. **Fixed redundant `shutil.which("helm")` check** — `_skip_if_helm_missing` now returns `bool` to eliminate the duplicate check in `_render_chart`. 15. **Improved test deque error handling** — Lifespan test receive mock now raises descriptive `AssertionError` instead of opaque `IndexError`. 16. **Scope type dispatch** — Changed `if/if/if` to `if/elif/elif` for mutually exclusive ASGI scope types. 17. **Dockerfile.server base image** — Standardised to `python:3.13-slim` (floating minor) consistent with CLI Dockerfile. 18. **Dockerfile layer caching** — Split `uv pip install build` and `python -m build` into separate `RUN` instructions. 19. **Removed extraneous double blank line** in Dockerfile.server. ### Deferred items (acknowledged, not in scope) - PodDisruptionBudget, HorizontalPodAutoscaler, NetworkPolicy — Follow-up for production hardening. - `appVersion: "1.0.0"` placeholder — Needs tracking issue for release versioning alignment. - Readiness probe with downstream dependency checks — Documented limitation. - Cross-system test for probe paths matching ASGI routes — Test enhancement. - Improved benchmarks (helm template timing vs PyYAML parsing) — Benchmark quality improvement. - CI DRY violation (Helm install 3×) — Code quality improvement, consider composite action. - File length limits exceeded (`k8s_helm_chart_steps.py` 551 lines, `helper_k8s_helm_chart.py` 678 lines) — Non-blocking, can be split in follow-up. - `runAsGroup: 1000` in pod security context — Defense-in-depth improvement. - HEAD method support on known paths — RFC compliance, does not affect K8s probes. - `click.Choice` log-level validation via CLI runner test — Test gap. ### Scope note: status-check CI gate The `status-check` job now includes `integration_tests`, `e2e_tests`, and `helm` in its `needs` list. The `helm` job is new in this PR. The `integration_tests` and `e2e_tests` additions fix previously-missing gate checks — included here since this PR modifies both of those jobs to install Helm. ### Quality gates - `nox -e lint` ✅ - `nox -e typecheck` ✅ - `nox -e unit_tests` ✅ (12,321 scenarios passed, 4 skipped) - `nox -e integration_tests` — 3 pre-existing failures in unrelated areas (plan correction, resource types) - `nox -e e2e_tests` — pre-existing failures (LLM API keys not available in local env) - `nox -e coverage_report` ✅ (**97.7%**) Reviewed-on: cleveragents/cleveragents-core#1085 Reviewed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com> Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com> Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
290 lines
13 KiB
Gherkin
290 lines
13 KiB
Gherkin
Feature: Kubernetes Helm Chart Structure
|
|
As a DevOps engineer
|
|
I want a valid Helm chart in the k8s/ directory
|
|
So that I can deploy the CleverAgents server to Kubernetes
|
|
|
|
Background:
|
|
Given the project root directory
|
|
|
|
# -- Chart.yaml validation --
|
|
|
|
Scenario: Chart.yaml exists with required Helm v2 fields
|
|
When I read the Helm chart metadata
|
|
Then the chart API version should be "v2"
|
|
And the chart name should be "cleveragents"
|
|
And the chart type should be "application"
|
|
And the chart version should be set
|
|
|
|
Scenario: Chart.yaml declares optional Redis dependency
|
|
When I read the Helm chart metadata
|
|
Then the chart should have a dependency named "redis"
|
|
And the Redis dependency should be conditional on "redis.enabled"
|
|
|
|
# -- values.yaml validation --
|
|
|
|
Scenario: values.yaml has configurable replica count
|
|
When I read the Helm chart values
|
|
Then the values should contain key "replicaCount"
|
|
And the default replica count should be 1
|
|
|
|
Scenario: values.yaml has configurable resource limits
|
|
When I read the Helm chart values
|
|
Then the values should contain key "resources"
|
|
And the resources should have CPU and memory limits
|
|
And the resources should have CPU and memory requests
|
|
|
|
Scenario: values.yaml has ingress configuration with TLS support
|
|
When I read the Helm chart values
|
|
Then the values should contain key "ingress"
|
|
And the ingress should have an enabled toggle
|
|
And the ingress should have a TLS configuration section
|
|
|
|
Scenario: values.yaml has optional Redis configuration
|
|
When I read the Helm chart values
|
|
Then the values should contain key "redis"
|
|
And the Redis configuration should have an enabled toggle
|
|
And the Redis configuration should have authentication settings
|
|
|
|
Scenario: values.yaml has server configuration
|
|
When I read the Helm chart values
|
|
Then the values should contain key "server"
|
|
And the server config should have host setting
|
|
And the server config should have port setting
|
|
And the server config should have workers setting
|
|
|
|
Scenario: values.yaml has database configuration
|
|
When I read the Helm chart values
|
|
Then the values should contain key "database"
|
|
And the database config should support existing secrets
|
|
|
|
Scenario: values.yaml has security context for non-root execution
|
|
When I read the Helm chart values
|
|
Then the pod security context should enforce non-root execution
|
|
And the container security context should drop all capabilities
|
|
|
|
# -- Template file content validation --
|
|
|
|
Scenario: Deployment template has correct kind and API version
|
|
When I read the template "k8s/templates/deployment.yaml"
|
|
Then the template should contain "kind: Deployment"
|
|
And the template should contain "apiVersion: apps/v1"
|
|
And the template should contain "volumeMounts:"
|
|
And the template should contain "mountPath: /tmp"
|
|
And the template should contain "mountPath: /app/data"
|
|
And the template should contain "sizeLimit: 100Mi"
|
|
And the template should contain "sizeLimit: 1Gi"
|
|
|
|
Scenario: Service template has correct kind and API version
|
|
When I read the template "k8s/templates/service.yaml"
|
|
Then the template should contain "kind: Service"
|
|
And the template should contain "apiVersion: v1"
|
|
|
|
Scenario: Ingress template has correct kind and API version
|
|
When I read the template "k8s/templates/ingress.yaml"
|
|
Then the template should contain "kind: Ingress"
|
|
And the template should contain "networking.k8s.io/v1"
|
|
And the template should match pattern "^\s+tls:"
|
|
|
|
Scenario: ConfigMap template has correct kind and API version
|
|
When I read the template "k8s/templates/configmap.yaml"
|
|
Then the template should contain "kind: ConfigMap"
|
|
And the template should contain "apiVersion: v1"
|
|
|
|
Scenario: ServiceAccount template has correct kind and API version
|
|
When I read the template "k8s/templates/serviceaccount.yaml"
|
|
Then the template should contain "kind: ServiceAccount"
|
|
And the template should contain "apiVersion: v1"
|
|
And the template should contain "automountServiceAccountToken: false"
|
|
|
|
Scenario: Secrets template has correct kind
|
|
When I read the template "k8s/templates/secrets.yaml"
|
|
Then the template should contain "kind: Secret"
|
|
|
|
Scenario: Helpers template exists with chart helpers
|
|
When I read the template "k8s/templates/_helpers.tpl"
|
|
Then the template should contain "cleveragents.fullname"
|
|
And the template should contain "cleveragents.labels"
|
|
|
|
Scenario: NOTES.txt template exists with deployment info
|
|
When I read the template "k8s/templates/NOTES.txt"
|
|
Then the template should contain "CleverAgents Server has been deployed"
|
|
|
|
# -- Server Dockerfile --
|
|
|
|
Scenario: Server-specific Dockerfile exists
|
|
Then the chart file "Dockerfile.server" should exist
|
|
|
|
Scenario: Server Dockerfile uses multi-stage build
|
|
When I read the server Dockerfile
|
|
Then it should have a builder stage
|
|
And it should have a runtime stage
|
|
|
|
Scenario: Server Dockerfile uses non-root user
|
|
When I read the server Dockerfile
|
|
Then it should create a non-root user with uid 1000
|
|
|
|
Scenario: Server Dockerfile exposes port 8000
|
|
When I read the server Dockerfile
|
|
Then it should expose port 8000
|
|
|
|
Scenario: Server Dockerfile pins uv image version
|
|
When I read the server Dockerfile
|
|
Then it should not use unpinned uv latest tag
|
|
|
|
# -- Secrets structure --
|
|
|
|
Scenario: Secrets template guards database secret with existingSecret check
|
|
When I read the template "k8s/templates/secrets.yaml"
|
|
Then the template should contain "database.existingSecret"
|
|
And the template should contain "database.url"
|
|
And the template should contain "database-url"
|
|
|
|
Scenario: Secrets template guards Redis secret with existingSecret check
|
|
When I read the template "k8s/templates/secrets.yaml"
|
|
Then the template should contain "redis.auth.existingSecret"
|
|
And the template should contain "redis.auth.password"
|
|
And the template should contain "redis-password"
|
|
|
|
# -- Deployment command/args --
|
|
|
|
Scenario: Deployment template overrides command with CLI server serve values
|
|
When I read the template "k8s/templates/deployment.yaml"
|
|
Then the template should contain "python"
|
|
And the template should contain "cleveragents"
|
|
And the template should contain "server"
|
|
And the template should contain "serve"
|
|
And the template should contain "--app"
|
|
And the template should contain "cleveragents.a2a.asgi:app"
|
|
And the template should contain "server.host"
|
|
And the template should contain "server.port"
|
|
And the template should contain "server.workers"
|
|
And the template should contain "server.logLevel"
|
|
|
|
# -- envFrom wiring --
|
|
|
|
Scenario: Deployment template injects ConfigMap via envFrom
|
|
When I read the template "k8s/templates/deployment.yaml"
|
|
Then the template should contain "envFrom"
|
|
And the template should contain "configMapRef"
|
|
|
|
# -- Redis host helper --
|
|
|
|
Scenario: Helpers template defines Redis host using Release.Name
|
|
When I read the template "k8s/templates/_helpers.tpl"
|
|
Then the template should contain "cleveragents.redisHost"
|
|
And the template should contain ".Release.Name"
|
|
|
|
# -- Probes --
|
|
|
|
Scenario: values.yaml splits liveness and readiness probe endpoints
|
|
When I read the Helm chart values
|
|
Then the liveness probe should target the live endpoint
|
|
And the readiness probe should target the ready endpoint
|
|
|
|
# -- Image defaults --
|
|
|
|
Scenario: values.yaml has sensible image defaults
|
|
When I read the Helm chart values
|
|
Then the image pull policy should be "IfNotPresent"
|
|
And the image tag should default to empty for appVersion fallback
|
|
|
|
# -- Service defaults --
|
|
|
|
Scenario: values.yaml service defaults to ClusterIP on port 8000
|
|
When I read the Helm chart values
|
|
Then the service type should be "ClusterIP"
|
|
And the service port should be 8000
|
|
|
|
# -- Conditional template branch validation --
|
|
|
|
Scenario: Ingress template is guarded by ingress.enabled condition
|
|
When I read the template "k8s/templates/ingress.yaml"
|
|
Then the template should match pattern "^\{\{-\s*if\s+\.Values\.ingress\.enabled"
|
|
|
|
Scenario: Ingress template guards ssl-redirect against user annotation override
|
|
When I read the template "k8s/templates/ingress.yaml"
|
|
Then the template should match pattern "not.*hasKey.*\.Values\.ingress\.annotations.*ssl-redirect"
|
|
|
|
Scenario: ServiceAccount template is guarded by serviceAccount.create condition
|
|
When I read the template "k8s/templates/serviceaccount.yaml"
|
|
Then the template should match pattern "^\{\{-\s*if\s+\.Values\.serviceAccount\.create"
|
|
|
|
Scenario: Secrets template guards database secret with conditional logic
|
|
When I read the template "k8s/templates/secrets.yaml"
|
|
Then the template should match pattern "if\s+and\s+\.Values\.database\.url.*not\s+\.Values\.database\.existingSecret"
|
|
|
|
Scenario: Secrets template guards Redis secret with conditional logic
|
|
When I read the template "k8s/templates/secrets.yaml"
|
|
Then the template should match pattern "if\s+and\s+\.Values\.redis\.enabled\s+\.Values\.redis\.auth\.enabled\s+\.Values\.redis\.auth\.password.*not\s+\.Values\.redis\.auth\.existingSecret"
|
|
|
|
Scenario: Deployment template has database credential 3-way branching
|
|
When I read the template "k8s/templates/deployment.yaml"
|
|
Then the template should match pattern "if\s+\.Values\.database\.existingSecret"
|
|
And the template should match pattern "else\s+if\s+\.Values\.database\.url"
|
|
|
|
Scenario: Deployment template has Redis credential 3-way branching
|
|
When I read the template "k8s/templates/deployment.yaml"
|
|
Then the template should match pattern "if\s+\.Values\.redis\.auth\.existingSecret"
|
|
And the template should match pattern "else\s+if\s+\.Values\.redis\.auth\.password"
|
|
And the template should contain "name: {{ .Release.Name }}-redis"
|
|
And the template should contain "key: redis-password"
|
|
|
|
Scenario: Deployment template gates Redis env vars behind redis.enabled
|
|
When I read the template "k8s/templates/deployment.yaml"
|
|
Then the template should match pattern "if\s+\.Values\.redis\.enabled"
|
|
And the template should match pattern "CLEVERAGENTS_REDIS_HOST"
|
|
And the template should match pattern "CLEVERAGENTS_REDIS_PASSWORD"
|
|
|
|
Scenario: Deployment template requires explicit database configuration
|
|
When I read the template "k8s/templates/deployment.yaml"
|
|
Then the template should contain "database configuration is required"
|
|
|
|
Scenario: Helm render fails fast without database configuration
|
|
When I render the Helm chart without database configuration
|
|
Then the render should fail with required database configuration error
|
|
|
|
Scenario: Helm render fails when ingress is enabled without TLS by default
|
|
When I render the Helm chart with ingress enabled and no TLS
|
|
Then the render should fail with required ingress TLS configuration error
|
|
|
|
Scenario: Helm render allows explicit insecure ingress opt-out for development
|
|
When I render the Helm chart with ingress enabled and insecure opt-out enabled
|
|
Then the ingress render should succeed in insecure dev mode
|
|
|
|
Scenario: Helm render succeeds when ingress TLS is explicitly configured
|
|
When I render the Helm chart with ingress enabled and TLS configured
|
|
Then the ingress render should succeed with TLS enabled
|
|
|
|
Scenario: Deployment template disables service account token mounting
|
|
When I read the template "k8s/templates/deployment.yaml"
|
|
Then the template should match pattern "^\s+automountServiceAccountToken:\s+false"
|
|
|
|
Scenario: Security context includes seccomp profile
|
|
When I read the Helm chart values
|
|
Then the container security context should have a seccomp profile
|
|
|
|
Scenario: Security context enforces readOnlyRootFilesystem
|
|
When I read the Helm chart values
|
|
Then the container security context should set readOnlyRootFilesystem to true
|
|
|
|
Scenario: Security context disables privilege escalation
|
|
When I read the Helm chart values
|
|
Then the container security context should set allowPrivilegeEscalation to false
|
|
|
|
Scenario: Server Dockerfile pins uv version to specific semver
|
|
When I read the server Dockerfile
|
|
Then the uv image reference should use a specific semver tag
|
|
|
|
Scenario: Server Dockerfile removes uv from runtime image
|
|
When I read the server Dockerfile
|
|
Then the Dockerfile should remove uv after installation
|
|
|
|
# -- Deployment README --
|
|
|
|
Scenario: Deployment README exists with instructions
|
|
When I read the deployment README
|
|
Then it should contain quick start instructions
|
|
And it should contain TLS configuration documentation
|
|
And it should contain Redis configuration documentation
|
|
And it should contain scaling instructions
|