Files
brent.edwards b51df2ee0f
CI / docker (push) Blocked by required conditions
CI / status-check (push) Blocked by required conditions
CI / build (push) Successful in 20s
CI / helm (push) Successful in 23s
CI / lint (push) Successful in 3m19s
CI / typecheck (push) Successful in 3m57s
CI / benchmark-regression (push) Has been skipped
CI / security (push) Successful in 4m5s
CI / integration_tests (push) Successful in 7m25s
CI / unit_tests (push) Failing after 13m18s
CI / quality (push) Failing after 13m19s
CI / e2e_tests (push) Failing after 18m16s
CI / coverage (push) Failing after 19m20s
CI / benchmark-publish (push) Failing after 28m16s
feat(server): add Kubernetes Helm chart for server deployment (#1085)
## Summary

This PR adds Kubernetes Helm deployment support for the CleverAgents server.

Closes #928

### What this PR includes

- Helm chart under `k8s/` with Deployment, Service, optional Ingress, ConfigMap,
  ServiceAccount, Secrets, NOTES, and optional Redis subchart configuration.
- Multi-stage `Dockerfile.server` for server runtime deployment.
- Deployment-focused docs in `k8s/README.md`.
- Behave + Robot + benchmark coverage for chart/deployment wiring.

### Review fixes applied (cycle 11 — hurui200320 review #2687)

**Critical fix:**

1. **CI SHA256 checksum verification fixed** — All 3 Helm install blocks (`unit_tests`, `integration_tests`, `helm` jobs) now save the tarball using its original filename (`helm-v3.16.4-linux-amd64.tar.gz`) instead of `helm.tgz`, so the `.sha256sum` file can correctly locate and verify it. This was causing all 3 CI Helm jobs to fail with "No such file or directory".

**Major fixes (test coverage gaps):**

2. **405 Allow header now tested** — The existing "POST to known path returns method not allowed" scenario now asserts that the `Allow: GET` header is present. Added a second 405 scenario testing `POST /live` for broader `_KNOWN_PATHS` coverage (review item #11).
3. **Security-hardening headers now tested** — New scenario "HTTP responses include security-hardening headers" verifies `content-length`, `x-content-type-options: nosniff`, and `cache-control: no-store` headers are present on HTTP responses.
4. **Lifespan warning logging now tested** — New scenario "Unrecognised lifespan message type logs warning and continues" queues `lifespan.startup` → `lifespan.bogus` → `lifespan.shutdown` and verifies: (a) the app completes the lifespan cycle cleanly, (b) a warning is logged mentioning the unrecognised type.

### Review fixes applied (cycle 10)

**Critical/Major fixes (from hurui200320's prior REQUEST_CHANGES):**

1. **Rebased branch onto master** — Removed merge commit per CONTRIBUTING.md rebase-only policy. Clean linear history restored.
2. **Fixed commit message body** — Replaced literal `\n` sequences with actual newlines. `ISSUES CLOSED: #928` footer is now on its own line after a blank separator.
3. **Moved uvicorn import to module top level** — `from uvicorn import run as uvicorn_run` is now at the top of `src/cleveragents/cli/commands/server.py` per Import Guidelines. Updated test mock target from `uvicorn.run` to `cleveragents.cli.commands.server.uvicorn_run`.
4. **Added SHA256 checksum verification** — All three Helm CLI install blocks in `.forgejo/workflows/ci.yml` now download and verify `helm.sha256sum` before extracting the binary.

**Minor fixes:**

5. **Added `Dockerfile.server` build to CI** — New "Build Docker image (Server)" step in the `docker` job validates the server Dockerfile.
6. **ASGI 405 Method Not Allowed** — Known paths (`/`, `/live`, `/ready`, `/health`) now return 405 with `Allow: GET` header for non-GET methods, per RFC 9110 §15.5.6. Added `_KNOWN_PATHS` frozenset.
7. **WebSocket close protocol fix** — App now calls `await receive()` to consume the `websocket.connect` event before closing. Changed close code from 1000 (Normal Closure) to 1008 (Policy Violation).
8. **Lifespan handler logging** — Unrecognised lifespan message types are now logged as warnings instead of silently consumed.
9. **Security-hardening headers** — `_send_response` now includes `content-length`, `x-content-type-options: nosniff`, and `cache-control: no-store` on all HTTP responses.
10. **`.dockerignore` credential patterns** — Added `*.pem`, `*.key`, `*.p12`, `*.pfx`, `credentials*.json`.
11. **`--log-level` validation** — Constrained to `click.Choice(["critical", "error", "warning", "info", "debug", "trace"])` for clean CLI validation errors.
12. **Reverted unrelated semgrep pre-commit change** — `pass_filenames` and `entry` restored to original values per atomic commit hygiene.
13. **Removed unused `ReceiveCallable` type alias** and `Callable`/`Awaitable` imports from `asgi_app_steps.py`.
14. **Fixed redundant `shutil.which("helm")` check** — `_skip_if_helm_missing` now returns `bool` to eliminate the duplicate check in `_render_chart`.
15. **Improved test deque error handling** — Lifespan test receive mock now raises descriptive `AssertionError` instead of opaque `IndexError`.
16. **Scope type dispatch** — Changed `if/if/if` to `if/elif/elif` for mutually exclusive ASGI scope types.
17. **Dockerfile.server base image** — Standardised to `python:3.13-slim` (floating minor) consistent with CLI Dockerfile.
18. **Dockerfile layer caching** — Split `uv pip install build` and `python -m build` into separate `RUN` instructions.
19. **Removed extraneous double blank line** in Dockerfile.server.

### Deferred items (acknowledged, not in scope)

- PodDisruptionBudget, HorizontalPodAutoscaler, NetworkPolicy — Follow-up for production hardening.
- `appVersion: "1.0.0"` placeholder — Needs tracking issue for release versioning alignment.
- Readiness probe with downstream dependency checks — Documented limitation.
- Cross-system test for probe paths matching ASGI routes — Test enhancement.
- Improved benchmarks (helm template timing vs PyYAML parsing) — Benchmark quality improvement.
- CI DRY violation (Helm install 3×) — Code quality improvement, consider composite action.
- File length limits exceeded (`k8s_helm_chart_steps.py` 551 lines, `helper_k8s_helm_chart.py` 678 lines) — Non-blocking, can be split in follow-up.
- `runAsGroup: 1000` in pod security context — Defense-in-depth improvement.
- HEAD method support on known paths — RFC compliance, does not affect K8s probes.
- `click.Choice` log-level validation via CLI runner test — Test gap.

### Scope note: status-check CI gate

The `status-check` job now includes `integration_tests`, `e2e_tests`, and `helm` in its `needs` list. The `helm` job is new in this PR. The `integration_tests` and `e2e_tests` additions fix previously-missing gate checks — included here since this PR modifies both of those jobs to install Helm.

### Quality gates

- `nox -e lint` 
- `nox -e typecheck` 
- `nox -e unit_tests`  (12,321 scenarios passed, 4 skipped)
- `nox -e integration_tests` — 3 pre-existing failures in unrelated areas (plan correction, resource types)
- `nox -e e2e_tests` — pre-existing failures (LLM API keys not available in local env)
- `nox -e coverage_report`  (**97.7%**)

Reviewed-on: #1085
Reviewed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com>
Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
2026-03-27 23:53:22 +00:00

284 lines
9.6 KiB
Python

"""Step definitions for ASGI protocol behavior scenarios."""
from __future__ import annotations
import asyncio
import logging
from collections import deque
from typing import Any
from behave import given, then, when
from behave.runner import Context
from cleveragents.a2a.asgi import app
SendMessage = dict[str, Any]
@given("the ASGI app module is loaded")
def step_asgi_module_loaded(context: Context) -> None:
context.asgi_app = app
@when('I send an HTTP GET request to "{path}" through the ASGI app')
def step_send_http_request(context: Context, path: str) -> None:
sent_messages: list[SendMessage] = []
async def receive() -> dict[str, Any]:
return {"type": "http.request", "body": b"", "more_body": False}
async def send(message: SendMessage) -> None:
sent_messages.append(message)
scope = {"type": "http", "method": "GET", "path": path}
asyncio.run(context.asgi_app(scope, receive, send))
context.asgi_sent_messages = sent_messages
@when('I send an HTTP POST request to "{path}" through the ASGI app')
def step_send_http_post_request(context: Context, path: str) -> None:
sent_messages: list[SendMessage] = []
async def receive() -> dict[str, Any]:
return {"type": "http.request", "body": b"", "more_body": False}
async def send(message: SendMessage) -> None:
sent_messages.append(message)
scope = {"type": "http", "method": "POST", "path": path}
asyncio.run(context.asgi_app(scope, receive, send))
context.asgi_sent_messages = sent_messages
@then("the HTTP response status should be {status:d}")
def step_http_status(context: Context, status: int) -> None:
start = next(
msg
for msg in context.asgi_sent_messages
if msg.get("type") == "http.response.start"
)
assert start.get("status") == status, (
f"Expected HTTP status {status}, got {start.get('status')}"
)
@then('the HTTP response body should be "{body}"')
def step_http_body(context: Context, body: str) -> None:
response_body = next(
msg
for msg in context.asgi_sent_messages
if msg.get("type") == "http.response.body"
)
normalized_body = body.replace(r"\"", '"')
assert response_body.get("body") == normalized_body.encode("utf-8"), (
f"Expected body {normalized_body!r}, got {response_body.get('body')!r}"
)
@when("I run ASGI lifespan startup then shutdown")
def step_run_lifespan(context: Context) -> None:
sent_messages: list[SendMessage] = []
incoming = deque(
[
{"type": "lifespan.startup"},
{"type": "lifespan.shutdown"},
]
)
async def receive() -> dict[str, Any]:
if not incoming:
raise AssertionError(
"ASGI app issued an unexpected extra receive() call; "
"all queued lifespan messages have already been consumed"
)
return incoming.popleft()
async def send(message: SendMessage) -> None:
sent_messages.append(message)
scope = {"type": "lifespan"}
asyncio.run(context.asgi_app(scope, receive, send))
context.asgi_sent_messages = sent_messages
@then("the ASGI app should emit lifespan completion messages")
def step_assert_lifespan_messages(context: Context) -> None:
message_types = [str(msg.get("type")) for msg in context.asgi_sent_messages]
assert message_types == [
"lifespan.startup.complete",
"lifespan.shutdown.complete",
], f"Unexpected lifespan messages: {message_types}"
@then("no HTTP response frames should be emitted")
def step_no_http_frames(context: Context) -> None:
message_types = [str(msg.get("type")) for msg in context.asgi_sent_messages]
assert not any(
msg_type.startswith("http.response") for msg_type in message_types
), f"Unexpected HTTP response frames: {message_types}"
@when("I invoke the ASGI app with a websocket scope")
def step_websocket_scope(context: Context) -> None:
sent_messages: list[SendMessage] = []
async def receive() -> dict[str, Any]:
return {"type": "websocket.connect"}
async def send(message: SendMessage) -> None:
sent_messages.append(message)
scope = {"type": "websocket", "path": "/ws"}
asyncio.run(context.asgi_app(scope, receive, send))
context.asgi_sent_messages = sent_messages
@when("I invoke the ASGI app with an unsupported scope type")
def step_unsupported_scope(context: Context) -> None:
async def receive() -> dict[str, Any]:
return {"type": "unsupported.receive"}
async def send(message: SendMessage) -> None:
del message
scope = {"type": "unsupported", "path": "/"}
try:
asyncio.run(context.asgi_app(scope, receive, send))
except RuntimeError as exc:
context.asgi_error = exc
return
raise AssertionError("Expected RuntimeError for unsupported ASGI scope type")
@then("the ASGI app should emit a websocket close frame")
def step_websocket_close_frame(context: Context) -> None:
assert context.asgi_sent_messages == [{"type": "websocket.close", "code": 1008}], (
f"Unexpected websocket frames: {context.asgi_sent_messages}"
)
@then("the ASGI invocation should raise an unsupported scope runtime error")
def step_unsupported_scope_error(context: Context) -> None:
error = getattr(context, "asgi_error", None)
assert isinstance(error, RuntimeError), f"Expected RuntimeError, got {error!r}"
assert "Unsupported ASGI scope type" in str(error), (
f"Unexpected runtime error message: {error}"
)
# --- Allow header assertion for 405 responses ---
@then('the HTTP response should include an Allow header with value "{value}"')
def step_http_allow_header(context: Context, value: str) -> None:
start = next(
msg
for msg in context.asgi_sent_messages
if msg.get("type") == "http.response.start"
)
headers: list[tuple[bytes, bytes]] = start.get("headers", [])
allow_values = [v.decode("utf-8") for k, v in headers if k.lower() == b"allow"]
assert allow_values, f"Expected Allow header, but none found in: {headers}"
assert value in allow_values, (
f"Expected Allow header value {value!r}, got {allow_values!r}"
)
# --- Security-hardening header assertions ---
@then("the HTTP response should include a content-length header")
def step_http_content_length_header(context: Context) -> None:
start = next(
msg
for msg in context.asgi_sent_messages
if msg.get("type") == "http.response.start"
)
headers: list[tuple[bytes, bytes]] = start.get("headers", [])
header_names = [k.lower() for k, _v in headers]
assert b"content-length" in header_names, (
f"Expected content-length header, but found: {headers}"
)
@then('the HTTP response should include header "{name}" with value "{value}"')
def step_http_header_with_value(context: Context, name: str, value: str) -> None:
start = next(
msg
for msg in context.asgi_sent_messages
if msg.get("type") == "http.response.start"
)
headers: list[tuple[bytes, bytes]] = start.get("headers", [])
name_bytes = name.lower().encode("utf-8")
matched_values = [v.decode("utf-8") for k, v in headers if k.lower() == name_bytes]
assert matched_values, f"Expected {name} header, but none found in: {headers}"
assert value in matched_values, (
f"Expected {name} header value {value!r}, got {matched_values!r}"
)
# --- Unrecognised lifespan message type ---
@when("I run ASGI lifespan with an unrecognised message type")
def step_run_lifespan_with_unrecognised_type(context: Context) -> None:
sent_messages: list[SendMessage] = []
incoming = deque(
[
{"type": "lifespan.startup"},
{"type": "lifespan.bogus"},
{"type": "lifespan.shutdown"},
]
)
async def receive() -> dict[str, Any]:
if not incoming:
raise AssertionError(
"ASGI app issued an unexpected extra receive() call; "
"all queued lifespan messages have already been consumed"
)
return incoming.popleft()
async def send(message: SendMessage) -> None:
sent_messages.append(message)
scope = {"type": "lifespan"}
logger_name = "cleveragents.a2a.asgi"
with _capture_log_records(logger_name, logging.WARNING) as records:
asyncio.run(context.asgi_app(scope, receive, send))
context.asgi_sent_messages = sent_messages
context.asgi_log_records = records
@then("a warning should be logged for the unrecognised lifespan message type")
def step_assert_lifespan_warning_logged(context: Context) -> None:
records: list[logging.LogRecord] = context.asgi_log_records
warning_messages = [r.getMessage() for r in records]
assert any("lifespan.bogus" in msg for msg in warning_messages), (
f"Expected a warning about 'lifespan.bogus', got: {warning_messages}"
)
class _capture_log_records:
"""Context manager that captures log records for a named logger."""
def __init__(self, logger_name: str, level: int) -> None:
self._logger = logging.getLogger(logger_name)
self._level = level
self._handler: logging.Handler | None = None
self.records: list[logging.LogRecord] = []
def __enter__(self) -> list[logging.LogRecord]:
handler = logging.Handler()
handler.setLevel(self._level)
handler.emit = self.records.append # type: ignore[assignment]
self._handler = handler
self._logger.addHandler(handler)
self._logger.setLevel(self._level)
return self.records
def __exit__(self, *_args: object) -> None:
if self._handler is not None:
self._logger.removeHandler(self._handler)