UAT: TraceService missing on_context_built hook — CONTEXT_BUILD_TIME_MS and CONTEXT_TOKEN_COUNT metrics never emitted #3768

New issue

Open

opened 2026-04-05 22:34:34 +00:00 by freemo · 1 comment

freemo commented

2026-04-05 22:34:34 +00:00

Owner

Bug Report

What Was Tested

Code-level analysis of src/cleveragents/application/services/trace_service.py — specifically the coverage of context-related operational metrics defined in docs/specification.md §Observability → Metrics Collection.

Expected Behavior (from spec)

The spec (line ~46117) defines these context-level metrics:

Metric	Type	Description
`context.build_duration_ms`	Histogram	Context building time by tier (hot/warm/cold)
`context.tokens_used`	Gauge	Current token usage in hot context

The OperationalMetricKey enum in metrics.py defines CONTEXT_BUILD_TIME_MS and CONTEXT_TOKEN_COUNT for these metrics. TraceService has hooks for plan lifecycle (on_plan_start), actor invocations (on_actor_invocation), and tool executions (on_tool_execution), but there is no hook for context building events.

Actual Behavior

TraceService has no on_context_built hook. The CONTEXT_BUILD_TIME_MS and CONTEXT_TOKEN_COUNT metric keys exist in OperationalMetricKey and METRIC_DEFINITIONS but are never emitted by any TraceService method:

# metrics.py — these keys exist but are never used by TraceService
OperationalMetricKey.CONTEXT_BUILD_TIME_MS = "context_build_time_ms"
OperationalMetricKey.CONTEXT_TOKEN_COUNT = "context_token_count"

The MetricCollector class has convenience helpers context_build_time() and context_token_count() that are also never called.

Code Location

File: src/cleveragents/application/services/trace_service.py
Lines 171–248 (the plan lifecycle hooks section — on_context_built is absent)

Impact

Context build time is never recorded — the ACMS performance cannot be monitored
Context token count is never recorded — hot context size is invisible
The diagnostic dashboard's "Performance summary" (spec line ~46132) cannot show "context build times" without this metric
The CONTEXT_BUILD_TIME_MS and CONTEXT_TOKEN_COUNT metric keys in OperationalMetricKey are dead code

Fix Required

Add an on_context_built hook to TraceService:

def on_context_built(
    self,
    plan_id: str,
    build_time_ms: float,
    token_count: int,
    *,
    tier: str = "hot",
) -> list[MetricEntry]:
    """Hook: called after context is built for an actor invocation.

    Args:
        plan_id: ULID of the plan.
        build_time_ms: Time taken to build the context in milliseconds.
        token_count: Number of tokens in the built context.
        tier: Context tier (hot/warm/cold).

    Returns:
        List of MetricEntry objects for context metrics.
    """
    return [
        MetricCollector.record(
            OperationalMetricKey.CONTEXT_BUILD_TIME_MS,
            build_time_ms,
            plan_id,
            labels={"tier": tier},
        ),
        MetricCollector.record(
            OperationalMetricKey.CONTEXT_TOKEN_COUNT,
            float(token_count),
            plan_id,
            labels={"tier": tier},
        ),
    ]

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-uat-tester

## Bug Report ### What Was Tested Code-level analysis of `src/cleveragents/application/services/trace_service.py` — specifically the coverage of context-related operational metrics defined in `docs/specification.md` §Observability → Metrics Collection. ### Expected Behavior (from spec) The spec (line ~46117) defines these context-level metrics: | Metric | Type | Description | |---|---|---| | `context.build_duration_ms` | Histogram | Context building time by tier (hot/warm/cold) | | `context.tokens_used` | Gauge | Current token usage in hot context | The `OperationalMetricKey` enum in `metrics.py` defines `CONTEXT_BUILD_TIME_MS` and `CONTEXT_TOKEN_COUNT` for these metrics. `TraceService` has hooks for plan lifecycle (`on_plan_start`), actor invocations (`on_actor_invocation`), and tool executions (`on_tool_execution`), but there is no hook for context building events. ### Actual Behavior `TraceService` has no `on_context_built` hook. The `CONTEXT_BUILD_TIME_MS` and `CONTEXT_TOKEN_COUNT` metric keys exist in `OperationalMetricKey` and `METRIC_DEFINITIONS` but are never emitted by any `TraceService` method: ```python # metrics.py — these keys exist but are never used by TraceService OperationalMetricKey.CONTEXT_BUILD_TIME_MS = "context_build_time_ms" OperationalMetricKey.CONTEXT_TOKEN_COUNT = "context_token_count" ``` The `MetricCollector` class has convenience helpers `context_build_time()` and `context_token_count()` that are also never called. ### Code Location - **File**: `src/cleveragents/application/services/trace_service.py` - **Lines 171–248** (the plan lifecycle hooks section — `on_context_built` is absent) ### Impact - Context build time is never recorded — the ACMS performance cannot be monitored - Context token count is never recorded — hot context size is invisible - The diagnostic dashboard's "Performance summary" (spec line ~46132) cannot show "context build times" without this metric - The `CONTEXT_BUILD_TIME_MS` and `CONTEXT_TOKEN_COUNT` metric keys in `OperationalMetricKey` are dead code ### Fix Required Add an `on_context_built` hook to `TraceService`: ```python def on_context_built( self, plan_id: str, build_time_ms: float, token_count: int, *, tier: str = "hot", ) -> list[MetricEntry]: """Hook: called after context is built for an actor invocation. Args: plan_id: ULID of the plan. build_time_ms: Time taken to build the context in milliseconds. token_count: Number of tokens in the built context. tier: Context tier (hot/warm/cold). Returns: List of MetricEntry objects for context metrics. """ return [ MetricCollector.record( OperationalMetricKey.CONTEXT_BUILD_TIME_MS, build_time_ms, plan_id, labels={"tier": tier}, ), MetricCollector.record( OperationalMetricKey.CONTEXT_TOKEN_COUNT, float(token_count), plan_id, labels={"tier": tier}, ), ] ``` --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-uat-tester

freemo referenced this issue

2026-04-05 22:34:54 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 (Resumed) #3654

freemo referenced this issue

2026-04-05 22:34:59 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 (Resumed) #3654

freemo added the

Points

label

2026-04-06 06:21:43 +00:00

freemo commented

2026-04-06 06:21:57 +00:00

Author

Owner

Issue triaged by project owner:

State: Verified
Priority: Backlog (confirmed)
Story Points: 3 — M — Adding a single hook method to TraceService with two metric recordings. Well-scoped with a clear implementation path provided in the issue.
MoSCoW: Should Have — The spec explicitly defines these metrics in the Observability section. Dead metric keys indicate incomplete observability implementation. Important for monitoring ACMS performance but not blocking core functionality.
Parent Epic: Needs assignment — this relates to observability/tracing. No obvious parent Epic identified; recommend creating one or linking to the ACMS Epic if one exists.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: Backlog (confirmed) - **Story Points**: 3 — M — Adding a single hook method to TraceService with two metric recordings. Well-scoped with a clear implementation path provided in the issue. - **MoSCoW**: Should Have — The spec explicitly defines these metrics in the Observability section. Dead metric keys indicate incomplete observability implementation. Important for monitoring ACMS performance but not blocking core functionality. - **Parent Epic**: Needs assignment — this relates to observability/tracing. No obvious parent Epic identified; recommend creating one or linking to the ACMS Epic if one exists. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner

freemo referenced this issue

2026-04-06 06:22:50 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo referenced this issue

2026-04-06 09:59:07 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo referenced this issue

2026-04-06 12:44:53 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775