UAT: `session show` JSON/YAML output incorrectly redacts `input_tokens` and `output_tokens` as secrets #1427

New issue

Open

opened 2026-04-02 17:50:03 +00:00 by freemo · 0 comments

freemo commented

2026-04-02 17:50:03 +00:00

Owner

Metadata

Branch: bugfix/m6-session-show-redacts-input-output-tokens
Commit Message: fix(shared): add input_tokens and output_tokens to redaction false-positive keys
Milestone: v3.5.0
Parent Epic: (needs manual linking — no open CLI Session Management Epic identified at creation time; see related session bugs #1347, #1349, #1425)

Bug Report

Feature Area: Session Management — session show CLI output / shared/redaction.py

What Was Tested

The agents session show <SESSION_ID> --format json and --format yaml command output.

Expected Behavior (from spec)

According to docs/specification.md (session show JSON output section), the token_usage object should contain numeric values:

"token_usage": {
  "input_tokens": 3420,
  "output_tokens": 1185,
  "estimated_cost": "$0.0184"
}

Actual Behavior

The input_tokens and output_tokens fields are replaced with ***REDACTED*** in JSON and YAML output:

"token_usage": {
  "input_tokens": "***REDACTED***",
  "output_tokens": "***REDACTED***",
  "estimated_cost": 0.0
}

Root Cause

In src/cleveragents/shared/redaction.py, the _SENSITIVE_SUBSTRINGS set includes "token". The is_sensitive_key() function checks if any sensitive substring is present in the key name. The keys input_tokens and output_tokens both contain "token" and are therefore incorrectly flagged as sensitive.

The _FALSE_POSITIVE_KEYS set includes token_usage, total_tokens, prompt_tokens, completion_tokens, token_count, token_limit, token_estimate, max_tokens — but NOT input_tokens or output_tokens.

Steps to Reproduce

Initialize the database: CLEVERAGENTS_DATA_DIR=/tmp/test uv run agents db upgrade
Create a session: CLEVERAGENTS_DATA_DIR=/tmp/test uv run agents session create
Send a message: CLEVERAGENTS_DATA_DIR=/tmp/test uv run agents session tell --session <ID> "Hello"
Show session in JSON format: CLEVERAGENTS_DATA_DIR=/tmp/test uv run agents session show <ID> --format json
Observe input_tokens and output_tokens are "***REDACTED***" instead of numeric values

Code Location

src/cleveragents/shared/redaction.py — _FALSE_POSITIVE_KEYS set (lines ~43–55)
src/cleveragents/cli/commands/session.py — show command uses format_output() which applies redaction

Subtasks

Add "input_tokens" and "output_tokens" to the _FALSE_POSITIVE_KEYS set in src/cleveragents/shared/redaction.py
Verify no other token_usage-adjacent keys are missing from _FALSE_POSITIVE_KEYS (e.g., audit for any other false positives introduced by the "token" substring match)
Write Behave unit test scenario: session show --format json produces numeric input_tokens and output_tokens values (not ***REDACTED***)
Write Behave unit test scenario: session show --format yaml produces numeric input_tokens and output_tokens values
Write Behave unit test for is_sensitive_key("input_tokens") returning False
Write Behave unit test for is_sensitive_key("output_tokens") returning False
Run nox -s unit_tests and confirm all tests pass
Run nox -s coverage_report and confirm coverage >= 97%
Run nox (all default sessions) and fix any errors

Definition of Done

input_tokens and output_tokens are present in _FALSE_POSITIVE_KEYS in src/cleveragents/shared/redaction.py
agents session show <ID> --format json outputs numeric values for input_tokens and output_tokens (not ***REDACTED***)
agents session show <ID> --format yaml outputs numeric values for input_tokens and output_tokens (not ***REDACTED***)
is_sensitive_key("input_tokens") returns False
is_sensitive_key("output_tokens") returns False
All new Behave scenarios pass without @expected_fail
A PR is opened from the branch to master, reviewed, and merged
All nox stages pass
Coverage >= 97%

## Metadata - **Branch**: `bugfix/m6-session-show-redacts-input-output-tokens` - **Commit Message**: `fix(shared): add input_tokens and output_tokens to redaction false-positive keys` - **Milestone**: v3.5.0 - **Parent Epic**: *(needs manual linking — no open CLI Session Management Epic identified at creation time; see related session bugs #1347, #1349, #1425)* ## Bug Report **Feature Area:** Session Management — `session show` CLI output / `shared/redaction.py` ### What Was Tested The `agents session show <SESSION_ID> --format json` and `--format yaml` command output. ### Expected Behavior (from spec) According to `docs/specification.md` (session show JSON output section), the `token_usage` object should contain numeric values: ```json "token_usage": { "input_tokens": 3420, "output_tokens": 1185, "estimated_cost": "$0.0184" } ``` ### Actual Behavior The `input_tokens` and `output_tokens` fields are replaced with `***REDACTED***` in JSON and YAML output: ```json "token_usage": { "input_tokens": "***REDACTED***", "output_tokens": "***REDACTED***", "estimated_cost": 0.0 } ``` ### Root Cause In `src/cleveragents/shared/redaction.py`, the `_SENSITIVE_SUBSTRINGS` set includes `"token"`. The `is_sensitive_key()` function checks if any sensitive substring is present in the key name. The keys `input_tokens` and `output_tokens` both contain `"token"` and are therefore incorrectly flagged as sensitive. The `_FALSE_POSITIVE_KEYS` set includes `token_usage`, `total_tokens`, `prompt_tokens`, `completion_tokens`, `token_count`, `token_limit`, `token_estimate`, `max_tokens` — but NOT `input_tokens` or `output_tokens`. ### Steps to Reproduce 1. Initialize the database: `CLEVERAGENTS_DATA_DIR=/tmp/test uv run agents db upgrade` 2. Create a session: `CLEVERAGENTS_DATA_DIR=/tmp/test uv run agents session create` 3. Send a message: `CLEVERAGENTS_DATA_DIR=/tmp/test uv run agents session tell --session <ID> "Hello"` 4. Show session in JSON format: `CLEVERAGENTS_DATA_DIR=/tmp/test uv run agents session show <ID> --format json` 5. Observe `input_tokens` and `output_tokens` are `"***REDACTED***"` instead of numeric values ### Code Location - `src/cleveragents/shared/redaction.py` — `_FALSE_POSITIVE_KEYS` set (lines ~43–55) - `src/cleveragents/cli/commands/session.py` — `show` command uses `format_output()` which applies redaction ## Subtasks - [ ] Add `"input_tokens"` and `"output_tokens"` to the `_FALSE_POSITIVE_KEYS` set in `src/cleveragents/shared/redaction.py` - [ ] Verify no other `token_usage`-adjacent keys are missing from `_FALSE_POSITIVE_KEYS` (e.g., audit for any other false positives introduced by the `"token"` substring match) - [ ] Write Behave unit test scenario: `session show --format json` produces numeric `input_tokens` and `output_tokens` values (not `***REDACTED***`) - [ ] Write Behave unit test scenario: `session show --format yaml` produces numeric `input_tokens` and `output_tokens` values - [ ] Write Behave unit test for `is_sensitive_key("input_tokens")` returning `False` - [ ] Write Behave unit test for `is_sensitive_key("output_tokens")` returning `False` - [ ] Run `nox -s unit_tests` and confirm all tests pass - [ ] Run `nox -s coverage_report` and confirm coverage >= 97% - [ ] Run `nox` (all default sessions) and fix any errors ## Definition of Done - [ ] `input_tokens` and `output_tokens` are present in `_FALSE_POSITIVE_KEYS` in `src/cleveragents/shared/redaction.py` - [ ] `agents session show <ID> --format json` outputs numeric values for `input_tokens` and `output_tokens` (not `***REDACTED***`) - [ ] `agents session show <ID> --format yaml` outputs numeric values for `input_tokens` and `output_tokens` (not `***REDACTED***`) - [ ] `is_sensitive_key("input_tokens")` returns `False` - [ ] `is_sensitive_key("output_tokens")` returns `False` - [ ] All new Behave scenarios pass without `@expected_fail` - [ ] A PR is opened from the branch to `master`, reviewed, and merged - All nox stages pass - Coverage >= 97%

Rows
Columns

UAT: session show JSON/YAML output incorrectly redacts input_tokens and output_tokens as secrets #1427