UAT: `input_tokens` and `output_tokens` incorrectly redacted as sensitive data in session JSON output #3866

New issue

Open

opened 2026-04-06 07:02:24 +00:00 by freemo · 0 comments

freemo commented

2026-04-06 07:02:24 +00:00

Owner

Metadata

Branch: fix/redaction-false-positive-input-output-tokens
Commit Message: fix(redaction): add input_tokens and output_tokens to _FALSE_POSITIVE_KEYS
Milestone: (backlog — see note below)
Parent Epic: #400

Background

During UAT testing of session management, input_tokens and output_tokens fields are incorrectly classified as sensitive data and redacted with ***REDACTED*** in JSON output from agents session show --format json.

What Was Tested

agents session show --format json with a session that has token usage data.

Expected Behavior

Per the specification, the JSON output should include actual integer values for input_tokens and output_tokens in the token_usage block:

{
  "token_usage": {
    "input_tokens": 1000,
    "output_tokens": 500,
    "estimated_cost": "$0.0184"
  }
}

Actual Behavior

input_tokens and output_tokens are redacted:

{
  "token_usage": {
    "input_tokens": "***REDACTED***",
    "output_tokens": "***REDACTED***",
    "estimated_cost": "$0.0184"
  }
}

Root Cause

In src/cleveragents/shared/redaction.py, the _SENSITIVE_SUBSTRINGS set contains "token". The is_sensitive_key() function checks if any sensitive substring appears in the key name. Since input_tokens and output_tokens both contain "token", they are incorrectly classified as sensitive.

The _FALSE_POSITIVE_KEYS set (which exempts legitimate token-related keys) includes token_usage, max_tokens, total_tokens, prompt_tokens, completion_tokens, token_estimate, hot_max_tokens, and summary_max_tokens — but is missing input_tokens and output_tokens.

Code location: src/cleveragents/shared/redaction.py, _FALSE_POSITIVE_KEYS set (around line 44).

Fix

Add input_tokens and output_tokens to the _FALSE_POSITIVE_KEYS set:

_FALSE_POSITIVE_KEYS: set[str] = {
    "token_count",
    "token_limit",
    "token_usage",
    "max_tokens",
    "total_tokens",
    "prompt_tokens",
    "completion_tokens",
    "token_estimate",
    "hot_max_tokens",
    "summary_max_tokens",
    "input_tokens",    # ADD THIS
    "output_tokens",   # ADD THIS
    "auth_method",
    "auth_type",
    "auth_enabled",
}

Steps to Reproduce

Create a session: agents session create
Send a message: agents session tell --session <ID> "Hello"
Show session in JSON format: agents session show --format json <ID>
Observe input_tokens and output_tokens are ***REDACTED***

Impact

This breaks programmatic consumers of the agents session show --format json command that need to read token usage data. The estimated_cost field is correctly NOT redacted, making the redaction inconsistent.

Subtasks

Write a failing Behave scenario in features/ that demonstrates input_tokens and output_tokens are incorrectly redacted
Add input_tokens and output_tokens to _FALSE_POSITIVE_KEYS in src/cleveragents/shared/redaction.py
Verify the Behave scenario now passes
Run nox -e lint and nox -e typecheck to confirm no regressions
Run nox -e unit_tests to confirm all tests pass
Run nox -e coverage_report to confirm coverage >= 97%
Update PR description with Closes #<this issue>

Definition of Done

input_tokens and output_tokens are present in _FALSE_POSITIVE_KEYS
agents session show --format json returns actual integer values for input_tokens and output_tokens
estimated_cost continues to be correctly NOT redacted
A Behave scenario covers the false-positive redaction case
All nox stages pass
Coverage >= 97%
PR merged and linked to this issue

Backlog note: This issue was discovered during autonomous operation
on milestone v3.6.0. It does not block milestone completion and has been
placed in the backlog for human review and future milestone assignment.

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-new-issue-creator

## Metadata - **Branch**: `fix/redaction-false-positive-input-output-tokens` - **Commit Message**: `fix(redaction): add input_tokens and output_tokens to _FALSE_POSITIVE_KEYS` - **Milestone**: *(backlog — see note below)* - **Parent Epic**: #400 ## Background During UAT testing of session management, `input_tokens` and `output_tokens` fields are incorrectly classified as sensitive data and redacted with `***REDACTED***` in JSON output from `agents session show --format json`. ## What Was Tested `agents session show --format json` with a session that has token usage data. ## Expected Behavior Per the specification, the JSON output should include actual integer values for `input_tokens` and `output_tokens` in the `token_usage` block: ```json { "token_usage": { "input_tokens": 1000, "output_tokens": 500, "estimated_cost": "$0.0184" } } ``` ## Actual Behavior `input_tokens` and `output_tokens` are redacted: ```json { "token_usage": { "input_tokens": "***REDACTED***", "output_tokens": "***REDACTED***", "estimated_cost": "$0.0184" } } ``` ## Root Cause In `src/cleveragents/shared/redaction.py`, the `_SENSITIVE_SUBSTRINGS` set contains `"token"`. The `is_sensitive_key()` function checks if any sensitive substring appears in the key name. Since `input_tokens` and `output_tokens` both contain `"token"`, they are incorrectly classified as sensitive. The `_FALSE_POSITIVE_KEYS` set (which exempts legitimate token-related keys) includes `token_usage`, `max_tokens`, `total_tokens`, `prompt_tokens`, `completion_tokens`, `token_estimate`, `hot_max_tokens`, and `summary_max_tokens` — but is **missing** `input_tokens` and `output_tokens`. **Code location**: `src/cleveragents/shared/redaction.py`, `_FALSE_POSITIVE_KEYS` set (around line 44). ## Fix Add `input_tokens` and `output_tokens` to the `_FALSE_POSITIVE_KEYS` set: ```python _FALSE_POSITIVE_KEYS: set[str] = { "token_count", "token_limit", "token_usage", "max_tokens", "total_tokens", "prompt_tokens", "completion_tokens", "token_estimate", "hot_max_tokens", "summary_max_tokens", "input_tokens", # ADD THIS "output_tokens", # ADD THIS "auth_method", "auth_type", "auth_enabled", } ``` ## Steps to Reproduce 1. Create a session: `agents session create` 2. Send a message: `agents session tell --session <ID> "Hello"` 3. Show session in JSON format: `agents session show --format json <ID>` 4. Observe `input_tokens` and `output_tokens` are `***REDACTED***` ## Impact This breaks programmatic consumers of the `agents session show --format json` command that need to read token usage data. The `estimated_cost` field is correctly NOT redacted, making the redaction inconsistent. ## Subtasks - [ ] Write a failing Behave scenario in `features/` that demonstrates `input_tokens` and `output_tokens` are incorrectly redacted - [ ] Add `input_tokens` and `output_tokens` to `_FALSE_POSITIVE_KEYS` in `src/cleveragents/shared/redaction.py` - [ ] Verify the Behave scenario now passes - [ ] Run `nox -e lint` and `nox -e typecheck` to confirm no regressions - [ ] Run `nox -e unit_tests` to confirm all tests pass - [ ] Run `nox -e coverage_report` to confirm coverage >= 97% - [ ] Update PR description with `Closes #<this issue>` ## Definition of Done - [ ] `input_tokens` and `output_tokens` are present in `_FALSE_POSITIVE_KEYS` - [ ] `agents session show --format json` returns actual integer values for `input_tokens` and `output_tokens` - [ ] `estimated_cost` continues to be correctly NOT redacted - [ ] A Behave scenario covers the false-positive redaction case - [ ] All nox stages pass - [ ] Coverage >= 97% - [ ] PR merged and linked to this issue > **Backlog note:** This issue was discovered during autonomous operation > on milestone v3.6.0. It does not block milestone completion and has been > placed in the backlog for human review and future milestone assignment. --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-new-issue-creator

freemo added a new dependency

2026-04-06 07:02:34 +00:00

#400 Epic: Post-MVP Security

freemo added the

Points

label

2026-04-06 07:03:32 +00:00

freemo referenced this issue

2026-04-06 07:13:52 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo referenced this issue

2026-04-06 07:21:26 +00:00