feat(context): implement ContextBudgetCalculator for token counting and size estimation #5231

Open
opened 2026-04-09 03:49:22 +00:00 by HAL9000 · 1 comment
Owner

Background

Part of Epic #5167 (Context Budget Enforcement). Implements the ContextBudgetCalculator that estimates token usage and byte sizes for context views, enabling accurate budget enforcement before sending context to LLM providers.

Expected Behavior

  • ContextBudgetCalculator.estimate_tokens(text) returns token count estimate
  • ContextBudgetCalculator.estimate_bytes(text) returns byte size
  • Token estimation uses tiktoken or similar library for accuracy
  • Calculator is provider-aware (different models have different tokenizers)

Subtasks

  • Implement ContextBudgetCalculator with token counting (tiktoken)
  • Add provider-aware tokenizer selection (Claude, OpenAI, etc.)
  • Implement byte size estimation for file content
  • Add tiktoken or equivalent to project dependencies
  • Write Behave unit tests for budget calculations

Definition of Done

  • Token counting works for Claude and OpenAI models
  • Byte size estimation is accurate
  • Provider-aware tokenizer selection works
  • All nox stages pass
  • Coverage >= 97%

Metadata

  • Branch: feat/v3.4.0/acms-budget-calculator
  • Commit Message: feat(context): implement ContextBudgetCalculator for token counting
  • Milestone: v3.4.0
  • Parent Epic: #5167
  • Depends on: #5230 (settings must be correct first)

Automated by CleverAgents Bot
Supervisor: Epic Planning | Agent: epic-planner

## Background Part of Epic #5167 (Context Budget Enforcement). Implements the `ContextBudgetCalculator` that estimates token usage and byte sizes for context views, enabling accurate budget enforcement before sending context to LLM providers. ## Expected Behavior - `ContextBudgetCalculator.estimate_tokens(text)` returns token count estimate - `ContextBudgetCalculator.estimate_bytes(text)` returns byte size - Token estimation uses tiktoken or similar library for accuracy - Calculator is provider-aware (different models have different tokenizers) ## Subtasks - [ ] Implement `ContextBudgetCalculator` with token counting (tiktoken) - [ ] Add provider-aware tokenizer selection (Claude, OpenAI, etc.) - [ ] Implement byte size estimation for file content - [ ] Add `tiktoken` or equivalent to project dependencies - [ ] Write Behave unit tests for budget calculations ## Definition of Done - [ ] Token counting works for Claude and OpenAI models - [ ] Byte size estimation is accurate - [ ] Provider-aware tokenizer selection works - [ ] All nox stages pass - [ ] Coverage >= 97% ## Metadata - **Branch**: `feat/v3.4.0/acms-budget-calculator` - **Commit Message**: `feat(context): implement ContextBudgetCalculator for token counting` - **Milestone**: v3.4.0 - **Parent Epic**: #5167 - **Depends on**: #5230 (settings must be correct first) --- **Automated by CleverAgents Bot** Supervisor: Epic Planning | Agent: epic-planner
HAL9000 added this to the v3.4.0 milestone 2026-04-09 03:49:39 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • Priority: High — ACMS context implementation required for v3.4.0 milestone
  • Story Points: 5 (L) — substantial implementation work
  • MoSCoW: Must Have — context management is core ACMS functionality
  • Assignee: HAL9000 — default assignment to maintain velocity

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: High — ACMS context implementation required for v3.4.0 milestone - **Story Points**: 5 (L) — substantial implementation work - **MoSCoW**: Must Have — context management is core ACMS functionality - **Assignee**: HAL9000 — default assignment to maintain velocity --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner
HAL9000 self-assigned this 2026-04-09 03:53:24 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#5231
No description provided.