cleveragents/cleveragents-core

Fork 3

TDD: Write failing test for #1152 — budget eviction permanently deletes hot-tier fragments #1183

New issue

Closed

opened 2026-03-28 23:24:04 +00:00 by freemo · 5 comments

freemo commented

2026-03-28 23:24:04 +00:00

Owner

Metadata

Commit Message: test: add TDD bug-capture test for #1152 — budget eviction deletes instead of demotes
Branch: tdd/m5-budget-eviction-delete

Background and Context

This is the TDD counterpart to bug #1152. Per the project's Test-Driven Development workflow for bugs (see CONTRIBUTING.md > Bug Fix Workflow), the first step in fixing any bug is to write a test that captures the buggy behavior. The test is tagged with @tdd_bug, @tdd_bug_1152, and @tdd_expected_fail so that it passes CI while the bug is still unfixed.

See #1152 for full bug details. The core issue: when the ACMS budget eviction runs and a hot-tier fragment exceeds the budget, it permanently deletes the fragment instead of demoting it to the warm tier.

Expected Behavior

A new test exists that:

Captures the exact failure — hot-tier fragments are deleted instead of demoted to warm tier during budget eviction.
Is tagged with @tdd_bug, @tdd_bug_1152, and @tdd_expected_fail.
Passes CI via the expected-failure mechanism.
Would fail CI if the bug were fixed without removing the @tdd_expected_fail tag.

Acceptance Criteria

A test is written that captures the bug behavior described in #1152.
The test is tagged with @tdd_bug, @tdd_bug_1152, and @tdd_expected_fail.
The @tdd_expected_fail tag causes the test to pass CI.
The test is specific enough that it will pass normally only when the bug is genuinely fixed.
Tag validation rules pass.
A pull request is opened from tdd/m5-budget-eviction-delete to master, CI passes, and the PR is merged.

Definition of Done

This issue is complete when:

All subtasks below are completed and checked off.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, CI passes, and the PR is merged before this issue is marked done.

Subtasks

Code: Analyze bug #1152 to understand the eviction logic path that deletes instead of demotes.
Code: Determine the appropriate test type and file location.
Tests (Behave): Write a Behave scenario that adds a fragment to the hot tier, triggers budget eviction, and asserts the fragment was demoted to warm (not deleted). Tag with @tdd_bug, @tdd_bug_1152, @tdd_expected_fail.
Tests (Robot): If integration-level behavior is involved, add a Robot test with equivalent tags.
Docs: Add comment explaining this test captures bug #1152.
Quality: Verify CI passes with the tagged test.
Quality: Verify tag validation rules pass.
Quality: Verify coverage >=97% via nox -s coverage_report.
Quality: Run nox (all default sessions), fix any errors.

## Metadata - **Commit Message**: `test: add TDD bug-capture test for #1152 — budget eviction deletes instead of demotes` - **Branch**: `tdd/m5-budget-eviction-delete` ## Background and Context This is the TDD counterpart to bug #1152. Per the project's Test-Driven Development workflow for bugs (see `CONTRIBUTING.md` > Bug Fix Workflow), the first step in fixing any bug is to write a test that captures the buggy behavior. The test is tagged with `@tdd_bug`, `@tdd_bug_1152`, and `@tdd_expected_fail` so that it passes CI while the bug is still unfixed. See #1152 for full bug details. The core issue: when the ACMS budget eviction runs and a hot-tier fragment exceeds the budget, it permanently deletes the fragment instead of demoting it to the warm tier. ## Expected Behavior A new test exists that: 1. Captures the exact failure — hot-tier fragments are deleted instead of demoted to warm tier during budget eviction. 2. Is tagged with `@tdd_bug`, `@tdd_bug_1152`, and `@tdd_expected_fail`. 3. Passes CI via the expected-failure mechanism. 4. Would fail CI if the bug were fixed without removing the `@tdd_expected_fail` tag. ## Acceptance Criteria - [x] A test is written that captures the bug behavior described in #1152. - [x] The test is tagged with `@tdd_bug`, `@tdd_bug_1152`, and `@tdd_expected_fail`. - [x] The `@tdd_expected_fail` tag causes the test to pass CI. - [x] The test is specific enough that it will pass normally only when the bug is genuinely fixed. - [x] Tag validation rules pass. - [ ] A pull request is opened from `tdd/m5-budget-eviction-delete` to `master`, CI passes, and the PR is merged. ## Definition of Done This issue is complete when: - All subtasks below are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, CI passes, and the PR is **merged** before this issue is marked done. ## Subtasks - [x] Code: Analyze bug #1152 to understand the eviction logic path that deletes instead of demotes. - [x] Code: Determine the appropriate test type and file location. - [x] Tests (Behave): Write a Behave scenario that adds a fragment to the hot tier, triggers budget eviction, and asserts the fragment was demoted to warm (not deleted). Tag with `@tdd_bug`, `@tdd_bug_1152`, `@tdd_expected_fail`. - [x] Tests (Robot): If integration-level behavior is involved, add a Robot test with equivalent tags. - [x] Docs: Add comment explaining this test captures bug #1152. - [x] Quality: Verify CI passes with the tagged test. - [x] Quality: Verify tag validation rules pass. - [x] Quality: Verify coverage >=97% via `nox -s coverage_report`. - [x] Quality: Run `nox` (all default sessions), fix any errors.

freemo added the

labels

2026-03-28 23:24:13 +00:00

hamza.khyari was assigned by freemo

2026-03-28 23:24:13 +00:00

freemo added this to the v3.4.0 milestone

2026-03-28 23:24:13 +00:00

freemo referenced this issue

2026-03-28 23:24:18 +00:00

design(acms): budget eviction permanently deletes hot-tier fragments instead of demoting to warm #1152

brent.edwards added

and removed

labels

2026-03-31 03:47:57 +00:00

brent.edwards commented

2026-03-31 03:48:29 +00:00

Member

Implementation Notes — Subtask 1: Analyze bug #1152

Bug Analysis

Root cause: In src/cleveragents/application/services/tier_runtime.py, TierRuntimeMixin._enforce_hot_budget() (lines ~180–214) permanently deletes hot-tier fragments via del self._hot[oldest_id] when the token budget is exceeded. The evicted fragments are not demoted to the warm tier — they are destroyed.

Code path:

ContextTierService.store() stores a fragment in _hot dict
store() calls self._enforce_hot_budget() (from TierRuntimeMixin)
_enforce_hot_budget() sums all hot-tier tokens; if over budget, enters eviction loop
Inside the loop: del self._hot[oldest_id] — permanent deletion, no demotion
Emits TIER_EVICTED with to_tier=None (no destination)

Similarly affected: evict_lru() in context_tiers.py also permanently deletes via del store[fid].

Specification reference: Lines 25704–25713 describe "Hot context archived to warm. Warm context ages to cold based on retention policy" — implying a downward lifecycle flow, not permanent destruction.

Tag Convention Decision

The issue body requests @tdd_bug/@tdd_bug_1152 tags, but CONTRIBUTING.md (§TDD Issue Test Tags) and the tag validation infrastructure in features/environment.py (validate_tdd_tags()) require @tdd_issue/@tdd_issue_<N>. Using the CONTRIBUTING.md-mandated tags: @tdd_issue @tdd_issue_1152 @tdd_expected_fail. Also adding @mock_only since no external resources are needed.

## Implementation Notes — Subtask 1: Analyze bug #1152 ### Bug Analysis **Root cause:** In `src/cleveragents/application/services/tier_runtime.py`, `TierRuntimeMixin._enforce_hot_budget()` (lines ~180–214) permanently deletes hot-tier fragments via `del self._hot[oldest_id]` when the token budget is exceeded. The evicted fragments are not demoted to the warm tier — they are destroyed. **Code path:** 1. `ContextTierService.store()` stores a fragment in `_hot` dict 2. `store()` calls `self._enforce_hot_budget()` (from `TierRuntimeMixin`) 3. `_enforce_hot_budget()` sums all hot-tier tokens; if over budget, enters eviction loop 4. Inside the loop: `del self._hot[oldest_id]` — permanent deletion, no demotion 5. Emits `TIER_EVICTED` with `to_tier=None` (no destination) **Similarly affected:** `evict_lru()` in `context_tiers.py` also permanently deletes via `del store[fid]`. **Specification reference:** Lines 25704–25713 describe "Hot context archived to warm. Warm context ages to cold based on retention policy" — implying a downward lifecycle flow, not permanent destruction. ### Tag Convention Decision The issue body requests `@tdd_bug`/`@tdd_bug_1152` tags, but CONTRIBUTING.md (§TDD Issue Test Tags) and the tag validation infrastructure in `features/environment.py` (`validate_tdd_tags()`) require `@tdd_issue`/`@tdd_issue_<N>`. Using the CONTRIBUTING.md-mandated tags: `@tdd_issue @tdd_issue_1152 @tdd_expected_fail`. Also adding `@mock_only` since no external resources are needed.

brent.edwards commented

2026-03-31 04:53:40 +00:00

Member

Implementation Notes — Subtasks 2–9: Test implementation & quality gates

Files created

features/tdd_budget_eviction_deletes_not_demotes.feature — Behave BDD feature with two scenarios:
- Scenario 1: Budget-evicted hot-tier fragment is demoted to warm tier instead of deleted. Sets up a 100-token hot-tier budget, fills it with two 50-token fragments, stores a third to trigger eviction, and asserts the evicted fragment is in the warm tier (not destroyed).
- Scenario 2: evict_lru() also demotes to warm tier instead of deleting. Same setup, but calls evict_lru(HOT, 1) directly and asserts the result.
- Tags: @tdd_expected_fail @tdd_issue @tdd_issue_1152 @mock_only
features/steps/tdd_budget_eviction_deletes_not_demotes_steps.py — Step definitions with descriptive assertion messages referencing the bug number, the code path (_enforce_hot_budget / evict_lru), and the spec section.
robot/tdd_budget_eviction_deletes_not_demotes.robot — Robot Framework integration test with two test cases mirroring the Behave scenarios. Tags: tdd_expected_fail tdd_issue tdd_issue_1152.
robot/helper_tdd_budget_eviction_deletes_not_demotes.py — Python helper script for Robot tests with budget-eviction-demotes and evict-lru-demotes subcommands.

Design decisions

Used @tdd_issue/@tdd_issue_1152 tags (per CONTRIBUTING.md §TDD Issue Test Tags) rather than @tdd_bug/@tdd_bug_1152 (mentioned in issue body), because the tag validation in features/environment.py (validate_tdd_tags()) enforces @tdd_issue + @tdd_issue_<N> as prerequisites for @tdd_expected_fail.
Added @mock_only tag on the Behave feature since these tests exercise in-memory tier stores with no external dependencies.
Test assertions are specific: they check both that the fragment is NOT in the hot tier AND that it IS in the warm tier with correct tier metadata. This ensures the test passes only when the bug is genuinely fixed (demotion implemented, not just deletion removed).

Quality gate results

All 11 nox sessions pass:

Gate	Result
`lint`	✅ Pass
`format`	✅ Pass
`typecheck`	✅ Pass
`security_scan`	✅ Pass
`dead_code`	✅ Pass
`unit_tests`	✅ Pass (509 features, 12987 scenarios, 0 failures)
`integration_tests`	✅ Pass
`docs`	✅ Pass
`build`	✅ Pass
`benchmark`	✅ Pass
`coverage_report`	✅ Pass (97%)

TDD expected-fail verification

Both Behave scenarios correctly fail their assertions (proving the bug exists):

Scenario 1: _enforce_hot_budget() permanently deletes frag-evict-oldest — it's not found in service._warm.
Scenario 2: evict_lru() permanently deletes frag-evict-oldest — it's not found in service._warm.

The @tdd_expected_fail tag inverts these failures to passes in CI. When bug #1152 is fixed (by changing _enforce_hot_budget() and evict_lru() to demote instead of delete), the tag must be removed.

## Implementation Notes — Subtasks 2–9: Test implementation & quality gates ### Files created 1. **`features/tdd_budget_eviction_deletes_not_demotes.feature`** — Behave BDD feature with two scenarios: - *Scenario 1*: Budget-evicted hot-tier fragment is demoted to warm tier instead of deleted. Sets up a 100-token hot-tier budget, fills it with two 50-token fragments, stores a third to trigger eviction, and asserts the evicted fragment is in the warm tier (not destroyed). - *Scenario 2*: `evict_lru()` also demotes to warm tier instead of deleting. Same setup, but calls `evict_lru(HOT, 1)` directly and asserts the result. - Tags: `@tdd_expected_fail @tdd_issue @tdd_issue_1152 @mock_only` 2. **`features/steps/tdd_budget_eviction_deletes_not_demotes_steps.py`** — Step definitions with descriptive assertion messages referencing the bug number, the code path (`_enforce_hot_budget` / `evict_lru`), and the spec section. 3. **`robot/tdd_budget_eviction_deletes_not_demotes.robot`** — Robot Framework integration test with two test cases mirroring the Behave scenarios. Tags: `tdd_expected_fail tdd_issue tdd_issue_1152`. 4. **`robot/helper_tdd_budget_eviction_deletes_not_demotes.py`** — Python helper script for Robot tests with `budget-eviction-demotes` and `evict-lru-demotes` subcommands. ### Design decisions - Used `@tdd_issue`/`@tdd_issue_1152` tags (per CONTRIBUTING.md §TDD Issue Test Tags) rather than `@tdd_bug`/`@tdd_bug_1152` (mentioned in issue body), because the tag validation in `features/environment.py` (`validate_tdd_tags()`) enforces `@tdd_issue` + `@tdd_issue_<N>` as prerequisites for `@tdd_expected_fail`. - Added `@mock_only` tag on the Behave feature since these tests exercise in-memory tier stores with no external dependencies. - Test assertions are specific: they check both that the fragment is NOT in the hot tier AND that it IS in the warm tier with correct tier metadata. This ensures the test passes only when the bug is genuinely fixed (demotion implemented, not just deletion removed). ### Quality gate results All 11 nox sessions pass: | Gate | Result | |------|--------| | `lint` | ✅ Pass | | `format` | ✅ Pass | | `typecheck` | ✅ Pass | | `security_scan` | ✅ Pass | | `dead_code` | ✅ Pass | | `unit_tests` | ✅ Pass (509 features, 12987 scenarios, 0 failures) | | `integration_tests` | ✅ Pass | | `docs` | ✅ Pass | | `build` | ✅ Pass | | `benchmark` | ✅ Pass | | `coverage_report` | ✅ Pass (97%) | ### TDD expected-fail verification Both Behave scenarios correctly fail their assertions (proving the bug exists): - Scenario 1: `_enforce_hot_budget()` permanently deletes `frag-evict-oldest` — it's not found in `service._warm`. - Scenario 2: `evict_lru()` permanently deletes `frag-evict-oldest` — it's not found in `service._warm`. The `@tdd_expected_fail` tag inverts these failures to passes in CI. When bug #1152 is fixed (by changing `_enforce_hot_budget()` and `evict_lru()` to demote instead of delete), the tag must be removed.

brent.edwards referenced this issue from a commit

2026-03-31 04:54:12 +00:00

test: add TDD bug-capture test for #1152 — budget eviction deletes instead of demotes

brent.edwards referenced this issue from a pull request that will close it,

2026-03-31 04:54:24 +00:00

test: add TDD bug-capture test for #1152 — budget eviction deletes instead of demotes #1218

brent.edwards added

and removed

labels

2026-03-31 04:55:02 +00:00

hamza.khyari was unassigned by freemo

2026-04-02 06:13:52 +00:00

freemo self-assigned this

2026-04-02 06:13:52 +00:00

freemo referenced this issue from a commit

2026-04-02 06:53:39 +00:00

test: add TDD bug-capture test for #1152 — budget eviction deletes instead of demotes

freemo commented

2026-04-02 08:23:27 +00:00

Author

Owner

PR #1218 reviewed — changes requested.

The test design is excellent and correctly captures bug #1152 with both Behave BDD and Robot Framework tests. However, both the step file and Robot helper contain # type: ignore[arg-type] suppressions which are forbidden by CONTRIBUTING.md.

Required fix: Replace the dict[str, object] → **kwargs pattern in _make_eviction_fragment() and _make_fragment() with direct TieredFragment(...) construction using properly-typed keyword arguments. This eliminates the need for type suppressions entirely.

See the inline review comments on PR #1218 for the specific code fix.

**PR #1218 reviewed — changes requested.** The test design is excellent and correctly captures bug #1152 with both Behave BDD and Robot Framework tests. However, both the step file and Robot helper contain `# type: ignore[arg-type]` suppressions which are forbidden by CONTRIBUTING.md. **Required fix:** Replace the `dict[str, object]` → `**kwargs` pattern in `_make_eviction_fragment()` and `_make_fragment()` with direct `TieredFragment(...)` construction using properly-typed keyword arguments. This eliminates the need for type suppressions entirely. See the inline review comments on PR #1218 for the specific code fix.

freemo referenced this issue from a commit

2026-04-02 08:38:54 +00:00

fix(test): remove forbidden type:ignore by constructing TieredFragment directly

freemo referenced this issue from a commit

2026-04-02 08:39:54 +00:00

fix(test): remove forbidden type:ignore from Robot helper

freemo commented

2026-04-02 08:41:38 +00:00

Author

Owner

PR #1218 reviewed, approved, and merged (squash).

Fix applied during review: Removed 2 forbidden # type: ignore[arg-type] suppressions from features/steps/tdd_budget_eviction_deletes_not_demotes_steps.py and robot/helper_tdd_budget_eviction_deletes_not_demotes.py. The fix replaces dict[str, object] unpacking with direct TieredFragment keyword construction and post-construction assignment for the optional last_accessed field.

The TDD bug-capture tests for #1152 are now on master and will correctly fail (inverted by @tdd_expected_fail) until the underlying eviction bug is fixed.

PR #1218 reviewed, approved, and merged (squash). **Fix applied during review:** Removed 2 forbidden `# type: ignore[arg-type]` suppressions from `features/steps/tdd_budget_eviction_deletes_not_demotes_steps.py` and `robot/helper_tdd_budget_eviction_deletes_not_demotes.py`. The fix replaces `dict[str, object]` unpacking with direct `TieredFragment` keyword construction and post-construction assignment for the optional `last_accessed` field. The TDD bug-capture tests for #1152 are now on `master` and will correctly fail (inverted by `@tdd_expected_fail`) until the underlying eviction bug is fixed.

freemo closed this issue

2026-04-02 16:50:50 +00:00

freemo referenced this issue from a commit

2026-04-02 16:50:52 +00:00

test: add TDD bug-capture test for #1152 — budget eviction deletes instead of demotes

freemo commented

2026-04-02 16:50:59 +00:00

Author

Owner

PR #1218 reviewed, approved, and merged (squash). The TDD bug-capture tests for #1152 are now on master.

PR #1218 reviewed, approved, and merged (squash). The TDD bug-capture tests for #1152 are now on `master`.

freemo added