test(acms): TDD failing tests for context tier runtime logic (bug #821) (#1058)
CI / lint (push) Successful in 13s
CI / e2e_tests (push) Has started running
CI / quality (push) Successful in 23s
CI / benchmark-publish (push) Has started running
CI / typecheck (push) Successful in 43s
CI / benchmark-regression (push) Has been skipped
CI / build (push) Successful in 22s
CI / security (push) Successful in 59s
CI / integration_tests (push) Successful in 2m49s
CI / unit_tests (push) Successful in 4m1s
CI / docker (push) Successful in 1m26s
CI / coverage (push) Successful in 9m14s

## Summary

TDD expected-fail tests proving bug #821 exists: `ContextTierService` has data models for hot/warm/cold tiers but **no runtime logic** for automatic promotion, demotion, or eviction.

- **Promotion on access**: Accessing a cold-tier fragment repeatedly via `get()` does NOT auto-promote it — `get()` updates `access_count`/`last_accessed` but never calls `promote()`
- **Demotion on staleness**: No staleness enforcement method exists — tried `enforce_staleness()`, `apply_tier_policy()`, `tick()`, etc. — none are implemented
- **Eviction on budget overflow**: `store()` does NOT enforce `TierBudget.max_tokens_hot` — the hot tier grows without bound

### Files Added
| File | Purpose |
|------|---------|
| `features/tdd_context_tier_runtime.feature` | 3 Behave scenarios tagged `@tdd_expected_fail @tdd_bug @tdd_bug_821 @mock_only` |
| `features/steps/tdd_context_tier_runtime_steps.py` | Type-annotated step definitions exercising real `ContextTierService` |
| `robot/tdd_context_tier_runtime.robot` | 3 Robot Framework integration tests tagged `tdd_expected_fail` |
| `robot/helper_tdd_context_tier_runtime.py` | Helper script for Robot tests with 3 subcommands |

### Verification
- `nox -s lint` — passed
- `nox -s typecheck` — passed (0 errors)
- `nox -s unit_tests -- features/tdd_context_tier_runtime.feature` — **3 scenarios passed** (all assertions fail as expected, `@tdd_expected_fail` inverts to CI pass)

ISSUES CLOSED: #840

Reviewed-on: #1058
Co-authored-by: Brent Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent Edwards <brent.edwards@cleverthis.com>
This commit was merged in pull request #1058.
This commit is contained in:
2026-03-19 22:24:41 +00:00
committed by Forgejo
parent 60af2cae0c
commit 202c9bfe75
4 changed files with 626 additions and 0 deletions
@@ -0,0 +1,310 @@
"""Step definitions for TDD Bug #821 — context tier runtime logic.
These steps exercise ``ContextTierService`` and verify that it automatically
promotes fragments on repeated access, demotes stale fragments, and evicts
fragments when the hot tier budget overflows.
On ``master`` (before the fix), ``ContextTierService`` has data models and
manual ``promote()``/``demote()``/``evict_lru()`` methods but NO automatic
runtime logic:
- ``get()`` updates ``access_count`` and ``last_accessed`` but never
auto-promotes a heavily accessed cold/warm fragment.
- There is no staleness enforcement: no method inspects ``last_accessed``
timestamps and auto-demotes stale hot fragments.
- ``store()`` does not enforce ``TierBudget.max_tokens_hot``: storing
beyond the budget does not trigger automatic LRU eviction.
The assertions in these steps will **fail** until the bug is fixed,
proving the bug exists. The ``@tdd_expected_fail`` tag inverts the
result so CI passes.
"""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from behave import given, then, when
from behave.runner import Context
from cleveragents.application.services.context_tiers import ContextTierService
from cleveragents.domain.models.acms.tiers import (
ContextTier,
TierBudget,
TieredFragment,
)
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
_FRAGMENT_ID: str = "frag-tier-runtime-001"
_STALE_FRAGMENT_ID: str = "frag-stale-hot-001"
def _make_fragment(
fragment_id: str,
tier: ContextTier,
token_count: int = 50,
content: str = "test content",
last_accessed: datetime | None = None,
) -> TieredFragment:
"""Create a ``TieredFragment`` with the given properties."""
kwargs: dict[str, object] = {
"fragment_id": fragment_id,
"content": content,
"tier": tier,
"token_count": token_count,
"project_name": "test-project",
}
if last_accessed is not None:
kwargs["last_accessed"] = last_accessed
return TieredFragment(**kwargs) # type: ignore[arg-type]
# ---------------------------------------------------------------------------
# Scenario 1: Promotion on repeated access
# ---------------------------------------------------------------------------
@given("a context tier service with default budget")
def step_service_default_budget(context: Context) -> None:
"""Create a ContextTierService with the default TierBudget."""
context.tier_service = ContextTierService(settings=None)
@given("a fragment stored in the cold tier")
def step_fragment_in_cold(context: Context) -> None:
"""Store a fragment in the cold tier."""
frag: TieredFragment = _make_fragment(
fragment_id=_FRAGMENT_ID,
tier=ContextTier.COLD,
token_count=50,
)
service: ContextTierService = context.tier_service
service.store(frag)
# Verify it is actually in cold tier
stored: TieredFragment | None = service.get(_FRAGMENT_ID)
assert stored is not None, "Fragment was not stored"
assert stored.tier == ContextTier.COLD, (
f"Fragment should be in cold tier, got {stored.tier}"
)
context.fragment_id = _FRAGMENT_ID
@when("I access the fragment {n:d} times via get")
def step_access_n_times(context: Context, n: int) -> None:
"""Access the fragment N times through the get() method."""
service: ContextTierService = context.tier_service
frag_id: str = context.fragment_id
for _ in range(n):
result: TieredFragment | None = service.get(frag_id)
assert result is not None, "Fragment disappeared during access"
context.accessed_fragment = service.get(frag_id)
@then("the fragment should have been promoted to warm or hot tier")
def step_fragment_promoted(context: Context) -> None:
"""Assert the fragment was automatically promoted after repeated access.
Bug #821: ``get()`` increments ``access_count`` and updates
``last_accessed`` but never calls ``promote()`` or any equivalent
runtime tier-transition logic. The fragment remains in the cold
tier regardless of how many times it is accessed.
"""
service: ContextTierService = context.tier_service
frag: TieredFragment | None = service.get(context.fragment_id)
assert frag is not None, "Fragment not found"
# The fragment should have been auto-promoted out of cold tier
assert frag.tier != ContextTier.COLD, (
f"Fragment is still in COLD tier after repeated access "
f"(access_count={frag.access_count}). ContextTierService.get() "
f"updates access metadata but does not auto-promote fragments "
f"based on access patterns (bug #821). Expected tier to be "
f"WARM or HOT."
)
# ---------------------------------------------------------------------------
# Scenario 2: Demotion on staleness
# ---------------------------------------------------------------------------
@given("a fragment stored in the hot tier with a stale last_accessed timestamp")
def step_stale_hot_fragment(context: Context) -> None:
"""Store a fragment in the hot tier with a very old last_accessed."""
stale_time: datetime = datetime.now(tz=UTC) - timedelta(hours=24)
frag: TieredFragment = _make_fragment(
fragment_id=_STALE_FRAGMENT_ID,
tier=ContextTier.HOT,
token_count=100,
last_accessed=stale_time,
)
service: ContextTierService = context.tier_service
service.store(frag)
context.stale_fragment_id = _STALE_FRAGMENT_ID
@when("I invoke the staleness enforcement runtime")
def step_invoke_staleness(context: Context) -> None:
"""Attempt to invoke the staleness enforcement method.
Bug #821: There is no method on ContextTierService that enforces
staleness-based demotion. Methods like ``enforce_staleness()``,
``apply_tier_policy()``, or ``tick()`` do not exist. We attempt
to call the most likely candidate names; if none exist, we record
the failure.
"""
service: ContextTierService = context.tier_service
context.staleness_invoked = False
# Try plausible method names for staleness enforcement
for method_name in (
"enforce_staleness",
"apply_tier_policy",
"tick",
"enforce_demotion_policy",
"run_lifecycle",
"apply_lifecycle",
):
method = getattr(service, method_name, None)
if callable(method):
method()
context.staleness_invoked = True
return
# No runtime method found — record this for the assertion step
context.staleness_invoked = False
@then("the fragment should have been demoted to warm or cold tier")
def step_fragment_demoted(context: Context) -> None:
"""Assert the stale fragment was automatically demoted.
Bug #821: ContextTierService has no staleness enforcement runtime.
There is no method that inspects ``last_accessed`` timestamps and
auto-demotes stale hot-tier fragments. The fragment stays in the
hot tier indefinitely.
"""
service: ContextTierService = context.tier_service
# First check: was there even a method to invoke?
assert context.staleness_invoked, (
"No staleness enforcement method found on ContextTierService. "
"Tried: enforce_staleness(), apply_tier_policy(), tick(), "
"enforce_demotion_policy(), run_lifecycle(), apply_lifecycle(). "
"ContextTierService has no runtime logic for automatic demotion "
"of stale fragments (bug #821)."
)
# Second check: did the fragment actually move?
frag: TieredFragment | None = service.get(context.stale_fragment_id)
assert frag is not None, "Stale fragment not found"
assert frag.tier != ContextTier.HOT, (
f"Stale fragment is still in HOT tier "
f"(last_accessed={frag.last_accessed}). "
f"ContextTierService has no runtime staleness enforcement — "
f"stale fragments are never automatically demoted (bug #821). "
f"Expected tier to be WARM or COLD."
)
# ---------------------------------------------------------------------------
# Scenario 3: Eviction on budget overflow
# ---------------------------------------------------------------------------
@given("a context tier service with a small hot tier budget of {n:d} tokens")
def step_service_small_budget(context: Context, n: int) -> None:
"""Create a ContextTierService with a constrained hot tier budget."""
# ContextTierService accepts Settings; we pass None and override budget
service: ContextTierService = ContextTierService(settings=None)
service._budget = TierBudget(
max_tokens_hot=n,
max_decisions_warm=500,
max_decisions_cold=5000,
)
context.tier_service = service
context.hot_budget_tokens = n
@given("the hot tier is filled to its token budget limit")
def step_fill_hot_tier(context: Context) -> None:
"""Fill the hot tier to exactly the token budget limit."""
service: ContextTierService = context.tier_service
budget_tokens: int = context.hot_budget_tokens
# Store fragments totalling exactly the budget limit
# Use 50-token fragments so 100 tokens = 2 fragments
frag_size: int = 50
num_frags: int = budget_tokens // frag_size
context.initial_hot_fragment_ids = []
for i in range(num_frags):
frag_id: str = f"frag-budget-{i:03d}"
frag: TieredFragment = _make_fragment(
fragment_id=frag_id,
tier=ContextTier.HOT,
token_count=frag_size,
content="x" * frag_size,
last_accessed=datetime.now(tz=UTC) - timedelta(minutes=num_frags - i),
)
service.store(frag)
context.initial_hot_fragment_ids.append(frag_id)
# Record the oldest fragment (lowest last_accessed)
context.oldest_fragment_id = context.initial_hot_fragment_ids[0]
metrics = service.get_metrics()
assert metrics.hot_count == num_frags, (
f"Expected {num_frags} fragments in hot tier, got {metrics.hot_count}"
)
@when("I store one more fragment in the hot tier")
def step_store_overflow(context: Context) -> None:
"""Store one more fragment that would exceed the hot tier budget."""
service: ContextTierService = context.tier_service
overflow_frag: TieredFragment = _make_fragment(
fragment_id="frag-overflow-001",
tier=ContextTier.HOT,
token_count=50,
content="overflow content",
)
service.store(overflow_frag)
context.overflow_fragment_id = "frag-overflow-001"
@then("the oldest fragment should have been evicted from the hot tier")
def step_oldest_evicted(context: Context) -> None:
"""Assert the oldest hot-tier fragment was automatically evicted.
Bug #821: ``store()`` does not check ``TierBudget.max_tokens_hot``
and does not trigger ``evict_lru()`` when the budget is exceeded.
The hot tier grows without bound.
"""
service: ContextTierService = context.tier_service
budget_tokens: int = context.hot_budget_tokens
# Calculate total tokens now in hot tier
hot_total_tokens: int = sum(f.token_count for f in service._hot.values())
# The hot tier should not exceed its budget
assert hot_total_tokens <= budget_tokens, (
f"Hot tier has {hot_total_tokens} tokens but budget is "
f"{budget_tokens} tokens. store() does not enforce "
f"TierBudget.max_tokens_hot or trigger automatic LRU "
f"eviction when the budget is exceeded (bug #821)."
)
# The oldest fragment should have been evicted
oldest_id: str = context.oldest_fragment_id
oldest_frag: TieredFragment | None = service.get(oldest_id)
assert oldest_frag is None or oldest_frag.tier != ContextTier.HOT, (
f"Oldest fragment '{oldest_id}' is still in the hot tier after "
f"budget overflow. ContextTierService.store() does not auto-evict "
f"the least-recently-used fragment when hot tier budget is "
f"exceeded (bug #821)."
)
+39
View File
@@ -0,0 +1,39 @@
@tdd_expected_fail @tdd_bug @tdd_bug_821 @mock_only
Feature: TDD Bug #821 — context tier service has data models but no runtime logic
As a developer
I want to verify that ContextTierService automatically promotes, demotes,
and evicts fragments based on access patterns, staleness, and budget limits
So that the bug is captured and will be caught by a regression test
ContextTierService has well-defined data models for hot/warm/cold tiers
(ContextTier, TieredFragment, TierBudget) and manual promote()/demote()/
evict_lru() methods, but NO automatic runtime logic:
- get() touches access metadata but never auto-promotes a frequently
accessed cold/warm fragment to a higher tier.
- There is no staleness enforcement: no method checks last_accessed
timestamps and demotes stale hot fragments to warm/cold.
- store() does not enforce budget limits: storing beyond
max_tokens_hot does not trigger automatic eviction.
These tests assert the expected runtime behaviour and will FAIL until
the bug is fixed. The @tdd_expected_fail tag inverts the result so
CI passes.
Scenario: Promotion on repeated access moves fragment to a higher tier
Given a context tier service with default budget
And a fragment stored in the cold tier
When I access the fragment 5 times via get
Then the fragment should have been promoted to warm or hot tier
Scenario: Demotion on staleness moves fragment to a lower tier
Given a context tier service with default budget
And a fragment stored in the hot tier with a stale last_accessed timestamp
When I invoke the staleness enforcement runtime
Then the fragment should have been demoted to warm or cold tier
Scenario: Eviction on hot tier budget overflow removes oldest fragment
Given a context tier service with a small hot tier budget of 100 tokens
And the hot tier is filled to its token budget limit
When I store one more fragment in the hot tier
Then the oldest fragment should have been evicted from the hot tier
+232
View File
@@ -0,0 +1,232 @@
"""Helper script for tdd_context_tier_runtime.robot smoke tests.
Each subcommand exercises ``ContextTierService`` to reproduce bug #821.
The helper reports the **real** outcome: it exits 0 and prints the sentinel
when the expected behaviour is observed (bug fixed), and exits 1 when the
bug is still present. The ``tdd_expected_fail_listener`` on the Robot side
handles pass/fail inversion while the bug remains open.
"""
from __future__ import annotations
import sys
from collections.abc import Callable
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import NoReturn
# Ensure local source tree is importable.
_ROOT: Path = Path(__file__).resolve().parents[1]
_SRC: str = str(_ROOT / "src")
_ROBOT: str = str(_ROOT / "robot")
for _p in (_SRC, _ROBOT):
if _p not in sys.path:
sys.path.insert(0, _p)
from cleveragents.application.services.context_tiers import ( # noqa: E402
ContextTierService,
)
from cleveragents.domain.models.acms.tiers import ( # noqa: E402
ContextTier,
TierBudget,
TieredFragment,
)
def _fail(msg: str) -> NoReturn:
"""Print error message to stderr and exit with code 1."""
print(msg, file=sys.stderr)
sys.exit(1)
def _make_fragment(
fragment_id: str,
tier: ContextTier,
token_count: int = 50,
content: str = "test content",
last_accessed: datetime | None = None,
) -> TieredFragment:
"""Create a ``TieredFragment`` with the given properties."""
kwargs: dict[str, object] = {
"fragment_id": fragment_id,
"content": content,
"tier": tier,
"token_count": token_count,
"project_name": "test-project",
}
if last_accessed is not None:
kwargs["last_accessed"] = last_accessed
return TieredFragment(**kwargs) # type: ignore[arg-type]
# ---------------------------------------------------------------------------
# Subcommands
# ---------------------------------------------------------------------------
def _promote_on_access() -> None:
"""Verify that accessing a cold-tier fragment repeatedly auto-promotes it.
Bug #821: ``get()`` increments ``access_count`` and updates
``last_accessed`` but never calls ``promote()`` or any equivalent
runtime logic. The fragment stays in the cold tier forever.
"""
service: ContextTierService = ContextTierService(settings=None)
frag: TieredFragment = _make_fragment(
fragment_id="frag-promote-001",
tier=ContextTier.COLD,
token_count=50,
)
service.store(frag)
# Access the fragment 5 times
for _ in range(5):
result: TieredFragment | None = service.get("frag-promote-001")
if result is None:
_fail("Fragment disappeared during repeated access")
# Check if the fragment was promoted
final: TieredFragment | None = service.get("frag-promote-001")
if final is None:
_fail("Fragment not found after repeated access")
if final.tier == ContextTier.COLD:
_fail(
f"Fragment is still in COLD tier after 5 accesses "
f"(access_count={final.access_count}). "
f"ContextTierService.get() does not auto-promote fragments "
f"based on access patterns (bug #821)."
)
print("tdd-promote-on-access-ok")
def _demote_on_staleness() -> None:
"""Verify that a stale hot-tier fragment is auto-demoted.
Bug #821: ContextTierService has no staleness enforcement method.
No ``enforce_staleness()``, ``apply_tier_policy()``, ``tick()``,
or similar runtime method exists.
"""
service: ContextTierService = ContextTierService(settings=None)
stale_time: datetime = datetime.now(tz=UTC) - timedelta(hours=24)
frag: TieredFragment = _make_fragment(
fragment_id="frag-stale-001",
tier=ContextTier.HOT,
token_count=100,
last_accessed=stale_time,
)
service.store(frag)
# Attempt to invoke staleness enforcement
invoked: bool = False
for method_name in (
"enforce_staleness",
"apply_tier_policy",
"tick",
"enforce_demotion_policy",
"run_lifecycle",
"apply_lifecycle",
):
method: object = getattr(service, method_name, None)
if callable(method):
method()
invoked = True
break
if not invoked:
_fail(
"No staleness enforcement method found on ContextTierService. "
"Tried: enforce_staleness(), apply_tier_policy(), tick(), "
"enforce_demotion_policy(), run_lifecycle(), apply_lifecycle(). "
"ContextTierService has no runtime logic for automatic "
"demotion of stale fragments (bug #821)."
)
# Check if the fragment was demoted
result: TieredFragment | None = service.get("frag-stale-001")
if result is None:
_fail("Stale fragment not found after staleness enforcement")
if result.tier == ContextTier.HOT:
_fail(
f"Stale fragment is still in HOT tier "
f"(last_accessed={result.last_accessed}). "
f"Staleness enforcement did not demote it (bug #821)."
)
print("tdd-demote-on-staleness-ok")
def _evict_on_overflow() -> None:
"""Verify that hot tier budget overflow triggers auto-eviction.
Bug #821: ``store()`` does not check ``TierBudget.max_tokens_hot``
and does not call ``evict_lru()`` when the budget is exceeded.
"""
service: ContextTierService = ContextTierService(settings=None)
service._budget = TierBudget(
max_tokens_hot=100,
max_decisions_warm=500,
max_decisions_cold=5000,
)
# Fill hot tier to exactly 100 tokens (2 x 50-token fragments)
for i in range(2):
frag: TieredFragment = _make_fragment(
fragment_id=f"frag-budget-{i:03d}",
tier=ContextTier.HOT,
token_count=50,
content="x" * 50,
last_accessed=datetime.now(tz=UTC) - timedelta(minutes=2 - i),
)
service.store(frag)
oldest_id: str = "frag-budget-000"
# Store one more fragment that exceeds the budget
overflow: TieredFragment = _make_fragment(
fragment_id="frag-overflow-001",
tier=ContextTier.HOT,
token_count=50,
content="overflow",
)
service.store(overflow)
# Check hot tier total tokens
hot_total: int = sum(f.token_count for f in service._hot.values())
if hot_total > 100:
_fail(
f"Hot tier has {hot_total} tokens but budget is 100 tokens. "
f"store() does not enforce TierBudget.max_tokens_hot or "
f"trigger automatic LRU eviction (bug #821)."
)
# Check that the oldest fragment was evicted
oldest: TieredFragment | None = service.get(oldest_id)
if oldest is not None and oldest.tier == ContextTier.HOT:
_fail(
f"Oldest fragment '{oldest_id}' is still in hot tier after "
f"budget overflow. store() does not auto-evict (bug #821)."
)
print("tdd-evict-on-overflow-ok")
# ---------------------------------------------------------------------------
# Dispatcher
# ---------------------------------------------------------------------------
_COMMANDS: dict[str, Callable[[], None]] = {
"promote-on-access": _promote_on_access,
"demote-on-staleness": _demote_on_staleness,
"evict-on-overflow": _evict_on_overflow,
}
if __name__ == "__main__":
if len(sys.argv) < 2 or sys.argv[1] not in _COMMANDS:
print(
f"Usage: {sys.argv[0]} <{'|'.join(_COMMANDS)}>",
file=sys.stderr,
)
sys.exit(1)
cmd: Callable[[], None] = _COMMANDS[sys.argv[1]]
cmd()
+45
View File
@@ -0,0 +1,45 @@
*** Settings ***
Documentation TDD Bug #821 — context tier service has data models but no runtime logic
... for promotion, demotion, or eviction. Integration smoke tests verifying
... that ContextTierService automatically promotes fragments on repeated access,
... demotes stale fragments, and evicts fragments when the hot tier budget
... overflows. Currently the service has manual promote()/demote()/evict_lru()
... methods but no automatic runtime behaviour wired into get() or store().
... Tests are tagged tdd_expected_fail so CI passes via result inversion.
Resource ${CURDIR}/common.resource
Suite Setup Setup Test Environment
Suite Teardown Cleanup Test Environment
*** Variables ***
${HELPER} ${CURDIR}/helper_tdd_context_tier_runtime.py
*** Test Cases ***
TDD Promotion On Repeated Access
[Documentation] Verify that accessing a cold-tier fragment repeatedly via get()
... auto-promotes it to warm or hot tier.
[Tags] tdd_bug tdd_bug_821 tdd_expected_fail
${result}= Run Process ${PYTHON} ${HELPER} promote-on-access cwd=${WORKSPACE} timeout=30s
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} tdd-promote-on-access-ok
TDD Demotion On Staleness
[Documentation] Verify that a stale hot-tier fragment is automatically demoted
... to warm or cold tier by a staleness enforcement runtime.
[Tags] tdd_bug tdd_bug_821 tdd_expected_fail
${result}= Run Process ${PYTHON} ${HELPER} demote-on-staleness cwd=${WORKSPACE} timeout=30s
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} tdd-demote-on-staleness-ok
TDD Eviction On Budget Overflow
[Documentation] Verify that storing beyond the hot tier token budget triggers
... automatic LRU eviction of the oldest fragment.
[Tags] tdd_bug tdd_bug_821 tdd_expected_fail
${result}= Run Process ${PYTHON} ${HELPER} evict-on-overflow cwd=${WORKSPACE} timeout=30s
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} tdd-evict-on-overflow-ok