cleveragents/cleveragents-core

Fork 3

BUG-HUNT: [boundary] hydrate_tiers_from_project uses `break` on total budget overflow — smaller files after a large file are never indexed #6443

New issue

Open

opened 2026-04-09 21:03:34 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-09 21:03:34 +00:00

Owner

Bug Report: [boundary] `hydrate_tiers_from_project` total budget uses `break` not `continue` — under-populates context tier

Severity Assessment

Impact: When the total bytes budget (_MAX_TOTAL_BYTES = 10 MB) is exceeded by any single file, indexing stops immediately. Smaller files that appear after the overflow file in iteration order — and that would individually fit within the remaining budget — are never indexed. The context tier is under-populated, meaning the LLM receives less context than it should.
Likelihood: Medium — triggered whenever file listing order places a large file (e.g., 9–10 MB) before smaller important files (e.g., source files, READMEs). Git ls-files and os.walk both return files in a non-size-sorted order, making this common.
Priority: Medium

Location

File: src/cleveragents/application/services/context_tier_hydrator.py
Function: hydrate_tiers_from_project
Lines: 124–125

Description

The budget enforcement code is:

for rel_path in files:
    abs_path = os.path.join(resource_location, rel_path)

    try:
        size = os.path.getsize(abs_path)
    except OSError:
        continue

    if size > _MAX_FILE_BYTES:        # skip this individual file
        continue
    if total_bytes + size > _MAX_TOTAL_BYTES:  # BUG: break stops ALL further iteration
        break
    ...
    total_bytes += size

The break statement exits the entire loop when any single file would push the total past 10 MB. This means that if files is:

[a.py (9.9 MB), b.py (50 KB), c.py (50 KB)]

After a.py is indexed (total_bytes = 9.9 MB), b.py would push total to 9.95 MB which is under the 10 MB limit, so it continues. But if a.py was larger, the next smaller file causes the break:

[a.py (9.95 MB), b.py (60 KB), c.py (10 KB)]

After a.py: total_bytes = 9.95 MB. For b.py: 9.95 MB + 60 KB = 10.01 MB > 10 MB → break. c.py (10 KB) would have fit but is never reached.

The spec comment says "Maximum total bytes to index per project (10 MB)". The intent of this limit is to prevent excessive memory and IO, not to stop indexing just because one file causes overflow. Files that individually fit within the remaining budget should still be indexed.

Evidence

# context_tier_hydrator.py, lines 122–125
if size > _MAX_FILE_BYTES:
    continue
if total_bytes + size > _MAX_TOTAL_BYTES:
    break   # ← stops ALL further iteration, not just this file

# Contrast with the per-file limit which correctly uses continue:
if size > _MAX_FILE_BYTES:
    continue   # ← correct: skip this file, process remaining

Expected Behavior

When a file would push the total past _MAX_TOTAL_BYTES, that specific file should be skipped (continue), and the loop should continue processing smaller subsequent files. Indexing should stop only when the budget is fully exhausted (no remaining file can fit).

For example, a simple improvement: change break to continue and add a final budget check.

Actual Behavior

Indexing stops at the first file that would overflow the total budget. All subsequent files — regardless of their size — are never indexed.

Suggested Fix

if total_bytes + size > _MAX_TOTAL_BYTES:
    continue   # ← skip this oversized file but continue checking smaller ones

For better performance (avoid iterating when budget is fully used), add an early-exit condition:

if total_bytes >= _MAX_TOTAL_BYTES:
    break   # truly exhausted — no small file can fit
if total_bytes + size > _MAX_TOTAL_BYTES:
    continue  # this particular file is too big, but smaller ones may fit

TDD Note

After this bug issue is verified, a corresponding Type/Testing issue will be created for TDD. The test will use tags: @tdd_issue, @tdd_issue_<this-issue-number>, and @tdd_expected_fail to prove the bug exists before fixing it.

Automated by CleverAgents Bot
Supervisor: Bug Hunting | Agent: bug-hunter

## Bug Report: [boundary] `hydrate_tiers_from_project` total budget uses `break` not `continue` — under-populates context tier ### Severity Assessment - **Impact**: When the total bytes budget (`_MAX_TOTAL_BYTES = 10 MB`) is exceeded by any single file, indexing **stops immediately**. Smaller files that appear after the overflow file in iteration order — and that would individually fit within the remaining budget — are never indexed. The context tier is under-populated, meaning the LLM receives less context than it should. - **Likelihood**: Medium — triggered whenever file listing order places a large file (e.g., 9–10 MB) before smaller important files (e.g., source files, READMEs). Git ls-files and os.walk both return files in a non-size-sorted order, making this common. - **Priority**: Medium ### Location - **File**: `src/cleveragents/application/services/context_tier_hydrator.py` - **Function**: `hydrate_tiers_from_project` - **Lines**: 124–125 ### Description The budget enforcement code is: ```python for rel_path in files: abs_path = os.path.join(resource_location, rel_path) try: size = os.path.getsize(abs_path) except OSError: continue if size > _MAX_FILE_BYTES: # skip this individual file continue if total_bytes + size > _MAX_TOTAL_BYTES: # BUG: break stops ALL further iteration break ... total_bytes += size ``` The `break` statement exits the entire loop when any single file would push the total past 10 MB. This means that if `files` is: ``` [a.py (9.9 MB), b.py (50 KB), c.py (50 KB)] ``` After `a.py` is indexed (`total_bytes = 9.9 MB`), `b.py` would push total to `9.95 MB` which is under the 10 MB limit, so it continues. But if `a.py` was larger, the next smaller file causes the `break`: ``` [a.py (9.95 MB), b.py (60 KB), c.py (10 KB)] ``` After `a.py`: `total_bytes = 9.95 MB`. For `b.py`: `9.95 MB + 60 KB = 10.01 MB > 10 MB` → **`break`**. `c.py` (10 KB) would have fit but is never reached. The spec comment says "Maximum total bytes to index per project (10 MB)". The intent of this limit is to prevent excessive memory and IO, not to stop indexing just because one file causes overflow. Files that individually fit within the remaining budget should still be indexed. ### Evidence ```python # context_tier_hydrator.py, lines 122–125 if size > _MAX_FILE_BYTES: continue if total_bytes + size > _MAX_TOTAL_BYTES: break # ← stops ALL further iteration, not just this file ``` ```python # Contrast with the per-file limit which correctly uses continue: if size > _MAX_FILE_BYTES: continue # ← correct: skip this file, process remaining ``` ### Expected Behavior When a file would push the total past `_MAX_TOTAL_BYTES`, that **specific file** should be skipped (`continue`), and the loop should continue processing smaller subsequent files. Indexing should stop only when the budget is fully exhausted (no remaining file can fit). For example, a simple improvement: change `break` to `continue` and add a final budget check. ### Actual Behavior Indexing stops at the **first** file that would overflow the total budget. All subsequent files — regardless of their size — are never indexed. ### Suggested Fix ```python if total_bytes + size > _MAX_TOTAL_BYTES: continue # ← skip this oversized file but continue checking smaller ones ``` For better performance (avoid iterating when budget is fully used), add an early-exit condition: ```python if total_bytes >= _MAX_TOTAL_BYTES: break # truly exhausted — no small file can fit if total_bytes + size > _MAX_TOTAL_BYTES: continue # this particular file is too big, but smaller ones may fit ``` ### Category boundary ### TDD Note After this bug issue is verified, a corresponding Type/Testing issue will be created for TDD. The test will use tags: `@tdd_issue`, `@tdd_issue_<this-issue-number>`, and `@tdd_expected_fail` to prove the bug exists before fixing it. --- **Automated by CleverAgents Bot** Supervisor: Bug Hunting | Agent: bug-hunter

HAL9000 added the

labels

2026-04-09 21:08:15 +00:00

HAL9000 commented

2026-04-17 08:47:24 +00:00

Author

Owner

✅ Verified — Valid boundary bug. Using break instead of continue means smaller files after a large file are never indexed, producing incomplete context. MoSCoW: Should Have — affects ACMS indexing completeness.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Verified** — Valid boundary bug. Using break instead of continue means smaller files after a large file are never indexed, producing incomplete context. **MoSCoW: Should Have** — affects ACMS indexing completeness. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor

HAL9000 added this to the v3.4.0 milestone

2026-04-17 08:47:45 +00:00

HAL9000 added

and removed

labels

2026-04-18 09:12:22 +00:00