feat(agents): add offset-based pagination to file_read #83

New issue

Closed

opened 2026-07-30 15:47:38 +00:00 by CoreRasurae · 0 comments

CoreRasurae commented

2026-07-30 15:47:38 +00:00

Member

Metadata

Field	Value
Commit Message	feat(agents): add offset-based pagination to file_read
Branch	feature/m2-file-read-offset

Background and context

ToolAgent._file_read_tool (cleveractors.agents.tool) always reads a file from the beginning: f.read() loads the entire file, and the optional max_chars argument (added by ADR-2030 D-3) only bounds where the returned content stops — it always slices content[:max_chars], i.e. from position 0. There is no .seek()/.tell() or equivalent anywhere in the codebase.

This means once a large file is truncated at max_chars, there is no way to retrieve the next chunk. Per ADR-2030's own guidance embedded in the tool schema (cleveractors.agents.llm_tools._BUILTIN_TOOL_SCHEMAS["file_read"]: "Use 8000 for scanning, 16000 for analysis"), the LLM is explicitly steered toward truncated reads for large files, but is given no built-in mechanism to continue past the truncation point. Today the only workaround is falling back to the shell tool (tail -c +N, sed -n, dd skip=...) — exactly the kind of awkward, error-prone detour ADR-2030 (D-4) already eliminated for shell-commands-passed-as-paths.

Adding an offset argument lets the LLM page through a large file across multiple file_read calls without needing shell fallbacks, and without re-reading (and re-spending context on) content it already saw.

Current behavior

file_read has no offset argument in its schema (_BUILTIN_TOOL_SCHEMAS["file_read"]["parameters"]) or its implementation (ToolAgent._file_read_tool).
Every call reads from the start of the file. Two calls with the same file and max_chars always return the identical (truncated) prefix.
There is no mechanism — in the returned content, the [FILE_READ_SUCCESS] header, or elsewhere — for the LLM to learn that more content exists beyond a truncated read, or what offset to request next.

Expected behavior

file_read accepts an optional offset argument (default 0, preserving all current behavior when omitted) that starts the returned content at that position within the file instead of position 0.
Combined with max_chars, offset allows retrieving a bounded window [offset, offset + max_chars) of the file, enabling sequential pagination across multiple calls.
The [FILE_READ_SUCCESS] header (and/or the TRUNCATED marker) is extended so the LLM can tell, from the response alone, whether more content remains past the returned window and — ideally — what offset to pass next, without it having to compute offset + max_chars itself from scratch.
Out-of-range input (negative offset, offset beyond end-of-file) is handled with a clear, actionable ExecutionError rather than an unhandled exception or silent empty content.
No change to behavior when offset is omitted — this is purely additive, per §1.3 of the Actor Configuration Standard (compliant implementations MAY provide additional tool arguments).

Open design questions the ADR below MUST resolve

These are genuine design decisions, not implementation details to leave implicit:

Unit of offset: characters (consistent with max_chars, which already counts decoded characters since the file is opened in text mode) vs. bytes vs. lines. Characters is the natural choice for consistency with max_chars, but the ADR must state this explicitly and justify it.
Continuation signaling: exact wording/format of the header addition that tells the LLM whether more content remains and what the next offset would be (e.g. extending the existing TRUNCATED to N chars convention with something like | MORE_CONTENT_AT_OFFSET: N).
Out-of-range semantics: whether an offset beyond end-of-file is an error or an empty-content success; whether a negative offset is rejected outright.
Directory listings: whether offset should (eventually) paginate very large directory listings too, or is explicitly deferred to a future issue.
Efficiency: whether the initial implementation is allowed to keep reading the whole file into memory before slicing by character offset (as max_chars already does today), or whether byte-level seeking is required for large-file efficiency. Given _file_read_tool already reads the whole file for max_chars, keeping the same approach for offset is likely acceptable for v1 — but the ADR must say so explicitly rather than leaving it to be inferred from the diff.

Acceptance criteria

_BUILTIN_TOOL_SCHEMAS["file_read"]["parameters"] (cleveractors.agents.llm_tools) documents a new optional offset parameter, including its unit and interaction with max_chars.
ToolAgent._file_read_tool accepts offset, defaulting to 0; omitting it reproduces byte-for-byte identical output to current behavior (regression guard).
A call with offset=N returns content starting at character position N of the file (not position 0).
The response signals whether more content remains beyond the returned window, per the format decided in the ADR.
A negative offset or a non-integer offset raises ExecutionError with a clear message (mirroring the existing max_chars integer-validation pattern).
An offset at or beyond the file's length raises ExecutionError or returns a clearly-labeled empty-content response (per the ADR's decision) — not an unhandled IndexError/silent empty string.
docs/adr/ADR-2033-file-read-offset-pagination.md exists, is written in the Context/Decision/Consequences/Alternatives-Considered format used by ADR-2030/2031/2032, and resolves all the open design questions listed above.
nox -s coverage_report stays ≥ 97%.

Supporting information

Actor Configuration Standard, §4.5.1 (docs/index.md) — defines file_read's baseline argument set (file/args[0]) and states additional arguments MAY be handled by implementations.
docs/adr/ADR-2030-tool-calling-spec-extensions.md — D-2 (directory listing) and D-3 (max_chars truncation) are the direct precedent this issue extends; D-4 establishes the precedent of giving the LLM an actionable path instead of a dead end, which is the same motivation for offset.
Relevant symbols at commit a8f68b8:
- cleveractors.agents.tool.ToolAgent._file_read_tool
- cleveractors.agents.llm_tools._BUILTIN_TOOL_SCHEMAS["file_read"]
- cleveractors.agents.llm_tools.normalize_tool_entry
Parent Epic note: This does not cleanly fit Epic #77 ("LLM Agent Runtime Stabilization"), which explicitly excludes "new capabilities / feature build-out" from its scope (its own text calls out #8, #67, #73, #75 as excluded for this exact reason). This issue is thematically closest to those excluded siblings — the project may want to group this with #67/#73/#75 under a new "Tool-Calling Capability Build-Out" Epic during triage. Left unlinked pending that maintainer decision rather than force-fitting into #77.

Subtasks

Write docs/adr/ADR-2033-file-read-offset-pagination.md (Context/Decision/Consequences/Alternatives Considered), resolving the open design questions above — required, not optional, since this changes tool behavior and its LLM-facing schema
Add the offset parameter to _BUILTIN_TOOL_SCHEMAS["file_read"] in cleveractors.agents.llm_tools, per the ADR's decisions
Implement offset handling in ToolAgent._file_read_tool, including validation and continuation signaling, per the ADR's decisions
Tests (Behave): scenario reading a file with offset=0 matches current (no-offset) behavior exactly
Tests (Behave): scenario reading a file with offset=N returns the expected windowed content
Tests (Behave): scenario combining offset and max_chars to page through a file across two calls with no gap or overlap
Tests (Behave): scenarios for negative offset, non-integer offset, and offset beyond end-of-file
Verify coverage >=97% via nox -s coverage_report
Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
docs/adr/ADR-2033-file-read-offset-pagination.md is merged in the same PR as the implementation, with status accepted.
A Git commit is created where the first line matches the Commit Message in Metadata exactly.
The commit is pushed to the branch matching the Branch in Metadata exactly.
The commit is submitted as a PR to master, reviewed, and merged.

## Metadata | Field | Value | |-------|-------| | Commit Message | feat(agents): add offset-based pagination to file_read | | Branch | feature/m2-file-read-offset | ## Background and context `ToolAgent._file_read_tool` (`cleveractors.agents.tool`) always reads a file from the beginning: `f.read()` loads the entire file, and the optional `max_chars` argument (added by ADR-2030 D-3) only bounds where the returned content *stops* — it always slices `content[:max_chars]`, i.e. from position 0. There is no `.seek()`/`.tell()` or equivalent anywhere in the codebase. This means once a large file is truncated at `max_chars`, there is no way to retrieve the *next* chunk. Per ADR-2030's own guidance embedded in the tool schema (`cleveractors.agents.llm_tools._BUILTIN_TOOL_SCHEMAS["file_read"]`: "Use 8000 for scanning, 16000 for analysis"), the LLM is explicitly steered toward truncated reads for large files, but is given no built-in mechanism to continue past the truncation point. Today the only workaround is falling back to the `shell` tool (`tail -c +N`, `sed -n`, `dd skip=...`) — exactly the kind of awkward, error-prone detour ADR-2030 (D-4) already eliminated for shell-commands-passed-as-paths. Adding an `offset` argument lets the LLM page through a large file across multiple `file_read` calls without needing shell fallbacks, and without re-reading (and re-spending context on) content it already saw. ## Current behavior 1. `file_read` has no `offset` argument in its schema (`_BUILTIN_TOOL_SCHEMAS["file_read"]["parameters"]`) or its implementation (`ToolAgent._file_read_tool`). 2. Every call reads from the start of the file. Two calls with the same `file` and `max_chars` always return the identical (truncated) prefix. 3. There is no mechanism — in the returned content, the `[FILE_READ_SUCCESS]` header, or elsewhere — for the LLM to learn that more content exists beyond a truncated read, or what offset to request next. ## Expected behavior 1. `file_read` accepts an optional `offset` argument (default `0`, preserving all current behavior when omitted) that starts the returned content at that position within the file instead of position 0. 2. Combined with `max_chars`, `offset` allows retrieving a bounded window `[offset, offset + max_chars)` of the file, enabling sequential pagination across multiple calls. 3. The `[FILE_READ_SUCCESS]` header (and/or the `TRUNCATED` marker) is extended so the LLM can tell, from the response alone, whether more content remains past the returned window and — ideally — what offset to pass next, without it having to compute `offset + max_chars` itself from scratch. 4. Out-of-range input (negative offset, offset beyond end-of-file) is handled with a clear, actionable `ExecutionError` rather than an unhandled exception or silent empty content. 5. No change to behavior when `offset` is omitted — this is purely additive, per §1.3 of the Actor Configuration Standard (compliant implementations MAY provide additional tool arguments). ### Open design questions the ADR below MUST resolve These are genuine design decisions, not implementation details to leave implicit: - **Unit of `offset`**: characters (consistent with `max_chars`, which already counts decoded characters since the file is opened in text mode) vs. bytes vs. lines. Characters is the natural choice for consistency with `max_chars`, but the ADR must state this explicitly and justify it. - **Continuation signaling**: exact wording/format of the header addition that tells the LLM whether more content remains and what the next `offset` would be (e.g. extending the existing `TRUNCATED to N chars` convention with something like `| MORE_CONTENT_AT_OFFSET: N`). - **Out-of-range semantics**: whether an offset beyond end-of-file is an error or an empty-content success; whether a negative offset is rejected outright. - **Directory listings**: whether `offset` should (eventually) paginate very large directory listings too, or is explicitly deferred to a future issue. - **Efficiency**: whether the initial implementation is allowed to keep reading the whole file into memory before slicing by character offset (as `max_chars` already does today), or whether byte-level seeking is required for large-file efficiency. Given `_file_read_tool` already reads the whole file for `max_chars`, keeping the same approach for `offset` is likely acceptable for v1 — but the ADR must say so explicitly rather than leaving it to be inferred from the diff. ## Acceptance criteria - [x] `_BUILTIN_TOOL_SCHEMAS["file_read"]["parameters"]` (`cleveractors.agents.llm_tools`) documents a new optional `offset` parameter, including its unit and interaction with `max_chars`. - [x] `ToolAgent._file_read_tool` accepts `offset`, defaulting to `0`; omitting it reproduces byte-for-byte identical output to current behavior (regression guard). - [x] A call with `offset=N` returns content starting at character position `N` of the file (not position 0). - [x] The response signals whether more content remains beyond the returned window, per the format decided in the ADR. - [x] A negative `offset` or a non-integer `offset` raises `ExecutionError` with a clear message (mirroring the existing `max_chars` integer-validation pattern). - [x] An `offset` at or beyond the file's length raises `ExecutionError` or returns a clearly-labeled empty-content response (per the ADR's decision) — not an unhandled `IndexError`/silent empty string. - [x] `docs/adr/ADR-2033-file-read-offset-pagination.md` exists, is written in the Context/Decision/Consequences/Alternatives-Considered format used by ADR-2030/2031/2032, and resolves all the open design questions listed above. - [ ] `nox -s coverage_report` stays ≥ 97%. ## Supporting information - Actor Configuration Standard, §4.5.1 (`docs/index.md`) — defines `file_read`'s baseline argument set (`file`/`args[0]`) and states additional arguments MAY be handled by implementations. - `docs/adr/ADR-2030-tool-calling-spec-extensions.md` — D-2 (directory listing) and D-3 (`max_chars` truncation) are the direct precedent this issue extends; D-4 establishes the precedent of giving the LLM an actionable path instead of a dead end, which is the same motivation for `offset`. - Relevant symbols at commit `a8f68b8`: - `cleveractors.agents.tool.ToolAgent._file_read_tool` - `cleveractors.agents.llm_tools._BUILTIN_TOOL_SCHEMAS["file_read"]` - `cleveractors.agents.llm_tools.normalize_tool_entry` - **Parent Epic note**: This does not cleanly fit Epic #77 ("LLM Agent Runtime Stabilization"), which explicitly excludes "new capabilities / feature build-out" from its scope (its own text calls out #8, #67, #73, #75 as excluded for this exact reason). This issue is thematically closest to those excluded siblings — the project may want to group this with #67/#73/#75 under a new "Tool-Calling Capability Build-Out" Epic during triage. Left unlinked pending that maintainer decision rather than force-fitting into #77. ## Subtasks - [x] Write `docs/adr/ADR-2033-file-read-offset-pagination.md` (Context/Decision/Consequences/Alternatives Considered), resolving the open design questions above — **required**, not optional, since this changes tool behavior and its LLM-facing schema - [x] Add the `offset` parameter to `_BUILTIN_TOOL_SCHEMAS["file_read"]` in `cleveractors.agents.llm_tools`, per the ADR's decisions - [x] Implement `offset` handling in `ToolAgent._file_read_tool`, including validation and continuation signaling, per the ADR's decisions - [x] Tests (Behave): scenario reading a file with `offset=0` matches current (no-offset) behavior exactly - [x] Tests (Behave): scenario reading a file with `offset=N` returns the expected windowed content - [x] Tests (Behave): scenario combining `offset` and `max_chars` to page through a file across two calls with no gap or overlap - [x] Tests (Behave): scenarios for negative offset, non-integer offset, and offset beyond end-of-file - [ ] Verify coverage >=97% via nox -s coverage_report - [ ] Run nox (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - `docs/adr/ADR-2033-file-read-offset-pagination.md` is merged in the same PR as the implementation, with status `accepted`. - A Git commit is created where the first line matches the Commit Message in Metadata exactly. - The commit is pushed to the branch matching the Branch in Metadata exactly. - The commit is submitted as a PR to master, reviewed, and merged.

Rows
Columns