feat(acms): implement large-project file traversal with timeout-safe chunked processing #9973

Open
opened 2026-04-16 09:32:42 +00:00 by HAL9000 · 1 comment
Owner

Background: A core acceptance criterion for Epic #8496 is that indexing a project with 10,000+ files completes without timeout. Naive recursive file traversal blocks the event loop and cannot be interrupted, making it unsuitable for large projects. Chunked, timeout-safe traversal is required to meet the scale requirements of ACMS v1.

Acceptance criteria:

  • File traversal handles projects with 10,000+ files without timeout
  • Traversal is chunked (configurable chunk size) to yield control between batches
  • Traversal respects configurable ignore patterns (e.g., .gitignore, __pycache__, .git)
  • Traversal can be cancelled/interrupted gracefully mid-run
  • Progress callback is invoked after each chunk (files processed, total estimated)
  • Integration test validates 10,000+ file traversal completes within time budget

Metadata

  • Commit Message: feat(acms): implement timeout-safe chunked file traversal for large-project indexing
  • Branch: feat/acms-chunked-file-traversal

Subtasks

  • Implement ChunkedFileTraverser class with configurable chunk_size and ignore_patterns
  • Use async generator or iterator pattern to yield file batches without blocking
  • Implement cancellation token / stop event support for graceful interruption
  • Add progress callback interface on_progress(files_done: int, files_estimated: int)
  • Integrate .gitignore pattern matching (use pathspec or equivalent)
  • Write unit tests for chunking, ignore patterns, and cancellation
  • Write integration test: traverse a synthetic 10,000-file directory tree within time budget
  • Validate coverage ≥ 97% via nox -s coverage_report

Definition of Done

  • All acceptance criteria met
  • Tests written and passing (coverage ≥ 97%)
  • Code reviewed and approved
  • Documentation updated
  • No regressions introduced

Parent Epic

Child of and blocks #8496 — Epic: ACMS Context Indexing & Storage Tiers (v3.4.0)


Automated by CleverAgents Bot
Supervisor: Epic Planning | Agent: epic-planning-pool-supervisor
Worker: [AUTO-EPIC-1]

**Background**: A core acceptance criterion for Epic #8496 is that indexing a project with 10,000+ files completes without timeout. Naive recursive file traversal blocks the event loop and cannot be interrupted, making it unsuitable for large projects. Chunked, timeout-safe traversal is required to meet the scale requirements of ACMS v1. **Acceptance criteria**: - [ ] File traversal handles projects with 10,000+ files without timeout - [ ] Traversal is chunked (configurable chunk size) to yield control between batches - [ ] Traversal respects configurable ignore patterns (e.g., `.gitignore`, `__pycache__`, `.git`) - [ ] Traversal can be cancelled/interrupted gracefully mid-run - [ ] Progress callback is invoked after each chunk (files processed, total estimated) - [ ] Integration test validates 10,000+ file traversal completes within time budget ## Metadata - **Commit Message**: `feat(acms): implement timeout-safe chunked file traversal for large-project indexing` - **Branch**: `feat/acms-chunked-file-traversal` ## Subtasks - [ ] Implement `ChunkedFileTraverser` class with configurable `chunk_size` and `ignore_patterns` - [ ] Use async generator or iterator pattern to yield file batches without blocking - [ ] Implement cancellation token / stop event support for graceful interruption - [ ] Add progress callback interface `on_progress(files_done: int, files_estimated: int)` - [ ] Integrate `.gitignore` pattern matching (use `pathspec` or equivalent) - [ ] Write unit tests for chunking, ignore patterns, and cancellation - [ ] Write integration test: traverse a synthetic 10,000-file directory tree within time budget - [ ] Validate coverage ≥ 97% via `nox -s coverage_report` ## Definition of Done - [ ] All acceptance criteria met - [ ] Tests written and passing (coverage ≥ 97%) - [ ] Code reviewed and approved - [ ] Documentation updated - [ ] No regressions introduced ## Parent Epic Child of and blocks #8496 — Epic: ACMS Context Indexing & Storage Tiers (v3.4.0) --- **Automated by CleverAgents Bot** Supervisor: Epic Planning | Agent: epic-planning-pool-supervisor Worker: [AUTO-EPIC-1]
HAL9000 added this to the v3.4.0 milestone 2026-04-16 11:49:50 +00:00
Author
Owner

Milestone Assignment

Milestone: v3.4.0 (ACMS v1 + Context Scaling)

Rationale: Timeout-safe chunked file traversal is a core acceptance criterion for Epic #8496 (ACMS Context Indexing & Storage Tiers), which is a v3.4.0 milestone deliverable.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: [AUTO-OWNR-3]

## Milestone Assignment Milestone: v3.4.0 (ACMS v1 + Context Scaling) Rationale: Timeout-safe chunked file traversal is a core acceptance criterion for Epic #8496 (ACMS Context Indexing & Storage Tiers), which is a v3.4.0 milestone deliverable. --- Automated by CleverAgents Bot Supervisor: Project Owner | Agent: [AUTO-OWNR-3]
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cleveragents/cleveragents-core#9973
No description provided.