cleveragents-core/docs/reference/context_indexing.md

Repository Indexing Service

The Repository Indexing Service scans linked repository resources, building a persistent file-level index with language detection, content hashing, and token estimation. ACMS uses this index for efficient context assembly on projects with 10K+ files.

Architecture

resource_id + root_path
       |
       v
  ┌─── RepoIndexingService ─────┐
  │  walk filesystem            │
  │  apply include/exclude globs│
  │  enforce max_file_size      │
  │  enforce max_total_size     │
  │  SHA-256 content hashing    │
  │  extension language detect  │
  │  estimate token counts      │
  └──────────┬──────────────────┘
             v
  ┌─── SQLite persistence ──────┐
  │  repo_indexes table         │
  │  indexed_files table        │
  └─────────────────────────────┘

Domain Models

All models are frozen Pydantic v2 with ULID identifiers and UTC datetimes.

IndexStatus

Enum representing the state of an index:

Value	Description
pending	Index creation requested but not yet started
indexing	Filesystem walk in progress
ready	Index complete and available for queries
error	Indexing failed; see error_message
stale	Source files changed since last index

FileRecord

Per-file metadata stored during indexing:

Field	Type	Description
path	str	Relative path from the repository root
content_hash	str	SHA-256 hex digest of file contents
token_count	int	Estimated token count (size_bytes // 4)
language	str	Detected programming language
size_bytes	int	File size in bytes
last_modified	datetime	File modification timestamp (UTC)

IndexMetadata

Summary record for a repository index:

Field	Type	Description
index_id	str	ULID identifier for this index snapshot
resource_id	str	ULID of the linked resource
indexed_at	datetime	When indexing completed (UTC)
file_count	int	Total files in the index
token_estimate	int	Sum of all file token counts
primary_language	str	Most common language by token count (weighted)
status	IndexStatus	Current index state
error_message	`str	None`

RepoIndex

Composite object returned by index and refresh operations:

Field	Type	Description
metadata	IndexMetadata	Index summary
files	tuple[FileRecord, ...]	All indexed file records

Service API

RepoIndexingService(session_factory)

Constructor. Accepts a SQLAlchemy session factory (injected via DI container).

index_resource(resource_id, root_path, *, include_globs, exclude_globs, max_file_size, max_total_size) -> RepoIndex

Full index of a filesystem tree.

Parameter	Type	Default	Description
resource_id	str	required	ULID of the resource
root_path	str | Path	required	Absolute path to the repository root
include_globs	tuple[str, ...]	()	Only index files matching these globs (empty = all)
exclude_globs	tuple[str, ...]	()	Skip files matching these globs
max_file_size	int | None	None	Skip files larger than this (bytes); None = no limit
max_total_size	int | None	None	Stop indexing when cumulative size exceeds this; None = no limit

Raises ValueError if resource_id is empty. Raises FileNotFoundError if root_path does not exist.

refresh_index(resource_id, root_path, **kwargs) -> RepoIndex

Incremental refresh. Compares content hashes against the existing index and only re-processes changed files. Falls back to a full index if no prior index exists. Accepts the same keyword arguments as index_resource.

get_index(resource_id) -> RepoIndex | None

Retrieve a full index (metadata + file records) from the database. Returns None if no index exists.

get_index_status(resource_id) -> IndexMetadata | None

Lightweight query returning only the metadata (no file records). Used by agents project show for status display.

remove_index(resource_id) -> bool

Delete all index data for a resource. Returns True if records were deleted, False if no index existed.

cleanup_stale_indexing() -> int

Remove orphan INDEXING rows left by crashed processes. Should be called at application startup. Returns the number of stale rows removed.

Language Detection

Extension-based detection via detect_language(path). Supported mappings:

Extensions	Language
.py, .pyi, .pyx	python
.ts, .tsx	typescript
.js, .jsx, .mjs, .cjs	javascript
.rs	rust
.java	java
.kt, .kts	kotlin
.go	go
.c, .h	c
.cpp, .cc, .cxx, .hpp	cpp
.cs	csharp
.rb	ruby
.php	php
.swift	swift
.scala	scala
.r	r
.md, .mdx	markdown
.rst	restructuredtext
.json	json
.yaml, .yml	yaml
.toml	toml
.xml	xml
.html, .htm	html
.css, .scss	css
.sql	sql
.sh, .bash, .zsh	shell
.ps1	powershell
.dockerfile	dockerfile
.tf	terraform
.lua	lua
.zig	zig
.nim	nim
.ex, .exs	elixir
.erl	erlang
.hs	haskell
.ml, .mli	ocaml
.clj	clojure
.dart	dart
.v	v
.jl	julia
Makefile, makefile, GNUmakefile	makefile
Dockerfile, Dockerfile.*	dockerfile

Files with unrecognized extensions return "unknown".

Configuration

Indexing behaviour is controlled via the project's ContextConfig and ContextView:

Config Key	Model Field	Description
context.include_patterns	ContextConfig.include_patterns	Include globs
context.ignore_patterns	ContextConfig.ignore_patterns	Exclude globs
context.max_file_size	ContextConfig.max_file_size	Per-file size limit
context.max_total_size	ContextConfig.max_total_size	Total index size cap
context.indexing_strategy	ContextConfig.indexing_strategy	full_text or semantic
context.auto_refresh	ContextConfig.auto_refresh	Auto-refresh on access

Database Schema

Two tables are added:

repo_indexes

Column	Type	Constraints
index_id	String(26)	PK
resource_id	String(26)	NOT NULL, UNIQUE, INDEXED
indexed_at	String(40)	NOT NULL (ISO-8601 UTC)
file_count	Integer	NOT NULL, DEFAULT 0
token_estimate	Integer	NOT NULL, DEFAULT 0
primary_language	String(50)	NOT NULL, DEFAULT "unknown"
status	String(20)	NOT NULL, DEFAULT "pending", CHECK IN (pending, indexing, ready, stale, error)
error_message	Text	NULLABLE
created_at	String(40)	NOT NULL (ISO-8601 UTC)

indexed_files

Column	Type	Constraints
index_id	String(26)	PK (composite), FK -> repo_indexes.index_id ON DELETE CASCADE, INDEXED
path	String(1024)	PK (composite)
content_hash	String(64)	NOT NULL
token_count	Integer	NOT NULL, DEFAULT 0
size_bytes	Integer	NOT NULL, DEFAULT 0
language	String(50)	NOT NULL, DEFAULT "unknown"
last_modified	String(40)	NOT NULL

Spec References

• Lines 727-840: agents info repo indexing display
• Lines 2829-2916: project link-resource triggers indexing
• Lines 3322-3395: project show displays index status
• Lines 19719-19727: Project data model index fields
• Lines 28649-28664: context.* configuration keys