cleveragents-core/docs/reference/context_indexing.md

# Repository Indexing Service

The Repository Indexing Service scans linked repository resources, building a
persistent file-level index with language detection, content hashing, and token
estimation.  ACMS uses this index for efficient context assembly on projects
with 10K+ files.

## Architecture

```
resource_id + root_path
       |
       v
  ┌─── RepoIndexingService ─────┐
  │  walk filesystem            │
  │  apply include/exclude globs│
  │  enforce max_file_size      │
  │  enforce max_total_size     │
  │  SHA-256 content hashing    │
  │  extension language detect  │
  │  estimate token counts      │
  └──────────┬──────────────────┘
             v
  ┌─── SQLite persistence ──────┐
  │  repo_indexes table         │
  │  indexed_files table        │
  └─────────────────────────────┘
```

## Domain Models

All models are frozen Pydantic v2 with ULID identifiers and UTC datetimes.

### IndexStatus

Enum representing the state of an index:

| Value | Description |
|-------|-------------|
| `pending` | Index creation requested but not yet started |
| `indexing` | Filesystem walk in progress |
| `ready` | Index complete and available for queries |
| `error` | Indexing failed; see `error_message` |
| `stale` | Source files changed since last index |

### FileRecord

Per-file metadata stored during indexing:

| Field | Type | Description |
|-------|------|-------------|
| `path` | `str` | Relative path from the repository root |
| `content_hash` | `str` | SHA-256 hex digest of file contents |
| `token_count` | `int` | Estimated token count (`size_bytes // 4`) |
| `language` | `str` | Detected programming language |
| `size_bytes` | `int` | File size in bytes |
| `last_modified` | `datetime` | File modification timestamp (UTC) |

### IndexMetadata

Summary record for a repository index:

| Field | Type | Description |
|-------|------|-------------|
| `index_id` | `str` | ULID identifier for this index snapshot |
| `resource_id` | `str` | ULID of the linked resource |
| `indexed_at` | `datetime` | When indexing completed (UTC) |
| `file_count` | `int` | Total files in the index |
| `token_estimate` | `int` | Sum of all file token counts |
| `primary_language` | `str` | Most common language by token count (weighted) |
| `status` | `IndexStatus` | Current index state |
| `error_message` | `str | None` | Error details when `status == error` |

### RepoIndex

Composite object returned by index and refresh operations:

| Field | Type | Description |
|-------|------|-------------|
| `metadata` | `IndexMetadata` | Index summary |
| `files` | `tuple[FileRecord, ...]` | All indexed file records |

## Service API

### `RepoIndexingService(session_factory)`

Constructor.  Accepts a SQLAlchemy session factory (injected via DI container).

### `index_resource(resource_id, root_path, *, include_globs, exclude_globs, max_file_size, max_total_size) -> RepoIndex`

Full index of a filesystem tree.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `resource_id` | `str` | required | ULID of the resource |
| `root_path` | `str \| Path` | required | Absolute path to the repository root |
| `include_globs` | `tuple[str, ...]` | `()` | Only index files matching these globs (empty = all) |
| `exclude_globs` | `tuple[str, ...]` | `()` | Skip files matching these globs |
| `max_file_size` | `int \| None` | `None` | Skip files larger than this (bytes); `None` = no limit |
| `max_total_size` | `int \| None` | `None` | Stop indexing when cumulative size exceeds this; `None` = no limit |

Raises `ValueError` if `resource_id` is empty.  Raises `FileNotFoundError` if
`root_path` does not exist.

### `refresh_index(resource_id, root_path, **kwargs) -> RepoIndex`

Incremental refresh.  Compares content hashes against the existing index and
only re-processes changed files.  Falls back to a full index if no prior index
exists.  Accepts the same keyword arguments as `index_resource`.

### `get_index(resource_id) -> RepoIndex | None`

Retrieve a full index (metadata + file records) from the database.  Returns
`None` if no index exists.

### `get_index_status(resource_id) -> IndexMetadata | None`

Lightweight query returning only the metadata (no file records).  Used by
`agents project show` for status display.

### `remove_index(resource_id) -> bool`

Delete all index data for a resource.  Returns `True` if records were deleted,
`False` if no index existed.

### `cleanup_stale_indexing() -> int`

Remove orphan `INDEXING` rows left by crashed processes.  Should be called at
application startup.  Returns the number of stale rows removed.

## Language Detection

Extension-based detection via `detect_language(path)`.  Supported mappings:

| Extensions | Language |
|-----------|----------|
| `.py`, `.pyi`, `.pyx` | python |
| `.ts`, `.tsx` | typescript |
| `.js`, `.jsx`, `.mjs`, `.cjs` | javascript |
| `.rs` | rust |
| `.java` | java |
| `.kt`, `.kts` | kotlin |
| `.go` | go |
| `.c`, `.h` | c |
| `.cpp`, `.cc`, `.cxx`, `.hpp` | cpp |
| `.cs` | csharp |
| `.rb` | ruby |
| `.php` | php |
| `.swift` | swift |
| `.scala` | scala |
| `.r` | r |
| `.md`, `.mdx` | markdown |
| `.rst` | restructuredtext |
| `.json` | json |
| `.yaml`, `.yml` | yaml |
| `.toml` | toml |
| `.xml` | xml |
| `.html`, `.htm` | html |
| `.css`, `.scss` | css |
| `.sql` | sql |
| `.sh`, `.bash`, `.zsh` | shell |
| `.ps1` | powershell |
| `.dockerfile` | dockerfile |
| `.tf` | terraform |
| `.lua` | lua |
| `.zig` | zig |
| `.nim` | nim |
| `.ex`, `.exs` | elixir |
| `.erl` | erlang |
| `.hs` | haskell |
| `.ml`, `.mli` | ocaml |
| `.clj` | clojure |
| `.dart` | dart |
| `.v` | v |
| `.jl` | julia |
| `Makefile`, `makefile`, `GNUmakefile` | makefile |
| `Dockerfile`, `Dockerfile.*` | dockerfile |

Files with unrecognized extensions return `"unknown"`.

## Configuration

Indexing behaviour is controlled via the project's `ContextConfig` and
`ContextView`:

| Config Key | Model Field | Description |
|-----------|-------------|-------------|
| `context.include_patterns` | `ContextConfig.include_patterns` | Include globs |
| `context.ignore_patterns` | `ContextConfig.ignore_patterns` | Exclude globs |
| `context.max_file_size` | `ContextConfig.max_file_size` | Per-file size limit |
| `context.max_total_size` | `ContextConfig.max_total_size` | Total index size cap |
| `context.indexing_strategy` | `ContextConfig.indexing_strategy` | `full_text` or `semantic` |
| `context.auto_refresh` | `ContextConfig.auto_refresh` | Auto-refresh on access |

## Database Schema

Two tables are added:

### `repo_indexes`

| Column | Type | Constraints |
|--------|------|-------------|
| `index_id` | `String(26)` | PK |
| `resource_id` | `String(26)` | NOT NULL, UNIQUE, INDEXED |
| `indexed_at` | `String(40)` | NOT NULL (ISO-8601 UTC) |
| `file_count` | `Integer` | NOT NULL, DEFAULT 0 |
| `token_estimate` | `Integer` | NOT NULL, DEFAULT 0 |
| `primary_language` | `String(50)` | NOT NULL, DEFAULT "unknown" |
| `status` | `String(20)` | NOT NULL, DEFAULT "pending", CHECK IN (`pending`, `indexing`, `ready`, `stale`, `error`) |
| `error_message` | `Text` | NULLABLE |
| `created_at` | `String(40)` | NOT NULL (ISO-8601 UTC) |

### `indexed_files`

| Column | Type | Constraints |
|--------|------|-------------|
| `index_id` | `String(26)` | PK (composite), FK -> `repo_indexes.index_id` ON DELETE CASCADE, INDEXED |
| `path` | `String(1024)` | PK (composite) |
| `content_hash` | `String(64)` | NOT NULL |
| `token_count` | `Integer` | NOT NULL, DEFAULT 0 |
| `size_bytes` | `Integer` | NOT NULL, DEFAULT 0 |
| `language` | `String(50)` | NOT NULL, DEFAULT "unknown" |
| `last_modified` | `String(40)` | NOT NULL |

## Spec References

- Lines 727-840: `agents info` repo indexing display
- Lines 2829-2916: `project link-resource` triggers indexing
- Lines 3322-3395: `project show` displays index status
- Lines 19719-19727: Project data model index fields
- Lines 28649-28664: `context.*` configuration keys