cleveragents-core/docs/showcase/cli-tools/repo-indexing-workflows.md

Repository Indexing Workflows with CleverAgents

Overview

CleverAgents provides a built-in repository indexing engine that scans your codebase and builds a searchable token-aware index. The agents repo command group lets you index any registered resource, check its status, and trigger incremental or full re-indexes — all from the command line. This guide walks through the complete workflow with real output captured from a live installation.

Prerequisites

• CleverAgents installed (pip install cleveragents or from source with uv sync)
• Python 3.12 or higher
• At least one resource registered in the resource registry (see Resource and Skill Management)

What You'll Learn

• How to register a git-checkout or fs-directory resource for indexing
• How to run an incremental index (fast, only changed files)
• How to run a full re-index (complete rescan)
• How to check index status in both rich text and JSON formats
• How to use --timeout-seconds to bound long-running indexes
• How to integrate index status into scripts with --format json

---

Part 1: Explore the agents repo Command Group

Step 1: View Available Commands

$ agents repo --help

Actual Output:

 Usage: agents repo [OPTIONS] COMMAND [ARGS]...

 Repository indexing management

╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help          Show this message and exit.                                  │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ index   Index or re-index a repository resource.                             │
│ status  Show indexing status for a repository resource.                      │
╰──────────────────────────────────────────────────────────────────────────────╯

Step 2: Explore agents repo index Options

$ agents repo index --help

Actual Output:

 Usage: agents repo index [OPTIONS] RESOURCE_NAME

 Index or re-index a repository resource.

 By default performs an incremental refresh (only changed files).
 Use ``--full`` to force a complete re-index.

 Examples:
     agents repo index local/my-repo
     agents repo index local/my-repo --full
     agents repo index local/my-repo --format json

╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ *    resource_name      TEXT  Resource name or ULID to index (e.g.,          │
│                               local/my-repo)                                 │
│                               [required]                                     │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --full                            Force full re-index (not incremental)      │
│ --timeout-seconds          FLOAT  Abort indexing when elapsed runtime        │
│                                   exceeds this many seconds                  │
│ --format           -f      TEXT   Output format: text or json (default:      │
│                                   text)                                      │
│                                   [default: text]                            │
│ --help                            Show this message and exit.                │
╰──────────────────────────────────────────────────────────────────────────────╯

Step 3: Explore agents repo status Options

$ agents repo status --help

Actual Output:

 Usage: agents repo status [OPTIONS] RESOURCE_NAME

 Show indexing status for a repository resource.

 Displays index metadata including file count, token estimate,
 primary language, status, and last indexed timestamp.

 Examples:
     agents repo status local/my-repo
     agents repo status local/my-repo --format json

╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ *    resource_name      TEXT  Resource name or ULID to check status (e.g.,   │
│                               local/my-repo)                                 │
│                               [required]                                     │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --format  -f      TEXT  Output format: text or json (default: text)          │
│                         [default: text]                                      │
│ --help                  Show this message and exit.                          │
╰──────────────────────────────────────────────────────────────────────────────╯

---

Part 2: Complete End-to-End Workflow

This section demonstrates the full lifecycle: registering a resource, indexing it, checking its status, and running a full re-index.

Step 1: Register a Resource

Before indexing, you need a registered resource. Use agents resource add to register a git-checkout resource pointing to your repository:

$ agents resource add git-checkout local/src-only \
    --path /app/src \
    --description "CleverAgents source code only"

Actual Output:

Added resource: local/src-only (id: 01KNM6K5B0E39GK5YXSHGZ4783)

Tip: The resource name uses a namespace/identifier format. The local/ namespace is the conventional namespace for resources on the local machine. You can also use agents resource add fs-directory for non-git directories.

Step 2: Check Status Before Indexing

Before the first index run, the resource has no index:

$ agents repo status local/src-only

Actual Output:

No index found for resource: local/src-only

This is the expected response for a freshly registered resource that has never been indexed.

Step 3: Run an Incremental Index

The default index mode is incremental — it only processes files that have changed since the last run (or all files on the first run):

$ agents repo index local/src-only

Actual Output:

Incremental refresh complete: local/src-only
╭──────────────── Index Result ────────────────╮
│ Status: ready                                │
│ Files: 504                                   │
│ Tokens: 1,467,634                            │
│ Language: python                             │
│ Indexed at: 2026-04-07T14:47:34.444813+00:00 │
╰──────────────────────────────────────────────╯

The index completed in under 6 seconds for a 504-file Python codebase, producing a token estimate of ~1.5M tokens. The primary language is automatically detected.

Step 4: Check Status After Indexing

$ agents repo status local/src-only

Actual Output:

Index status for: local/src-only
╭──────────────── Index Status ────────────────╮
│ Status: ready                                │
│ Files: 504                                   │
│ Tokens: 1,467,634                            │
│ Language: python                             │
│ Indexed at: 2026-04-07T14:47:34.444813+00:00 │
╰──────────────────────────────────────────────╯

The status panel shows:

• Status: ready — the index is complete and available
• Files: total number of indexed files
• Tokens: estimated token count across all files
• Language: auto-detected primary language
• Indexed at: ISO 8601 timestamp of the last index run

Step 5: Run a Full Re-Index

Use --full to force a complete rescan of all files, regardless of modification timestamps:

$ agents repo index local/src-only --full

Actual Output:

Full index complete: local/src-only
╭──────────────── Index Result ────────────────╮
│ Status: ready                                │
│ Files: 504                                   │
│ Tokens: 1,467,634                            │
│ Language: python                             │
│ Indexed at: 2026-04-07T14:47:50.537996+00:00 │
╰──────────────────────────────────────────────╯

The output header changes from Incremental refresh complete to Full index complete to confirm the mode used.

Step 6: Remove the Resource (Cleanup)

$ agents resource remove local/src-only --yes

Actual Output:

Removed resource: local/src-only

---

Part 3: JSON Output for Scripting

All agents repo commands support --format json for machine-readable output. This is ideal for CI pipelines, monitoring scripts, or integrating with other tools.

Check Status as JSON

$ agents repo status local/my-repo --format json

Actual Output:

{
  "command": "",
  "status": "ok",
  "exit_code": 0,
  "data": {
    "resource": "local/my-repo",
    "resource_id": "01KNKJY4X9K7ZDNEK7M6TKJTDG",
    "index_id": "01KNM6DB6B96BSQ8HN8D5P37PK",
    "status": "ready",
    "file_count": 19136,
    "token_estimate": 210001290,
    "primary_language": "html",
    "indexed_at": "2026-04-07T14:44:25.513344+00:00"
  },
  "timing": {
    "duration_ms": 0
  },
  "messages": [
    {
      "level": "ok",
      "text": "ok"
    }
  ]
}

Run Incremental Index and Capture JSON

$ agents repo index local/my-repo --format json

Actual Output:

{
  "command": "",
  "status": "ok",
  "exit_code": 0,
  "data": {
    "resource": "local/my-repo",
    "resource_id": "01KNKJY4X9K7ZDNEK7M6TKJTDG",
    "index_id": "01KNM6DB6B96BSQ8HN8D5P37PK",
    "status": "ready",
    "file_count": 19142,
    "token_estimate": 210003925,
    "primary_language": "html",
    "indexed_at": "2026-04-07T14:45:41.571648+00:00",
    "mode": "incremental"
  },
  "timing": {
    "duration_ms": 0
  },
  "messages": [
    {
      "level": "ok",
      "text": "ok"
    }
  ]
}

Note the "mode": "incremental" field in the index result — this distinguishes incremental from full re-index runs in JSON output.

Run Full Re-Index and Capture JSON

$ agents repo index local/my-repo --full --format json

Actual Output:

{
  "command": "",
  "status": "ok",
  "exit_code": 0,
  "data": {
    "resource": "local/my-repo",
    "resource_id": "01KNKJY4X9K7ZDNEK7M6TKJTDG",
    "index_id": "01KNM6GZ0292Z8D1HH8HQHVC9J",
    "status": "ready",
    "file_count": 19145,
    "token_estimate": 210012364,
    "primary_language": "html",
    "indexed_at": "2026-04-07T14:46:14.289238+00:00",
    "mode": "full"
  },
  "timing": {
    "duration_ms": 0
  },
  "messages": [
    {
      "level": "ok",
      "text": "ok"
    }
  ]
}

The "mode": "full" field confirms a complete re-index was performed. Note that the index_id changes between full re-indexes, while incremental runs update the same index record.

---

Part 4: Using --timeout-seconds

For large repositories, you can set a time budget for the indexing operation. If the index exceeds the timeout, it aborts gracefully:

$ agents repo index local/my-repo --timeout-seconds 60

Actual Output (completed within timeout):

Incremental refresh complete: local/my-repo
╭──────────────── Index Result ────────────────╮
│ Status: ready                                │
│ Files: 19147                                 │
│ Tokens: 210,012,517                          │
│ Language: html                               │
│ Indexed at: 2026-04-07T14:46:33.496394+00:00 │
╰──────────────────────────────────────────────╯

If the timeout is exceeded, the command exits with code 1 and prints:

Error: Indexing timed out after <N> seconds

This is useful in CI environments where you want to bound the indexing step.

---

Part 5: Error Handling

Resource Not Found

If you specify a resource that doesn't exist in the registry:

$ agents repo status nonexistent/repo

Actual Output:

Error: Resource not found: nonexistent/repo

Exit code: 1

$ agents repo index nonexistent/repo

Actual Output:

Error: Resource not found: nonexistent/repo

Exit code: 1

These errors are clean and scriptable — check the exit code to detect failures in automation pipelines.

---

Part 6: Indexing a Large Repository

For reference, here is the status of a large repository (the full CleverAgents codebase including all build artifacts and documentation):

$ agents repo status local/my-repo

Actual Output:

Index status for: local/my-repo
╭──────────────── Index Status ────────────────╮
│ Status: ready                                │
│ Files: 19136                                 │
│ Tokens: 210,001,290                          │
│ Language: html                               │
│ Indexed at: 2026-04-07T14:44:25.513344+00:00 │
╰──────────────────────────────────────────────╯

A 19,000+ file repository with 210M tokens indexes incrementally in ~10 seconds.

---

Quick Reference

Command	Description
agents repo index <resource>	Incremental index (only changed files)
agents repo index <resource> --full	Full re-index (all files)
agents repo index <resource> --timeout-seconds N	Index with time budget
agents repo index <resource> --format json	Index with JSON output
agents repo status <resource>	Show index status (rich text)
agents repo status <resource> --format json	Show index status (JSON)

Resource Name Format

Resources use a namespace/identifier format:

• local/my-repo — a resource in the local namespace
• local/src-only — another local resource
• You can also use the resource ULID directly (e.g., 01KNKJY4X9K7ZDNEK7M6TKJTDG)

Index Status Values

Status	Meaning
ready	Index is complete and available
indexing	Index is currently being built
error	Index failed (check error_message field in JSON output)

Supported Resource Types for Indexing

The repo indexing commands work with any resource that has a filesystem path:

• git-checkout — local git repositories (recommended)
• fs-directory — any local filesystem directory

---

Key Takeaways

1. Incremental by default — agents repo index only processes changed files, making it fast enough to run on every commit.

2. Auto-detects primary language — the indexer automatically identifies the dominant programming language in the repository.

3. Token-aware — the index tracks token estimates, which is critical for AI context window management.

4. JSON output for automation — --format json makes it easy to integrate index status into CI pipelines, dashboards, or monitoring scripts.

5. Graceful timeout — --timeout-seconds lets you set a time budget for large repositories without crashing the pipeline.

6. Clean error messages — resource-not-found errors exit with code 1 and print a clear message, making them easy to handle in scripts.

---

Try It Yourself

# 1. Register your project repository
agents resource add git-checkout local/my-project \
    --path /path/to/your/repo \
    --description "My project"

# 2. Check initial status (no index yet)
agents repo status local/my-project

# 3. Run the first index
agents repo index local/my-project

# 4. Check status after indexing
agents repo status local/my-project

# 5. Make some changes to your code, then run incremental refresh
agents repo index local/my-project

# 6. Force a full re-index
agents repo index local/my-project --full

# 7. Get machine-readable status for scripting
agents repo status local/my-project --format json | python3 -c \
    "import sys, json; d=json.load(sys.stdin)['data']; \
     print(f\"Files: {d['file_count']}, Tokens: {d['token_estimate']:,}\")"

---

This example was automatically generated and verified by the CleverAgents UAT system. Feature area: Repo indexing workflows | Test cycle: 1

---

Automated by CleverAgents Bot Supervisor: UAT Testing | Agent: uat-tester