cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity
Files
master
cleveragents-core/docs/reference/uko_indexer.md
T
hamza.khyari 1e606553d4
CI / benchmark-publish (pull_request) Has been skipped
Details
CI / lint (pull_request) Successful in 15s
Details
CI / build (pull_request) Successful in 15s
Details
CI / quality (pull_request) Successful in 18s
Details
CI / security (pull_request) Successful in 35s
Details
CI / typecheck (pull_request) Successful in 38s
Details
CI / unit_tests (pull_request) Successful in 2m32s
Details
CI / integration_tests (pull_request) Successful in 3m18s
Details
CI / docker (pull_request) Successful in 49s
Details
CI / coverage (pull_request) Successful in 5m29s
Details
CI / benchmark-regression (pull_request) Successful in 33m48s
Details
feat(acms): implement Real-time Index Sync / UKOIndexer with pluggable analyzers 
Implement the UKOIndexer service that produces UKO triples from resources
using pluggable domain-specific analyzers, wraps each triple with provenance
metadata, and simultaneously indexes into text, vector, and graph backends.

Key design decisions and components:

- UKOIndexer orchestrates the full index lifecycle: add_resource,
  update_resource (remove-then-add), remove_resource, and maintenance
  triggers. Each operation fires lifecycle hooks (on_indexed, on_removed,
  on_error) so callers can observe progress.

- Analyzer selection is pluggable via ContentAnalyzer protocol. The indexer
  accepts a registry mapping resource types to analyzers. PythonAnalyzer
  and MarkdownAnalyzer are provided as built-in implementations.

- LocationContentReader protocol abstracts file I/O with a base_dir
  parameter for path-traversal prevention (post-resolve validation rejects
  paths escaping the base directory and non-regular files).

- UKOTriple model includes a @model_validator ensuring at least one of
  object_uri or object_value is populated, preventing empty triples at
  construction time.

- Triple removal uses scoped deletion via uko:sourceResource predicate to
  avoid shared-subject collision — only triples originating from the
  specific resource are removed, not all triples for a shared subject.

- _resource_subjects.pop is deferred until after all backend removal
  operations succeed, preventing inconsistent state on partial failure.

- analyzer.analyze() is wrapped in try/except so that analyzer errors
  produce an IndexResult with error details rather than propagating
  exceptions to callers.

- All lifecycle hook calls are guarded via _fire_on_indexed,
  _fire_on_removed, and _fire_on_error helpers that catch and log hook
  exceptions without disrupting the indexing pipeline.

- max_triples parameter (default 50,000) bounds analyzer output size to
  prevent runaway resource consumption.

- ResourceFileWatcher monitors filesystem paths via watchdog and triggers
  re-indexing callbacks on file changes with configurable debouncing.
  Emits RESOURCE_MODIFIED domain events via EventBus when file changes
  are detected. Debounce timers coalesce rapid edits into a single
  callback invocation. Thread-safe design with daemon threads for clean
  shutdown.

- SearchResult.__post_init__ validates score is in [0.0, 1.0], correctly
  rejecting NaN values.

- Placeholder embedding uses [1.0] instead of [float(len(content))] to
  avoid leaking content size information.

- isinstance check on graph_backend ensures GraphIndexBackend protocol
  compliance at runtime.

- Test doubles extracted to features/mocks/uko_indexer_mocks.py for reuse
  across BDD steps and Robot helpers.

Spec reference: Architecture > ACMS > Real-time Index Synchronization
(specification.md lines ~43205-43300).

ISSUES CLOSED: #578
2026-03-11 00:40:07 +00:00

231 lines
8.9 KiB
Markdown
Raw Permalink Blame History

# UKO Indexer — Real-time Index Synchronization
The UKO Indexer orchestrates the analysis of resources into UKO triples and
simultaneously indexes content into text, vector, and graph backends. It
implements the index lifecycle (add/change/remove/maintenance) with graceful
degradation when optional backends are unavailable.
Based on `docs/specification.md` > ACMS > Real-time Index
Synchronization and Custom Index Backends.
## Architecture
```
Resource ──► AnalyzerRegistry.get_for_resource()
│
▼
ContentReader.read_content()
│
▼
Analyzer.analyze(content, resource_uri)
│
▼
attach_provenance(triples, resource, timestamp)
│
├──► GraphIndexBackend.add_triple() (required)
├──► TextIndexBackend.index_document() (optional — graceful degradation)
└──► VectorIndexBackend.index_embedding() (optional — graceful degradation)
```
## UKOIndexer
Main service class that produces UKO triples from resources and indexes them
into all available backends.
### Constructor
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `analyzer_registry` | `AnalyzerRegistry` | Yes | Registry of domain analyzers |
| `graph_backend` | `GraphIndexBackend` | Yes | Backend for UKO triple storage |
| `text_backend` | `TextIndexBackend \| None` | No | Backend for full-text indexing |
| `vector_backend` | `VectorIndexBackend \| None` | No | Backend for embedding indexing |
| `content_reader` | `ContentReader \| None` | No | Reader for resource content (default: `LocationContentReader`) |
| `lifecycle_hook` | `IndexLifecycleHook \| None` | No | Callback for lifecycle events (default: `DefaultLifecycleHook`) |
| `max_triples` | `int` | No | Maximum triples per resource (default: `50_000`). Must be positive. |
### Methods
| Method | Signature | Returns | Description |
|--------|-----------|---------|-------------|
| `index_resource` | `(resource: Resource, *, project: str)` | `IndexResult` | Full indexing pipeline |
| `remove_resource` | `(resource_id: str, *, project: str)` | `None` | Remove from all indices |
| `reindex_resource` | `(resource: Resource, *, project: str)` | `IndexResult` | Remove + re-index |
### Properties
| Property | Type | Description |
|----------|------|-------------|
| `analyzer_registry` | `AnalyzerRegistry` | The analyzer registry |
| `indexed_resource_count` | `int` | Number of currently indexed resources |
| `has_text_backend` | `bool` | Whether text backend is available |
| `has_vector_backend` | `bool` | Whether vector backend is available |
### Graceful Degradation
If `text_backend` or `vector_backend` is `None`, the corresponding indexing
step is silently skipped. The `graph_backend` is always required. Error
counts from individual backend failures are recorded in `IndexResult.errors`
without aborting the entire indexing operation.
## Index Backends (Write-side Protocols)
These are **write-side** protocols for indexing. They are distinct from the
existing read-side query protocols (`TextBackend`, `VectorBackend`,
`GraphBackend`) in `backends.py`.
### TextIndexBackend
| Method | Signature | Returns |
|--------|-----------|---------|
| `index_document` | `(project: str, doc_id: str, content: str, metadata: dict[str, str])` | `IndexedDocument` |
| `search` | `(project: str, query: str, *, limit: int = 20)` | `list[SearchResult]` |
| `remove_document` | `(project: str, doc_id: str)` | `None` |
| `rebuild_index` | `(project: str)` | `None` |
### VectorIndexBackend
| Method | Signature | Returns |
|--------|-----------|---------|
| `index_embedding` | `(project: str, doc_id: str, embedding: list[float], metadata: dict[str, str])` | `None` |
| `search_similar` | `(project: str, query_embedding: list[float], limit: int = 20, min_relevance: float = 0.0)` | `list[SearchResult]` |
| `remove_embedding` | `(project: str, doc_id: str)` | `None` |
### GraphIndexBackend
| Method | Signature | Returns |
|--------|-----------|---------|
| `add_triple` | `(project: str, subject: str, predicate: str, obj: str)` | `None` |
| `query` | `(project: str, sparql: str)` | `list[dict[str, str]]` |
| `remove_triples` | `(project: str, subject: str \| None, predicate: str \| None, obj: str \| None)` | `None` |
## Result Types
### IndexedDocument
| Field | Type | Description |
|-------|------|-------------|
| `project` | `str` | Namespaced project name |
| `doc_id` | `str` | Document identifier |
| `char_count` | `int` | Character count (non-negative, default 0) |
### SearchResult
| Field | Type | Description |
|-------|------|-------------|
| `doc_id` | `str` | Document identifier (non-empty) |
| `content` | `str` | Matched text snippet |
| `score` | `float` | Relevance score in `[0.0, 1.0]` |
| `metadata` | `dict[str, str]` | Backend-specific metadata |
## Provenance
### ProvenanceMetadata
Tracks the origin and validity of indexed triples.
| Field | Type | Description |
|-------|------|-------------|
| `source_resource` | `str` | Resource ULID that produced the triple |
| `source_path` | `str` | File path of the source resource |
| `source_range` | `str` | Line range (e.g. `"10-25"`), empty if whole file |
| `valid_from` | `datetime` | When the triple was indexed (defaults to `now(UTC)`) |
| `is_current` | `bool` | Whether the triple is still current (default: `True`) |
### ProvenancedTriple
Wrapper combining a `UKOTriple` with its `ProvenanceMetadata`.
| Field | Type | Description |
|-------|------|-------------|
| `triple` | `UKOTriple` | The underlying UKO triple |
| `provenance` | `ProvenanceMetadata` | Origin and validity metadata |
### IndexResult
Summary returned by `index_resource()` and `reindex_resource()`.
| Field | Type | Description |
|-------|------|-------------|
| `resource_id` | `str` | ULID of the indexed resource |
| `triple_count` | `int` | Number of triples produced |
| `text_docs_indexed` | `int` | Text documents indexed (0 or 1) |
| `embeddings_indexed` | `int` | Embeddings indexed (0 or 1) |
| `analyzer_domain` | `str` | Domain of the analyzer used |
| `errors` | `tuple[str, ...]` | Non-fatal error messages |
## Protocols
### ContentReader
Protocol for reading resource content. Decouples the indexer from the
filesystem.
| Method | Signature | Returns |
|--------|-----------|---------|
| `read_content` | `(resource: Resource)` | `str` |
**Default implementation**: `LocationContentReader` — reads from
`resource.location` via the local filesystem.
### IndexLifecycleHook
Callback protocol for index lifecycle events.
| Method | Signature | Returns |
|--------|-----------|---------|
| `on_indexed` | `(result: IndexResult)` | `None` |
| `on_removed` | `(resource_id: str, project: str)` | `None` |
| `on_error` | `(resource_id: str, error: str)` | `None` |
**Default implementation**: `DefaultLifecycleHook` — logs events via
structlog.
## In-Memory Stubs
For testing, three in-memory stub implementations are provided:
- `InMemoryTextIndexBackend` — dict-based full-text storage with substring search
- `InMemoryVectorIndexBackend` — dict-based embedding storage (no real similarity)
- `InMemoryGraphIndexBackend` — list-based triple storage with pattern matching
## ResourceFileWatcher
Monitors filesystem paths via `watchdog` and triggers re-indexing callbacks
on file changes with configurable debouncing.
### Constructor
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `indexer` | `UKOIndexer` | Yes | Indexer for re-index on change |
| `project` | `str` | Yes | Project namespace |
| `debounce_seconds` | `float` | No | Debounce delay (default: `0.5`) |
| `on_change` | `Callable[[str, str, FileChangeType], None] \| None` | No | Optional change callback |
| `event_bus` | `EventBus \| None` | No | Optional event bus for `RESOURCE_MODIFIED` emission |
### FileChangeType
`StrEnum` with values: `CREATED`, `MODIFIED`, `DELETED`, `MOVED`.
### Methods
| Method | Signature | Returns | Description |
|--------|-----------|---------|-------------|
| `watch` | `(resource_id: str, path: Path)` | `None` | Start watching a file path for a resource |
| `unwatch` | `(resource_id: str)` | `None` | Stop watching a resource |
| `shutdown` | `()` | `None` | Stop all observers and clean up |
## Known Limitations
- **Placeholder vector embedding**: The vector backend receives a
constant `[1.0]` placeholder instead of a real embedding. Real
embedding model integration is tracked as a follow-up to issue #578.
## Spec References
- Real-time Index Synchronization: `specification.md` > ACMS > Real-time Index Synchronization
- Custom Index Backends: `specification.md` > ACMS > Custom Index Backends
- Domain Analyzers: `specification.md` > ACMS > Domain Analyzers
- Config keys: `specification.md` > Configuration Service > Index Backend Keys
Reference in New Issue View Git Blame Copy Permalink