cleveragents-core/docs/reference/uko_indexer.md

UKO Indexer — Real-time Index Synchronization

The UKO Indexer orchestrates the analysis of resources into UKO triples and simultaneously indexes content into text, vector, and graph backends. It implements the index lifecycle (add/change/remove/maintenance) with graceful degradation when optional backends are unavailable.

Based on docs/specification.md > ACMS > Real-time Index Synchronization and Custom Index Backends.

Architecture

Resource ──► AnalyzerRegistry.get_for_resource()
                 │
                 ▼
            ContentReader.read_content()
                 │
                 ▼
           Analyzer.analyze(content, resource_uri)
                 │
                 ▼
           attach_provenance(triples, resource, timestamp)
                 │
                 ├──► GraphIndexBackend.add_triple()   (required)
                 ├──► TextIndexBackend.index_document() (optional — graceful degradation)
                 └──► VectorIndexBackend.index_embedding() (optional — graceful degradation)

UKOIndexer

Main service class that produces UKO triples from resources and indexes them into all available backends.

Constructor

Parameter	Type	Required	Description
analyzer_registry	AnalyzerRegistry	Yes	Registry of domain analyzers
graph_backend	GraphIndexBackend	Yes	Backend for UKO triple storage
text_backend	TextIndexBackend | None	No	Backend for full-text indexing
vector_backend	VectorIndexBackend | None	No	Backend for embedding indexing
content_reader	ContentReader | None	No	Reader for resource content (default: LocationContentReader)
lifecycle_hook	IndexLifecycleHook | None	No	Callback for lifecycle events (default: DefaultLifecycleHook)
max_triples	int	No	Maximum triples per resource (default: 50_000). Must be positive.

Methods

Method	Signature	Returns	Description
index_resource	(resource: Resource, *, project: str)	IndexResult	Full indexing pipeline
remove_resource	(resource_id: str, *, project: str)	None	Remove from all indices
reindex_resource	(resource: Resource, *, project: str)	IndexResult	Remove + re-index

Properties

Property	Type	Description
analyzer_registry	AnalyzerRegistry	The analyzer registry
indexed_resource_count	int	Number of currently indexed resources
has_text_backend	bool	Whether text backend is available
has_vector_backend	bool	Whether vector backend is available

Graceful Degradation

If text_backend or vector_backend is None, the corresponding indexing step is silently skipped. The graph_backend is always required. Error counts from individual backend failures are recorded in IndexResult.errors without aborting the entire indexing operation.

Index Backends (Write-side Protocols)

These are write-side protocols for indexing. They are distinct from the existing read-side query protocols (TextBackend, VectorBackend, GraphBackend) in backends.py.

TextIndexBackend

Method	Signature	Returns
index_document	(project: str, doc_id: str, content: str, metadata: dict[str, str])	IndexedDocument
search	(project: str, query: str, *, limit: int = 20)	list[SearchResult]
remove_document	(project: str, doc_id: str)	None
rebuild_index	(project: str)	None

VectorIndexBackend

Method	Signature	Returns
index_embedding	(project: str, doc_id: str, embedding: list[float], metadata: dict[str, str])	None
search_similar	(project: str, query_embedding: list[float], limit: int = 20, min_relevance: float = 0.0)	list[SearchResult]
remove_embedding	(project: str, doc_id: str)	None

GraphIndexBackend

Method	Signature	Returns
add_triple	(project: str, subject: str, predicate: str, obj: str)	None
query	(project: str, sparql: str)	list[dict[str, str]]
remove_triples	(project: str, subject: str | None, predicate: str | None, obj: str | None)	None

Result Types

IndexedDocument

Field	Type	Description
project	str	Namespaced project name
doc_id	str	Document identifier
char_count	int	Character count (non-negative, default 0)

SearchResult

Field	Type	Description
doc_id	str	Document identifier (non-empty)
content	str	Matched text snippet
score	float	Relevance score in [0.0, 1.0]
metadata	dict[str, str]	Backend-specific metadata

Provenance

ProvenanceMetadata

Tracks the origin and validity of indexed triples.

Field	Type	Description
source_resource	str	Resource ULID that produced the triple
source_path	str	File path of the source resource
source_range	str	Line range (e.g. "10-25"), empty if whole file
valid_from	datetime	When the triple was indexed (defaults to now(UTC))
is_current	bool	Whether the triple is still current (default: True)

ProvenancedTriple

Wrapper combining a UKOTriple with its ProvenanceMetadata.

Field	Type	Description
triple	UKOTriple	The underlying UKO triple
provenance	ProvenanceMetadata	Origin and validity metadata

IndexResult

Summary returned by index_resource() and reindex_resource().

Field	Type	Description
resource_id	str	ULID of the indexed resource
triple_count	int	Number of triples produced
text_docs_indexed	int	Text documents indexed (0 or 1)
embeddings_indexed	int	Embeddings indexed (0 or 1)
analyzer_domain	str	Domain of the analyzer used
errors	tuple[str, ...]	Non-fatal error messages

Protocols

ContentReader

Protocol for reading resource content. Decouples the indexer from the filesystem.

Method	Signature	Returns
read_content	(resource: Resource)	str

Default implementation: LocationContentReader — reads from resource.location via the local filesystem.

IndexLifecycleHook

Callback protocol for index lifecycle events.

Method	Signature	Returns
on_indexed	(result: IndexResult)	None
on_removed	(resource_id: str, project: str)	None
on_error	(resource_id: str, error: str)	None

Default implementation: DefaultLifecycleHook — logs events via structlog.

In-Memory Stubs

For testing, three in-memory stub implementations are provided:

• InMemoryTextIndexBackend — dict-based full-text storage with substring search
• InMemoryVectorIndexBackend — dict-based embedding storage (no real similarity)
• InMemoryGraphIndexBackend — list-based triple storage with pattern matching

ResourceFileWatcher

Monitors filesystem paths via watchdog and triggers re-indexing callbacks on file changes with configurable debouncing.

Constructor

Parameter	Type	Required	Description
indexer	UKOIndexer	Yes	Indexer for re-index on change
project	str	Yes	Project namespace
debounce_seconds	float	No	Debounce delay (default: 0.5)
on_change	Callable[[str, str, FileChangeType], None] | None	No	Optional change callback
event_bus	EventBus | None	No	Optional event bus for RESOURCE_MODIFIED emission

FileChangeType

StrEnum with values: CREATED, MODIFIED, DELETED, MOVED.

Methods

Method	Signature	Returns	Description
watch	(resource_id: str, path: Path)	None	Start watching a file path for a resource
unwatch	(resource_id: str)	None	Stop watching a resource
shutdown	()	None	Stop all observers and clean up

Known Limitations

• Placeholder vector embedding: The vector backend receives a constant [1.0] placeholder instead of a real embedding. Real embedding model integration is tracked as a follow-up to issue #578.

Spec References

• Real-time Index Synchronization: specification.md > ACMS > Real-time Index Synchronization
• Custom Index Backends: specification.md > ACMS > Custom Index Backends
• Domain Analyzers: specification.md > ACMS > Domain Analyzers
• Config keys: specification.md > Configuration Service > Index Backend Keys