cleveragents-core/docs/modules/uko-provenance.md

# UKO Provenance Tracking

**Package:** `cleveragents.application.services` (UKOIndexer, UKOGraphPersistence)
**Introduced:** v3.8.0 (issue #891)

This module guide covers the provenance tracking and temporal versioning
capabilities added to the UKO runtime in v3.8.0.  Provenance metadata
enables audit trails, point-in-time queries, and historical analysis of
the knowledge graph.

For the full UKO runtime API reference, see
[`docs/reference/uko_runtime.md`](../reference/uko_runtime.md).
For the temporal data model, see
[`docs/reference/temporal_data_model.md`](../reference/temporal_data_model.md).

---

## Purpose

Every typed triple produced by `UKOIndexer.index_graph()` now carries three
provenance fields:

| Field | OWL Property | Type | Description |
|-------|-------------|------|-------------|
| `sourceResource` | `uko:sourceResource` | `str` (ULID) | The CleverAgents resource that produced this triple |
| `validFrom` | `uko:validFrom` | `datetime` (UTC) | When this triple version became valid |
| `isCurrent` | `uko:isCurrent` | `bool` | Whether this is the current (non-superseded) version |

These fields are defined in UKO Layer 0 (see
[`docs/reference/uko.md`](../reference/uko.md#layer-0--universal-foundation-uko)).

---

## How Provenance Is Attached

When `UKOIndexer.index_graph()` runs, it calls `attach_provenance()` on
every triple before storing it in the graph backend:

```python
# Conceptual flow (simplified)
triples = analyzer.analyze(content, resource_uri)
implicit = inference_engine.infer(triples)
all_triples = triples + implicit

for triple in all_triples:
    provenanced = ProvenancedTriple(
        triple=triple,
        provenance=ProvenanceMetadata(
            source_resource=resource.id,
            source_path=resource.location,
            valid_from=datetime.now(UTC),
            is_current=True,
        ),
    )
    graph_backend.add_triple(project, provenanced)
```

---

## Revision Chain

A revision chain tracks the full history of ontology state across indexing
runs.  When a resource is re-indexed:

1. All existing triples for that resource are marked `isCurrent=False` and
   `validUntil` is set to the re-indexing timestamp.
2. New triples are created with `isCurrent=True` and a fresh `validFrom`.
3. A new revision entry is appended to the chain, linked to the previous
   revision.

This enables point-in-time queries: "what did the knowledge graph look like
at timestamp T?"

```
Revision 1 (2026-03-01)  →  Revision 2 (2026-03-15)  →  Revision 3 (2026-04-01)
   isCurrent=False            isCurrent=False              isCurrent=True
   validUntil=2026-03-15      validUntil=2026-04-01        validUntil=None
```

---

## Querying Provenance

### Current Triples Only

```python
from cleveragents.application.services.uko_query_interface import UKOQueryInterface

query = UKOQueryInterface(graph_backend=backend, project="local/my-app")

# Classify a resource using only current triples
result = query.classify_resource("01HXYZ...")
# result.layer, result.primary_type, result.all_types, result.implicit_relations
```

### Historical Triples (Point-in-Time)

```python
from datetime import datetime, timezone
from cleveragents.application.services.uko_persistence import UKOGraphPersistence

persistence = UKOGraphPersistence(
    graph_backend=backend,
    project="local/my-app",
    persist_path=Path("~/.config/cleveragents/uko-graph.json").expanduser(),
)

# Query triples valid at a specific point in time
historical = persistence.query_at_time(
    timestamp=datetime(2026, 3, 15, tzinfo=timezone.utc)
)
```

### Triples by Source Resource

```python
# Get all triples (current and historical) from a specific resource
triples = graph_backend.query(
    project="local/my-app",
    sparql="""
    SELECT ?s ?p ?o WHERE {
        ?s ?p ?o .
        ?s uko:sourceResource "01HXYZ..." .
    }
    """
)
```

---

## Persistence Format

`UKOGraphPersistence` serialises the full graph (including provenance) to
JSON.  The format includes all triples with their provenance metadata:

```json
{
    "version": "1",
    "project": "local/my-app",
    "revision": 3,
    "triples": [
        {
            "subject": "uko-py:class/AuthService",
            "predicate": "rdf:type",
            "object": "uko-oo:Class",
            "provenance": {
                "source_resource": "01HXYZ...",
                "source_path": "src/auth/service.py",
                "source_range": "15:1-87:0",
                "valid_from": "2026-04-01T10:00:00Z",
                "valid_until": null,
                "is_current": true
            }
        }
    ]
}
```

---

## BDD Coverage

The provenance tracking feature is covered by 47 BDD scenarios in
`features/uko_provenance.feature`:

- Typed triple provenance fields (`sourceResource`, `validFrom`, `isCurrent`)
- Temporal queries (point-in-time, current-only, historical)
- Revision chain creation and traversal
- Graph persistence with provenance (save/restore round-trip)
- Re-indexing marks previous triples as historical

---

## Gotchas

- **Naive datetimes are rejected.** All `validFrom` and `validUntil` values
  must be timezone-aware UTC datetimes. Passing a naive `datetime` raises
  `ValueError`.
- **Re-indexing is not atomic.** If `index_graph()` fails partway through,
  some triples may be marked historical while their replacements were not
  yet written. The indexer logs a warning in this case; re-running
  `reindex_resource()` will repair the state.
- **In-memory backend has no persistence.** The `InMemoryGraphIndexBackend`
  used in tests does not persist across process restarts. Use
  `UKOGraphPersistence` with a file backend in production.

---

## Related Documentation

- [UKO Runtime Services](../reference/uko_runtime.md) — full API reference
- [UKO Ontology](../reference/uko.md) — four-layer ontology and provenance property definitions
- [UKO Indexer](../reference/uko_indexer.md) — `ProvenanceMetadata` and `ProvenancedTriple` types
- [Temporal Data Model](../reference/temporal_data_model.md) — `TemporalNode`, `RevisionChain`, tier storage
- [ACMS](../reference/acms.md) — how provenance feeds into context assembly