docs: update and extend documentation for v3.8.0 changes

- docs/reference/plan_cli.md: add note about legacy/v3 plan workflow mixing being disallowed (introduced in v3.8.0, issue #1577) - docs/reference/uko_runtime.md: document v3.8.0 provenance tracking (sourceResource, validFrom, isCurrent on typed triples) and revision chain for temporal queries across indexing runs (issue #891) - docs/reference/decision_correction.md: add CorrectionAttemptRecord section documenting the correction_attempts table, state lifecycle, repository API, and usage examples (issue #920) - docs/modules/uko-provenance.md: new module guide for UKO provenance tracking covering purpose, how provenance is attached, revision chain mechanics, query patterns, persistence format, BDD coverage, and gotchas
2026-04-05 20:05:48 +00:00
parent bd5238f705
commit a0ac30d5de
4 changed files with 357 additions and 3 deletions
@@ -0,0 +1,195 @@
+# UKO Provenance Tracking
+
+**Package:** `cleveragents.application.services` (UKOIndexer, UKOGraphPersistence)
+**Introduced:** v3.8.0 (issue #891)
+
+This module guide covers the provenance tracking and temporal versioning
+capabilities added to the UKO runtime in v3.8.0.  Provenance metadata
+enables audit trails, point-in-time queries, and historical analysis of
+the knowledge graph.
+
+For the full UKO runtime API reference, see
+[`docs/reference/uko_runtime.md`](../reference/uko_runtime.md).
+For the temporal data model, see
+[`docs/reference/temporal_data_model.md`](../reference/temporal_data_model.md).
+
+---
+
+## Purpose
+
+Every typed triple produced by `UKOIndexer.index_graph()` now carries three
+provenance fields:
+
+| Field | OWL Property | Type | Description |
+|-------|-------------|------|-------------|
+| `sourceResource` | `uko:sourceResource` | `str` (ULID) | The CleverAgents resource that produced this triple |
+| `validFrom` | `uko:validFrom` | `datetime` (UTC) | When this triple version became valid |
+| `isCurrent` | `uko:isCurrent` | `bool` | Whether this is the current (non-superseded) version |
+
+These fields are defined in UKO Layer 0 (see
+[`docs/reference/uko.md`](../reference/uko.md#layer-0--universal-foundation-uko)).
+
+---
+
+## How Provenance Is Attached
+
+When `UKOIndexer.index_graph()` runs, it calls `attach_provenance()` on
+every triple before storing it in the graph backend:
+
+```python
+# Conceptual flow (simplified)
+triples = analyzer.analyze(content, resource_uri)
+implicit = inference_engine.infer(triples)
+all_triples = triples + implicit
+
+for triple in all_triples:
+    provenanced = ProvenancedTriple(
+        triple=triple,
+        provenance=ProvenanceMetadata(
+            source_resource=resource.id,
+            source_path=resource.location,
+            valid_from=datetime.now(UTC),
+            is_current=True,
+        ),
+    )
+    graph_backend.add_triple(project, provenanced)
+```
+
+---
+
+## Revision Chain
+
+A revision chain tracks the full history of ontology state across indexing
+runs.  When a resource is re-indexed:
+
+1. All existing triples for that resource are marked `isCurrent=False` and
+   `validUntil` is set to the re-indexing timestamp.
+2. New triples are created with `isCurrent=True` and a fresh `validFrom`.
+3. A new revision entry is appended to the chain, linked to the previous
+   revision.
+
+This enables point-in-time queries: "what did the knowledge graph look like
+at timestamp T?"
+
+```
+Revision 1 (2026-03-01)  →  Revision 2 (2026-03-15)  →  Revision 3 (2026-04-01)
+   isCurrent=False            isCurrent=False              isCurrent=True
+   validUntil=2026-03-15      validUntil=2026-04-01        validUntil=None
+```
+
+---
+
+## Querying Provenance
+
+### Current Triples Only
+
+```python
+from cleveragents.application.services.uko_query_interface import UKOQueryInterface
+
+query = UKOQueryInterface(graph_backend=backend, project="local/my-app")
+
+# Classify a resource using only current triples
+result = query.classify_resource("01HXYZ...")
+# result.layer, result.primary_type, result.all_types, result.implicit_relations
+```
+
+### Historical Triples (Point-in-Time)
+
+```python
+from datetime import datetime, timezone
+from cleveragents.application.services.uko_persistence import UKOGraphPersistence
+
+persistence = UKOGraphPersistence(
+    graph_backend=backend,
+    project="local/my-app",
+    persist_path=Path("~/.config/cleveragents/uko-graph.json").expanduser(),
+)
+
+# Query triples valid at a specific point in time
+historical = persistence.query_at_time(
+    timestamp=datetime(2026, 3, 15, tzinfo=timezone.utc)
+)
+```
+
+### Triples by Source Resource
+
+```python
+# Get all triples (current and historical) from a specific resource
+triples = graph_backend.query(
+    project="local/my-app",
+    sparql="""
+    SELECT ?s ?p ?o WHERE {
+        ?s ?p ?o .
+        ?s uko:sourceResource "01HXYZ..." .
+    }
+    """
+)
+```
+
+---
+
+## Persistence Format
+
+`UKOGraphPersistence` serialises the full graph (including provenance) to
+JSON.  The format includes all triples with their provenance metadata:
+
+```json
+{
+    "version": "1",
+    "project": "local/my-app",
+    "revision": 3,
+    "triples": [
+        {
+            "subject": "uko-py:class/AuthService",
+            "predicate": "rdf:type",
+            "object": "uko-oo:Class",
+            "provenance": {
+                "source_resource": "01HXYZ...",
+                "source_path": "src/auth/service.py",
+                "source_range": "15:1-87:0",
+                "valid_from": "2026-04-01T10:00:00Z",
+                "valid_until": null,
+                "is_current": true
+            }
+        }
+    ]
+}
+```
+
+---
+
+## BDD Coverage
+
+The provenance tracking feature is covered by 47 BDD scenarios in
+`features/uko_provenance.feature`:
+
+- Typed triple provenance fields (`sourceResource`, `validFrom`, `isCurrent`)
+- Temporal queries (point-in-time, current-only, historical)
+- Revision chain creation and traversal
+- Graph persistence with provenance (save/restore round-trip)
+- Re-indexing marks previous triples as historical
+
+---
+
+## Gotchas
+
+- **Naive datetimes are rejected.** All `validFrom` and `validUntil` values
+  must be timezone-aware UTC datetimes. Passing a naive `datetime` raises
+  `ValueError`.
+- **Re-indexing is not atomic.** If `index_graph()` fails partway through,
+  some triples may be marked historical while their replacements were not
+  yet written. The indexer logs a warning in this case; re-running
+  `reindex_resource()` will repair the state.
+- **In-memory backend has no persistence.** The `InMemoryGraphIndexBackend`
+  used in tests does not persist across process restarts. Use
+  `UKOGraphPersistence` with a file backend in production.
+
+---
+
+## Related Documentation
+
+- [UKO Runtime Services](../reference/uko_runtime.md) — full API reference
+- [UKO Ontology](../reference/uko.md) — four-layer ontology and provenance property definitions
+- [UKO Indexer](../reference/uko_indexer.md) — `ProvenanceMetadata` and `ProvenancedTriple` types
+- [Temporal Data Model](../reference/temporal_data_model.md) — `TemporalNode`, `RevisionChain`, tier storage
+- [ACMS](../reference/acms.md) — how provenance feeds into context assembly
@@ -97,3 +97,98 @@ PENDING → ANALYZING → EXECUTING → APPLIED
 | `list_corrections()`        | List corrections (optional plan_id filter)     |
 | `list_attempts()`           | List execution attempts for a correction       |
 | `cancel_correction()`       | Cancel a pending/analyzing correction          |
+
+---
+
+## Correction Attempts (v3.8.0+)
+
+Each execution of a correction is tracked as a `CorrectionAttemptRecord`.
+Multiple attempts may exist for a single correction (e.g. if a first attempt
+fails and the operator retries).
+
+### `CorrectionAttemptRecord`
+
+**Module:** `cleveragents.domain.models.core.correction_attempt`
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `id` | `str` | ULID of this attempt |
+| `correction_id` | `str` | Parent correction ULID |
+| `plan_id` | `str` | Plan this correction targets |
+| `original_decision_id` | `str` | Decision being corrected |
+| `new_decision_id` | `str \| None` | Replacement decision (set on success) |
+| `mode` | `CorrectionMode` | `revert` or `append` |
+| `state` | `CorrectionAttemptState` | Current state (see lifecycle below) |
+| `guidance` | `str` | Operator guidance text (1–10,000 chars) |
+| `created_at` | `datetime` | UTC creation timestamp |
+| `completed_at` | `datetime \| None` | UTC completion timestamp (terminal states only) |
+| `archived_artifacts_path` | `str \| None` | Path to archived artifacts (revert mode) |
+
+### `CorrectionAttemptState` Lifecycle
+
+```text
+pending → executing → complete
+                    → failed
+```
+
+Valid transitions:
+
+| From | To | Notes |
+|------|----|-------|
+| `pending` | `executing` | Attempt begins execution |
+| `executing` | `complete` | Correction applied successfully |
+| `executing` | `failed` | Correction failed; `completed_at` auto-set |
+
+Terminal states (`complete`, `failed`) cannot transition further.
+Self-transitions are rejected.
+
+### `CorrectionAttemptRepository`
+
+**Module:** `cleveragents.infrastructure.repositories.correction_attempt`
+
+| Method | Description |
+|--------|-------------|
+| `create(record)` | Persist a new attempt; raises `DatabaseError` on FK violation |
+| `get(attempt_id)` | Retrieve by ULID; raises `ResourceNotFoundError` if absent |
+| `list_by_plan(plan_id)` | List all attempts for a plan (ordered by `created_at`) |
+| `update_state(attempt_id, new_state, *, new_decision_id, completed_at, archived_artifacts_path)` | Transition state; auto-sets `completed_at` for terminal states |
+| `delete(attempt_id)` | Hard-delete an attempt record |
+
+### Usage Example
+
+```python
+from cleveragents.domain.models.core.correction_attempt import (
+    CorrectionAttemptRecord,
+    CorrectionAttemptState,
+)
+from cleveragents.domain.models.core.correction import CorrectionMode
+
+attempt = CorrectionAttemptRecord(
+    id="01HXYZ...",
+    correction_id="01HABC...",
+    plan_id="01HDEF...",
+    original_decision_id="01HGHI...",
+    mode=CorrectionMode.REVERT,
+    state=CorrectionAttemptState.PENDING,
+    guidance="Revert the database migration decision and retry with a safer approach.",
+)
+
+# Transition to executing
+repo.update_state(attempt.id, CorrectionAttemptState.EXECUTING)
+
+# Complete the attempt
+repo.update_state(
+    attempt.id,
+    CorrectionAttemptState.COMPLETE,
+    new_decision_id="01HJKL...",
+    archived_artifacts_path="/var/cleveragents/archives/01HXYZ",
+)
+```
+
+### Database Schema
+
+The `correction_attempts` table uses `RESTRICT` foreign keys for both
+`original_decision_id` and `new_decision_id`, preserving the correction
+audit trail when decisions are cleaned up. The ORM-level
+`cascade="all, delete-orphan"` on `LifecyclePlanModel` ensures that
+deleting a plan cascades to its correction attempts.
@@ -2,6 +2,11 @@

 The `agents plan` command group manages plans in the CleverAgents v3 plan lifecycle.

+> **Note (v3.8.0+):** Mixing legacy plan commands with v3 plan workflows in the same
+> session is no longer permitted. If you attempt to use legacy plan commands alongside
+> v3 plan commands, the CLI will detect the conflict and surface a clear error message
+> with migration guidance. Migrate fully to the v3 plan workflow before using `agents plan`.
+
 ## Commands

 | Command                        | Description                             |
@@ -2,11 +2,15 @@

 This document covers the three runtime services that operationalize the
 Universal Knowledge Ontology (UKO) within the ACMS context pipeline.
-These services were introduced in v3.7.0 (issue #891) and implement the
-runtime requirements from `docs/specification.md` §185, §20568, §43924.
+These services were introduced in v3.7.0 (issue #891) and extended in
+v3.8.0 (issue #891) with provenance tracking and temporal versioning.
+They implement the runtime requirements from `docs/specification.md`
+§185, §20568, §43924.

 For the ontology structure itself, see [`uko.md`](uko.md).
 For the indexer that populates the graph, see [`uko_indexer.md`](uko_indexer.md).
+For the temporal data model, see [`temporal_data_model.md`](temporal_data_model.md).
+For the dedicated provenance module guide, see [`../modules/uko-provenance.md`](../modules/uko-provenance.md).

 ## Overview

@@ -17,13 +21,66 @@ UKOIndexer.index_graph()
    │
    ├── UKOInferenceEngine.infer()   ← adds implicit triples (confidence 0.7)
    │
+    ├── ProvenanceMetadata attached  ← sourceResource, validFrom, isCurrent (v3.8.0+)
+    │
    └── GraphIndexBackend            ← stores all triples (explicit + implicit)
              │
              ├── UKOQueryInterface  ← ACMS context strategies query here
              │
-              └── UKOGraphPersistence ← serialise/restore across restarts
+              ├── UKOGraphPersistence ← serialise/restore across restarts
+              │
+              └── RevisionChain      ← tracks ontology state across indexing runs (v3.8.0+)
 ```

+## Provenance Tracking (v3.8.0+)
+
+Every typed triple produced by `UKOIndexer.index_graph()` now carries
+provenance metadata. This enables temporal queries over historical UKO
+snapshots and supports audit trails for context decisions.
+
+### Provenance Fields on Typed Triples
+
+| Field | OWL Property | Description |
+|-------|-------------|-------------|
+| `sourceResource` | `uko:sourceResource` | ULID of the CleverAgents resource that produced this triple |
+| `validFrom` | `uko:validFrom` | UTC timestamp when this triple version became valid |
+| `isCurrent` | `uko:isCurrent` | Whether this is the current (non-superseded) version of the triple |
+
+When a resource is re-indexed, the previous triples for that resource are
+marked `isCurrent=False` and `validUntil` is set to the re-indexing
+timestamp. New triples are created with `isCurrent=True` and a fresh
+`validFrom`.
+
+### Revision Chain
+
+A revision chain tracks the full history of ontology state across indexing
+runs. Each indexing run creates a new revision entry linked to the previous
+one, enabling point-in-time queries:
+
+```python
+from cleveragents.application.services.uko_persistence import UKOGraphPersistence
+
+persistence = UKOGraphPersistence(
+    graph_backend=backend,
+    project="local/my-app",
+    persist_path=Path("~/.config/cleveragents/uko-graph.json").expanduser(),
+)
+
+# Query triples valid at a specific point in time
+historical_triples = persistence.query_at_time(
+    timestamp=datetime(2026, 3, 15, tzinfo=timezone.utc)
+)
+```
+
+### Temporal Query Patterns
+
+| Query | Description |
+|-------|-------------|
+| `isCurrent=True` | Current (hot-tier) triples only |
+| `validFrom <= T < validUntil` | Triples valid at timestamp T |
+| `sourceResource = ULID` | All triples from a specific resource |
+| All triples | Full historical graph (cold-tier) |
+
 ---

 ## UKOQueryInterface
@@ -244,5 +301,7 @@ maps type URIs to layer numbers:

 - [UKO Ontology](uko.md) — four-layer ontology structure and URI conventions
 - [UKO Indexer](uko_indexer.md) — real-time index synchronization
+- [Temporal Data Model](temporal_data_model.md) — revision-aware RDF and three-tier storage
+- [UKO Provenance Module](../modules/uko-provenance.md) — provenance tracking and temporal versioning guide
 - [ACMS](acms.md) — Advanced Context Management System overview
 - [ACMS Fusion](acms_fusion.md) — Fragment Fusion pipeline (Phase 2 protocols)