diff --git a/docs/modules/uko-provenance.md b/docs/modules/uko-provenance.md new file mode 100644 index 000000000..e18608d56 --- /dev/null +++ b/docs/modules/uko-provenance.md @@ -0,0 +1,195 @@ +# UKO Provenance Tracking + +**Package:** `cleveragents.application.services` (UKOIndexer, UKOGraphPersistence) +**Introduced:** v3.8.0 (issue #891) + +This module guide covers the provenance tracking and temporal versioning +capabilities added to the UKO runtime in v3.8.0. Provenance metadata +enables audit trails, point-in-time queries, and historical analysis of +the knowledge graph. + +For the full UKO runtime API reference, see +[`docs/reference/uko_runtime.md`](../reference/uko_runtime.md). +For the temporal data model, see +[`docs/reference/temporal_data_model.md`](../reference/temporal_data_model.md). + +--- + +## Purpose + +Every typed triple produced by `UKOIndexer.index_graph()` now carries three +provenance fields: + +| Field | OWL Property | Type | Description | +|-------|-------------|------|-------------| +| `sourceResource` | `uko:sourceResource` | `str` (ULID) | The CleverAgents resource that produced this triple | +| `validFrom` | `uko:validFrom` | `datetime` (UTC) | When this triple version became valid | +| `isCurrent` | `uko:isCurrent` | `bool` | Whether this is the current (non-superseded) version | + +These fields are defined in UKO Layer 0 (see +[`docs/reference/uko.md`](../reference/uko.md#layer-0--universal-foundation-uko)). + +--- + +## How Provenance Is Attached + +When `UKOIndexer.index_graph()` runs, it calls `attach_provenance()` on +every triple before storing it in the graph backend: + +```python +# Conceptual flow (simplified) +triples = analyzer.analyze(content, resource_uri) +implicit = inference_engine.infer(triples) +all_triples = triples + implicit + +for triple in all_triples: + provenanced = ProvenancedTriple( + triple=triple, + provenance=ProvenanceMetadata( + source_resource=resource.id, + source_path=resource.location, + valid_from=datetime.now(UTC), + is_current=True, + ), + ) + graph_backend.add_triple(project, provenanced) +``` + +--- + +## Revision Chain + +A revision chain tracks the full history of ontology state across indexing +runs. When a resource is re-indexed: + +1. All existing triples for that resource are marked `isCurrent=False` and + `validUntil` is set to the re-indexing timestamp. +2. New triples are created with `isCurrent=True` and a fresh `validFrom`. +3. A new revision entry is appended to the chain, linked to the previous + revision. + +This enables point-in-time queries: "what did the knowledge graph look like +at timestamp T?" + +``` +Revision 1 (2026-03-01) → Revision 2 (2026-03-15) → Revision 3 (2026-04-01) + isCurrent=False isCurrent=False isCurrent=True + validUntil=2026-03-15 validUntil=2026-04-01 validUntil=None +``` + +--- + +## Querying Provenance + +### Current Triples Only + +```python +from cleveragents.application.services.uko_query_interface import UKOQueryInterface + +query = UKOQueryInterface(graph_backend=backend, project="local/my-app") + +# Classify a resource using only current triples +result = query.classify_resource("01HXYZ...") +# result.layer, result.primary_type, result.all_types, result.implicit_relations +``` + +### Historical Triples (Point-in-Time) + +```python +from datetime import datetime, timezone +from cleveragents.application.services.uko_persistence import UKOGraphPersistence + +persistence = UKOGraphPersistence( + graph_backend=backend, + project="local/my-app", + persist_path=Path("~/.config/cleveragents/uko-graph.json").expanduser(), +) + +# Query triples valid at a specific point in time +historical = persistence.query_at_time( + timestamp=datetime(2026, 3, 15, tzinfo=timezone.utc) +) +``` + +### Triples by Source Resource + +```python +# Get all triples (current and historical) from a specific resource +triples = graph_backend.query( + project="local/my-app", + sparql=""" + SELECT ?s ?p ?o WHERE { + ?s ?p ?o . + ?s uko:sourceResource "01HXYZ..." . + } + """ +) +``` + +--- + +## Persistence Format + +`UKOGraphPersistence` serialises the full graph (including provenance) to +JSON. The format includes all triples with their provenance metadata: + +```json +{ + "version": "1", + "project": "local/my-app", + "revision": 3, + "triples": [ + { + "subject": "uko-py:class/AuthService", + "predicate": "rdf:type", + "object": "uko-oo:Class", + "provenance": { + "source_resource": "01HXYZ...", + "source_path": "src/auth/service.py", + "source_range": "15:1-87:0", + "valid_from": "2026-04-01T10:00:00Z", + "valid_until": null, + "is_current": true + } + } + ] +} +``` + +--- + +## BDD Coverage + +The provenance tracking feature is covered by 47 BDD scenarios in +`features/uko_provenance.feature`: + +- Typed triple provenance fields (`sourceResource`, `validFrom`, `isCurrent`) +- Temporal queries (point-in-time, current-only, historical) +- Revision chain creation and traversal +- Graph persistence with provenance (save/restore round-trip) +- Re-indexing marks previous triples as historical + +--- + +## Gotchas + +- **Naive datetimes are rejected.** All `validFrom` and `validUntil` values + must be timezone-aware UTC datetimes. Passing a naive `datetime` raises + `ValueError`. +- **Re-indexing is not atomic.** If `index_graph()` fails partway through, + some triples may be marked historical while their replacements were not + yet written. The indexer logs a warning in this case; re-running + `reindex_resource()` will repair the state. +- **In-memory backend has no persistence.** The `InMemoryGraphIndexBackend` + used in tests does not persist across process restarts. Use + `UKOGraphPersistence` with a file backend in production. + +--- + +## Related Documentation + +- [UKO Runtime Services](../reference/uko_runtime.md) — full API reference +- [UKO Ontology](../reference/uko.md) — four-layer ontology and provenance property definitions +- [UKO Indexer](../reference/uko_indexer.md) — `ProvenanceMetadata` and `ProvenancedTriple` types +- [Temporal Data Model](../reference/temporal_data_model.md) — `TemporalNode`, `RevisionChain`, tier storage +- [ACMS](../reference/acms.md) — how provenance feeds into context assembly diff --git a/docs/reference/decision_correction.md b/docs/reference/decision_correction.md index 5a9c600c7..157e39ece 100644 --- a/docs/reference/decision_correction.md +++ b/docs/reference/decision_correction.md @@ -97,3 +97,98 @@ PENDING → ANALYZING → EXECUTING → APPLIED | `list_corrections()` | List corrections (optional plan_id filter) | | `list_attempts()` | List execution attempts for a correction | | `cancel_correction()` | Cancel a pending/analyzing correction | + +--- + +## Correction Attempts (v3.8.0+) + +Each execution of a correction is tracked as a `CorrectionAttemptRecord`. +Multiple attempts may exist for a single correction (e.g. if a first attempt +fails and the operator retries). + +### `CorrectionAttemptRecord` + +**Module:** `cleveragents.domain.models.core.correction_attempt` + +| Field | Type | Description | +|-------|------|-------------| +| `id` | `str` | ULID of this attempt | +| `correction_id` | `str` | Parent correction ULID | +| `plan_id` | `str` | Plan this correction targets | +| `original_decision_id` | `str` | Decision being corrected | +| `new_decision_id` | `str \| None` | Replacement decision (set on success) | +| `mode` | `CorrectionMode` | `revert` or `append` | +| `state` | `CorrectionAttemptState` | Current state (see lifecycle below) | +| `guidance` | `str` | Operator guidance text (1–10,000 chars) | +| `created_at` | `datetime` | UTC creation timestamp | +| `completed_at` | `datetime \| None` | UTC completion timestamp (terminal states only) | +| `archived_artifacts_path` | `str \| None` | Path to archived artifacts (revert mode) | + +### `CorrectionAttemptState` Lifecycle + +```text +pending → executing → complete + → failed +``` + +Valid transitions: + +| From | To | Notes | +|------|----|-------| +| `pending` | `executing` | Attempt begins execution | +| `executing` | `complete` | Correction applied successfully | +| `executing` | `failed` | Correction failed; `completed_at` auto-set | + +Terminal states (`complete`, `failed`) cannot transition further. +Self-transitions are rejected. + +### `CorrectionAttemptRepository` + +**Module:** `cleveragents.infrastructure.repositories.correction_attempt` + +| Method | Description | +|--------|-------------| +| `create(record)` | Persist a new attempt; raises `DatabaseError` on FK violation | +| `get(attempt_id)` | Retrieve by ULID; raises `ResourceNotFoundError` if absent | +| `list_by_plan(plan_id)` | List all attempts for a plan (ordered by `created_at`) | +| `update_state(attempt_id, new_state, *, new_decision_id, completed_at, archived_artifacts_path)` | Transition state; auto-sets `completed_at` for terminal states | +| `delete(attempt_id)` | Hard-delete an attempt record | + +### Usage Example + +```python +from cleveragents.domain.models.core.correction_attempt import ( + CorrectionAttemptRecord, + CorrectionAttemptState, +) +from cleveragents.domain.models.core.correction import CorrectionMode + +attempt = CorrectionAttemptRecord( + id="01HXYZ...", + correction_id="01HABC...", + plan_id="01HDEF...", + original_decision_id="01HGHI...", + mode=CorrectionMode.REVERT, + state=CorrectionAttemptState.PENDING, + guidance="Revert the database migration decision and retry with a safer approach.", +) + +# Transition to executing +repo.update_state(attempt.id, CorrectionAttemptState.EXECUTING) + +# Complete the attempt +repo.update_state( + attempt.id, + CorrectionAttemptState.COMPLETE, + new_decision_id="01HJKL...", + archived_artifacts_path="/var/cleveragents/archives/01HXYZ", +) +``` + +### Database Schema + +The `correction_attempts` table uses `RESTRICT` foreign keys for both +`original_decision_id` and `new_decision_id`, preserving the correction +audit trail when decisions are cleaned up. The ORM-level +`cascade="all, delete-orphan"` on `LifecyclePlanModel` ensures that +deleting a plan cascades to its correction attempts. diff --git a/docs/reference/plan_cli.md b/docs/reference/plan_cli.md index 17c7e5d83..dbabe08ad 100644 --- a/docs/reference/plan_cli.md +++ b/docs/reference/plan_cli.md @@ -2,6 +2,11 @@ The `agents plan` command group manages plans in the CleverAgents v3 plan lifecycle. +> **Note (v3.8.0+):** Mixing legacy plan commands with v3 plan workflows in the same +> session is no longer permitted. If you attempt to use legacy plan commands alongside +> v3 plan commands, the CLI will detect the conflict and surface a clear error message +> with migration guidance. Migrate fully to the v3 plan workflow before using `agents plan`. + ## Commands | Command | Description | diff --git a/docs/reference/uko_runtime.md b/docs/reference/uko_runtime.md index cb79c4808..9480a23f0 100644 --- a/docs/reference/uko_runtime.md +++ b/docs/reference/uko_runtime.md @@ -2,11 +2,15 @@ This document covers the three runtime services that operationalize the Universal Knowledge Ontology (UKO) within the ACMS context pipeline. -These services were introduced in v3.7.0 (issue #891) and implement the -runtime requirements from `docs/specification.md` §185, §20568, §43924. +These services were introduced in v3.7.0 (issue #891) and extended in +v3.8.0 (issue #891) with provenance tracking and temporal versioning. +They implement the runtime requirements from `docs/specification.md` +§185, §20568, §43924. For the ontology structure itself, see [`uko.md`](uko.md). For the indexer that populates the graph, see [`uko_indexer.md`](uko_indexer.md). +For the temporal data model, see [`temporal_data_model.md`](temporal_data_model.md). +For the dedicated provenance module guide, see [`../modules/uko-provenance.md`](../modules/uko-provenance.md). ## Overview @@ -17,13 +21,66 @@ UKOIndexer.index_graph() │ ├── UKOInferenceEngine.infer() ← adds implicit triples (confidence 0.7) │ + ├── ProvenanceMetadata attached ← sourceResource, validFrom, isCurrent (v3.8.0+) + │ └── GraphIndexBackend ← stores all triples (explicit + implicit) │ ├── UKOQueryInterface ← ACMS context strategies query here │ - └── UKOGraphPersistence ← serialise/restore across restarts + ├── UKOGraphPersistence ← serialise/restore across restarts + │ + └── RevisionChain ← tracks ontology state across indexing runs (v3.8.0+) ``` +## Provenance Tracking (v3.8.0+) + +Every typed triple produced by `UKOIndexer.index_graph()` now carries +provenance metadata. This enables temporal queries over historical UKO +snapshots and supports audit trails for context decisions. + +### Provenance Fields on Typed Triples + +| Field | OWL Property | Description | +|-------|-------------|-------------| +| `sourceResource` | `uko:sourceResource` | ULID of the CleverAgents resource that produced this triple | +| `validFrom` | `uko:validFrom` | UTC timestamp when this triple version became valid | +| `isCurrent` | `uko:isCurrent` | Whether this is the current (non-superseded) version of the triple | + +When a resource is re-indexed, the previous triples for that resource are +marked `isCurrent=False` and `validUntil` is set to the re-indexing +timestamp. New triples are created with `isCurrent=True` and a fresh +`validFrom`. + +### Revision Chain + +A revision chain tracks the full history of ontology state across indexing +runs. Each indexing run creates a new revision entry linked to the previous +one, enabling point-in-time queries: + +```python +from cleveragents.application.services.uko_persistence import UKOGraphPersistence + +persistence = UKOGraphPersistence( + graph_backend=backend, + project="local/my-app", + persist_path=Path("~/.config/cleveragents/uko-graph.json").expanduser(), +) + +# Query triples valid at a specific point in time +historical_triples = persistence.query_at_time( + timestamp=datetime(2026, 3, 15, tzinfo=timezone.utc) +) +``` + +### Temporal Query Patterns + +| Query | Description | +|-------|-------------| +| `isCurrent=True` | Current (hot-tier) triples only | +| `validFrom <= T < validUntil` | Triples valid at timestamp T | +| `sourceResource = ULID` | All triples from a specific resource | +| All triples | Full historical graph (cold-tier) | + --- ## UKOQueryInterface @@ -244,5 +301,7 @@ maps type URIs to layer numbers: - [UKO Ontology](uko.md) — four-layer ontology structure and URI conventions - [UKO Indexer](uko_indexer.md) — real-time index synchronization +- [Temporal Data Model](temporal_data_model.md) — revision-aware RDF and three-tier storage +- [UKO Provenance Module](../modules/uko-provenance.md) — provenance tracking and temporal versioning guide - [ACMS](acms.md) — Advanced Context Management System overview - [ACMS Fusion](acms_fusion.md) — Fragment Fusion pipeline (Phase 2 protocols)