Files
cleveragents-core/tools/_implementer_label_state.py
T
drew 2658deee94 feat(auto-agents): PR State Warmer substrate + supporting infra
Adds a long-lived sidecar (pr_state_warmer.py) that polls Forgejo's
/pulls endpoint every 30s and writes the full PR snapshot to a
shared SQLite store, eliminating the dispatcher's per-cycle
cold-cache stalls (24-30s rebuilds on flaky cycles) and the silent
50-PR pagination cap on the legacy single-page fetch.

Substrate
- tools/_pr_state_cache.py  — SQLite store with (owner, repo) PK,
  WAL mode, additive v2→v3 migration (comments_refreshed_updated_at),
  bounded fcntl.flock migration lock, threading.Lock for per-process
  init, @_with_reheal decorator (catches OperationalError no-such-
  table + DatabaseError corruption with file quarantine), atomic
  TEMP-table chunking for >32k seen-set, _normalize_updated_at to
  canonicalize Forgejo tz-marker drift
- tools/pr_state_warmer.py — poll/upsert/vanish/comments-refresh
  loop with fcntl.flock singleton (rejects second warmer), bounded
  comments-refresh cap, persistent deferral via SQL pending query,
  PermissionError-tolerant lock setup, cold-start log suppression
- tools/_pr_classification_cache.py — three-layer fall-through
  (warmer cache → list cache → live fetch) with staleness gate
  (PR_STATE_WARMER_STALE_AFTER_S floored at 30s in prod)

Comments cache hardening
- Bot-filter at write time drops bot status/claim/release/sentinel
  while preserving **Implementation Attempt** markers (94.6%
  reduction on bot-heavy PRs like #30's 19k-comment thread)
- _normalize_since_cursor strips microsecond precision before
  building ?since= query (fixes the live-observed Forgejo HTTP 422
  bug on PRs #25 + #28); handles uppercase Z, lowercase z, ±HH:MM
  offsets (including non-zero like +05:30), naive ISO
- Lazy migration of legacy null-key by_author entries on _read_cache
- _newest_cursor walks tail-back skipping malformed entries

Supporting infrastructure (cumulative dmpipeline-v2 work)
- Telemetry server: SSE live tail, run-sessions enumeration,
  cost/token tracking, app.js UI rewrite with collapsible sections
- MCP servers (mcp_ci_server, mcp_forgejo_server, mcp_git_server,
  mcp_handoff_server, mcp_graphify_server) for opencode worker
  context access
- Live log writer (tools/live_log_writer.py) — SSE-streaming
  dispatcher event log
- Tier-dispatcher escalation flow with prompts trimmed for budget
- Shared bot-logins resolver (tools/_bot_logins.py) replacing two
  drift-prone copies
- token_usage_audit.py for opencode cost analysis

Tests
- 2259 passing across 65 changed/new files
- New suites: test_pr_state_cache, test_pr_state_warmer,
  test_pr_state_warmer_integration, test_pr_classification_cache,
  test_pr_list_cache_backoff, test_mcp_* (5 servers),
  test_live_log_writer_sse, test_telemetry_run_sessions,
  test_review_post_ready_label
- Test_pr_comments_cache expanded with bot-filter coverage,
  cursor-normalization regression pins, format-drift, atomicity,
  failed-comments-not-stamped (silent-data-loss class)
- Parametrized @_with_reheal coverage across 7 wrapped APIs
- Real fault-inject atomicity test for chunked mark_vanished path
  via Connection wrapper class
- Subprocess-based singleton flock test (cross-process contract)
- Event-driven SIGTERM-mid-poll test (no fixed-sleep flake)

Architecture notes
- Schema v3 migration is additive (ALTER ADD COLUMN); v0/v1 still
  need destructive rebuild because pre-v2 column shape lacks
  owner/repo. Cross-process drop-table-ping-pong prevented by the
  fcntl migration lock + per-process _initialized flag.
- Comments-refresh deferral is persistent via
  comments_refreshed_updated_at column — survives warmer restart,
  picks up next cycle even if PR didn't change again. Replaces
  in-memory changed_numbers list.
- Rollback path: PR_STATE_WARMER_PREFER=0 bypasses the warmer
  cache and reverts to live-fetch behavior. PR_STATE_CACHE_DISABLE=1
  short-circuits the warmer process at startup.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 10:02:53 -04:00

189 lines
7.2 KiB
Python

"""Attempt-tier label mutation for the implementer dispatcher.
Owns the ``auto/last-attempt-tier-{0,1,2}`` label set that the
in-cycle escalation loop uses for both operator visibility AND
cross-cycle resumption seeding.
**Dual role (updated 2026-05-12).** Earlier iterations of the plan
described these labels as "observability-only" with crash recovery
restarting at Tier 0. The shipped design uses them more
ambitiously: in addition to being mutated for the Forgejo UI, the
dispatcher reads them at cycle start via
:func:`dispatch_implementer._read_start_tier_from_labels` to seed
``start_tier = min(labeled_tier + 1, max_tier)``. A PR that
exhausted Tier 1 in a previous cycle resumes at Tier 2 instead of
re-running Tier 0, saving a worker session per crash-recovery
cycle. See
``docs/development/implementer-in-cycle-escalation-plan.md`` §
"Cross-cycle resumption".
The labels are NOT a strict source of truth — a stale label
(operator-set, crash before clear) can mis-seed the next cycle by
one tier. The cost is bounded: worst case is one extra higher-tier
attempt, vs. the savings of skipping a known-failed tier.
Two operations:
- :func:`apply_attempt_label` — set the label for the just-launched
tier, clear the labels for any other tier. Called at every tier
boundary so the Forgejo UI shows "this PR is currently being
worked on at tier N" while the worker session is in flight.
- :func:`clear_attempt_labels` — remove all three labels. Called
when the cycle ends successfully (the labels are stale after
resolution) and as a safety sweep on cycle exit paths where the
dispatcher may have left a stale tier label behind.
Both are **best-effort**: if a label has not been provisioned in
Forgejo, the underlying HTTP call returns a 404 and
:func:`_claim_runtime._add_label` / ``_remove_label`` already return
``False`` (their helpers swallow the label-not-found state). We log
WARNING and proceed; escalation continues without label
observability.
"""
from __future__ import annotations
import logging
import sys
from pathlib import Path
from typing import Any
_TOOLS_DIR = str(Path(__file__).resolve().parent)
if _TOOLS_DIR not in sys.path:
sys.path.insert(0, _TOOLS_DIR)
from _loader import ( # noqa: E402 type: ignore[import-not-found]
load_sibling as _load_sibling,
)
_claim_runtime = _load_sibling("_claim_runtime", "_claim_runtime.py")
_logger = logging.getLogger("implementer_label_state")
# The four labels owned by the in-cycle escalation loop. Kept here
# rather than in _claim_runtime so the constant lives next to the
# code that mutates it.
#
# 2026-05-16 expansion: tier -1 (``tier-min``) was added after the
# run-15 doom-spiral inspection traced repeated cycles re-picking
# tier-min on PR #30 to the absence of a tier-min label slot. With
# no label to mark "this PR already failed at tier-min", every
# fresh dispatcher cycle treated the PR as a true first attempt and
# the estimator was free to re-pick tier-min. Adding the label
# closes the loop for the deterministic-walk path
# (``dispatch_implementer._read_start_tier_from_labels``).
#
# The name suffix is ``-min`` not ``--1`` because Forgejo label
# names with double hyphens are ugly in the UI and the parsing
# logic in the dispatcher handles the ``-min`` literal explicitly
# (see ``_read_start_tier_from_labels``).
ATTEMPT_TIER_LABELS_BY_TIER: dict[int, str] = {
-1: "auto/last-attempt-tier-min",
0: "auto/last-attempt-tier-0",
1: "auto/last-attempt-tier-1",
2: "auto/last-attempt-tier-2",
}
# Convenience tuple — preserved for callers that iterate (e.g.
# ``apply_attempt_label`` clearing siblings, ``clear_attempt_labels``
# wiping the lot). Order is "lowest tier first" so iteration is
# predictable.
ATTEMPT_TIER_LABELS: tuple[str, ...] = tuple(
ATTEMPT_TIER_LABELS_BY_TIER[t]
for t in sorted(ATTEMPT_TIER_LABELS_BY_TIER)
)
# Reverse lookup. Used by the dispatcher's label-read path to
# validate parsed integers and by tests for round-trip checks.
TIER_BY_LABEL: dict[str, int] = {
v: k for k, v in ATTEMPT_TIER_LABELS_BY_TIER.items()
}
def _label_for_tier(tier: int) -> str:
"""Map a tier integer to its label name. Raises ``ValueError``
on out-of-range — callers should never reach here with an
invalid tier (the escalation loop's max_tier guard catches it
earlier).
Accepts the full configured range including -1 (``tier-min``).
"""
if tier not in ATTEMPT_TIER_LABELS_BY_TIER:
valid = sorted(ATTEMPT_TIER_LABELS_BY_TIER)
raise ValueError(
f"tier {tier} out of range; valid tiers are {valid}"
)
return ATTEMPT_TIER_LABELS_BY_TIER[tier]
def apply_attempt_label(
cfg: Any, pr_number: int, tier: int
) -> dict[str, Any]:
"""Apply ``auto/last-attempt-tier-{tier}`` and clear the other
tier labels from the PR.
Side-effect-only — returns a small status dict for the cycle
archive so an operator can verify the mutation fired:
.. code-block:: python
{
"applied": "auto/last-attempt-tier-1",
"cleared": ["auto/last-attempt-tier-0"],
"skipped_provisioning_missing": False,
}
On label-not-provisioned (Forgejo returns 404 from the label
lookup), the dispatcher logs WARNING and proceeds — escalation
is unaffected, but operators won't see the per-tier label in
the Forgejo UI until the label provisioner is run. The
``skipped_provisioning_missing`` flag in the return value gives
the cycle archive a clean signal an operator can grep for.
"""
target_label = _label_for_tier(tier)
applied = _claim_runtime._add_label(pr_number, target_label, cfg)
if not applied:
_logger.warning(
"apply_attempt_label: failed to add %r to PR #%s "
"(label may not be provisioned in Forgejo — run the "
"label provisioner; escalation continues without "
"per-tier label observability)",
target_label,
pr_number,
)
cleared: list[str] = []
for label in ATTEMPT_TIER_LABELS:
if label == target_label:
continue
# Best-effort: _remove_label returns False on
# label-not-provisioned. We don't log the soft fail — it's
# the symmetric case of "add" failing and the same warning
# above already surfaces the root cause.
if _claim_runtime._remove_label(pr_number, label, cfg):
cleared.append(label)
return {
"applied": target_label if applied else None,
"cleared": cleared,
"skipped_provisioning_missing": not applied,
}
def clear_attempt_labels(cfg: Any, pr_number: int) -> dict[str, Any]:
"""Remove all three ``auto/last-attempt-tier-*`` labels from the
PR.
Called on successful cycle exit (the stale labels are no longer
informative — the PR's status moves on) and as a safety sweep
on END_CYCLE / EXHAUSTED paths.
Returns ``{"cleared": [...]}`` — the list of labels actually
removed (vs. labels that were already absent). Best-effort:
failures log WARNING + proceed.
"""
cleared: list[str] = []
for label in ATTEMPT_TIER_LABELS:
if _claim_runtime._remove_label(pr_number, label, cfg):
cleared.append(label)
return {"cleared": cleared}