2658deee94
Adds a long-lived sidecar (pr_state_warmer.py) that polls Forgejo's /pulls endpoint every 30s and writes the full PR snapshot to a shared SQLite store, eliminating the dispatcher's per-cycle cold-cache stalls (24-30s rebuilds on flaky cycles) and the silent 50-PR pagination cap on the legacy single-page fetch. Substrate - tools/_pr_state_cache.py — SQLite store with (owner, repo) PK, WAL mode, additive v2→v3 migration (comments_refreshed_updated_at), bounded fcntl.flock migration lock, threading.Lock for per-process init, @_with_reheal decorator (catches OperationalError no-such- table + DatabaseError corruption with file quarantine), atomic TEMP-table chunking for >32k seen-set, _normalize_updated_at to canonicalize Forgejo tz-marker drift - tools/pr_state_warmer.py — poll/upsert/vanish/comments-refresh loop with fcntl.flock singleton (rejects second warmer), bounded comments-refresh cap, persistent deferral via SQL pending query, PermissionError-tolerant lock setup, cold-start log suppression - tools/_pr_classification_cache.py — three-layer fall-through (warmer cache → list cache → live fetch) with staleness gate (PR_STATE_WARMER_STALE_AFTER_S floored at 30s in prod) Comments cache hardening - Bot-filter at write time drops bot status/claim/release/sentinel while preserving **Implementation Attempt** markers (94.6% reduction on bot-heavy PRs like #30's 19k-comment thread) - _normalize_since_cursor strips microsecond precision before building ?since= query (fixes the live-observed Forgejo HTTP 422 bug on PRs #25 + #28); handles uppercase Z, lowercase z, ±HH:MM offsets (including non-zero like +05:30), naive ISO - Lazy migration of legacy null-key by_author entries on _read_cache - _newest_cursor walks tail-back skipping malformed entries Supporting infrastructure (cumulative dmpipeline-v2 work) - Telemetry server: SSE live tail, run-sessions enumeration, cost/token tracking, app.js UI rewrite with collapsible sections - MCP servers (mcp_ci_server, mcp_forgejo_server, mcp_git_server, mcp_handoff_server, mcp_graphify_server) for opencode worker context access - Live log writer (tools/live_log_writer.py) — SSE-streaming dispatcher event log - Tier-dispatcher escalation flow with prompts trimmed for budget - Shared bot-logins resolver (tools/_bot_logins.py) replacing two drift-prone copies - token_usage_audit.py for opencode cost analysis Tests - 2259 passing across 65 changed/new files - New suites: test_pr_state_cache, test_pr_state_warmer, test_pr_state_warmer_integration, test_pr_classification_cache, test_pr_list_cache_backoff, test_mcp_* (5 servers), test_live_log_writer_sse, test_telemetry_run_sessions, test_review_post_ready_label - Test_pr_comments_cache expanded with bot-filter coverage, cursor-normalization regression pins, format-drift, atomicity, failed-comments-not-stamped (silent-data-loss class) - Parametrized @_with_reheal coverage across 7 wrapped APIs - Real fault-inject atomicity test for chunked mark_vanished path via Connection wrapper class - Subprocess-based singleton flock test (cross-process contract) - Event-driven SIGTERM-mid-poll test (no fixed-sleep flake) Architecture notes - Schema v3 migration is additive (ALTER ADD COLUMN); v0/v1 still need destructive rebuild because pre-v2 column shape lacks owner/repo. Cross-process drop-table-ping-pong prevented by the fcntl migration lock + per-process _initialized flag. - Comments-refresh deferral is persistent via comments_refreshed_updated_at column — survives warmer restart, picks up next cycle even if PR didn't change again. Replaces in-memory changed_numbers list. - Rollback path: PR_STATE_WARMER_PREFER=0 bypasses the warmer cache and reverts to live-fetch behavior. PR_STATE_CACHE_DISABLE=1 short-circuits the warmer process at startup. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
189 lines
7.2 KiB
Python
189 lines
7.2 KiB
Python
"""Attempt-tier label mutation for the implementer dispatcher.
|
|
|
|
Owns the ``auto/last-attempt-tier-{0,1,2}`` label set that the
|
|
in-cycle escalation loop uses for both operator visibility AND
|
|
cross-cycle resumption seeding.
|
|
|
|
**Dual role (updated 2026-05-12).** Earlier iterations of the plan
|
|
described these labels as "observability-only" with crash recovery
|
|
restarting at Tier 0. The shipped design uses them more
|
|
ambitiously: in addition to being mutated for the Forgejo UI, the
|
|
dispatcher reads them at cycle start via
|
|
:func:`dispatch_implementer._read_start_tier_from_labels` to seed
|
|
``start_tier = min(labeled_tier + 1, max_tier)``. A PR that
|
|
exhausted Tier 1 in a previous cycle resumes at Tier 2 instead of
|
|
re-running Tier 0, saving a worker session per crash-recovery
|
|
cycle. See
|
|
``docs/development/implementer-in-cycle-escalation-plan.md`` §
|
|
"Cross-cycle resumption".
|
|
|
|
The labels are NOT a strict source of truth — a stale label
|
|
(operator-set, crash before clear) can mis-seed the next cycle by
|
|
one tier. The cost is bounded: worst case is one extra higher-tier
|
|
attempt, vs. the savings of skipping a known-failed tier.
|
|
|
|
Two operations:
|
|
|
|
- :func:`apply_attempt_label` — set the label for the just-launched
|
|
tier, clear the labels for any other tier. Called at every tier
|
|
boundary so the Forgejo UI shows "this PR is currently being
|
|
worked on at tier N" while the worker session is in flight.
|
|
- :func:`clear_attempt_labels` — remove all three labels. Called
|
|
when the cycle ends successfully (the labels are stale after
|
|
resolution) and as a safety sweep on cycle exit paths where the
|
|
dispatcher may have left a stale tier label behind.
|
|
|
|
Both are **best-effort**: if a label has not been provisioned in
|
|
Forgejo, the underlying HTTP call returns a 404 and
|
|
:func:`_claim_runtime._add_label` / ``_remove_label`` already return
|
|
``False`` (their helpers swallow the label-not-found state). We log
|
|
WARNING and proceed; escalation continues without label
|
|
observability.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
import sys
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
_TOOLS_DIR = str(Path(__file__).resolve().parent)
|
|
if _TOOLS_DIR not in sys.path:
|
|
sys.path.insert(0, _TOOLS_DIR)
|
|
from _loader import ( # noqa: E402 type: ignore[import-not-found]
|
|
load_sibling as _load_sibling,
|
|
)
|
|
|
|
_claim_runtime = _load_sibling("_claim_runtime", "_claim_runtime.py")
|
|
|
|
_logger = logging.getLogger("implementer_label_state")
|
|
|
|
|
|
# The four labels owned by the in-cycle escalation loop. Kept here
|
|
# rather than in _claim_runtime so the constant lives next to the
|
|
# code that mutates it.
|
|
#
|
|
# 2026-05-16 expansion: tier -1 (``tier-min``) was added after the
|
|
# run-15 doom-spiral inspection traced repeated cycles re-picking
|
|
# tier-min on PR #30 to the absence of a tier-min label slot. With
|
|
# no label to mark "this PR already failed at tier-min", every
|
|
# fresh dispatcher cycle treated the PR as a true first attempt and
|
|
# the estimator was free to re-pick tier-min. Adding the label
|
|
# closes the loop for the deterministic-walk path
|
|
# (``dispatch_implementer._read_start_tier_from_labels``).
|
|
#
|
|
# The name suffix is ``-min`` not ``--1`` because Forgejo label
|
|
# names with double hyphens are ugly in the UI and the parsing
|
|
# logic in the dispatcher handles the ``-min`` literal explicitly
|
|
# (see ``_read_start_tier_from_labels``).
|
|
ATTEMPT_TIER_LABELS_BY_TIER: dict[int, str] = {
|
|
-1: "auto/last-attempt-tier-min",
|
|
0: "auto/last-attempt-tier-0",
|
|
1: "auto/last-attempt-tier-1",
|
|
2: "auto/last-attempt-tier-2",
|
|
}
|
|
|
|
# Convenience tuple — preserved for callers that iterate (e.g.
|
|
# ``apply_attempt_label`` clearing siblings, ``clear_attempt_labels``
|
|
# wiping the lot). Order is "lowest tier first" so iteration is
|
|
# predictable.
|
|
ATTEMPT_TIER_LABELS: tuple[str, ...] = tuple(
|
|
ATTEMPT_TIER_LABELS_BY_TIER[t]
|
|
for t in sorted(ATTEMPT_TIER_LABELS_BY_TIER)
|
|
)
|
|
|
|
# Reverse lookup. Used by the dispatcher's label-read path to
|
|
# validate parsed integers and by tests for round-trip checks.
|
|
TIER_BY_LABEL: dict[str, int] = {
|
|
v: k for k, v in ATTEMPT_TIER_LABELS_BY_TIER.items()
|
|
}
|
|
|
|
|
|
def _label_for_tier(tier: int) -> str:
|
|
"""Map a tier integer to its label name. Raises ``ValueError``
|
|
on out-of-range — callers should never reach here with an
|
|
invalid tier (the escalation loop's max_tier guard catches it
|
|
earlier).
|
|
|
|
Accepts the full configured range including -1 (``tier-min``).
|
|
"""
|
|
if tier not in ATTEMPT_TIER_LABELS_BY_TIER:
|
|
valid = sorted(ATTEMPT_TIER_LABELS_BY_TIER)
|
|
raise ValueError(
|
|
f"tier {tier} out of range; valid tiers are {valid}"
|
|
)
|
|
return ATTEMPT_TIER_LABELS_BY_TIER[tier]
|
|
|
|
|
|
def apply_attempt_label(
|
|
cfg: Any, pr_number: int, tier: int
|
|
) -> dict[str, Any]:
|
|
"""Apply ``auto/last-attempt-tier-{tier}`` and clear the other
|
|
tier labels from the PR.
|
|
|
|
Side-effect-only — returns a small status dict for the cycle
|
|
archive so an operator can verify the mutation fired:
|
|
|
|
.. code-block:: python
|
|
|
|
{
|
|
"applied": "auto/last-attempt-tier-1",
|
|
"cleared": ["auto/last-attempt-tier-0"],
|
|
"skipped_provisioning_missing": False,
|
|
}
|
|
|
|
On label-not-provisioned (Forgejo returns 404 from the label
|
|
lookup), the dispatcher logs WARNING and proceeds — escalation
|
|
is unaffected, but operators won't see the per-tier label in
|
|
the Forgejo UI until the label provisioner is run. The
|
|
``skipped_provisioning_missing`` flag in the return value gives
|
|
the cycle archive a clean signal an operator can grep for.
|
|
"""
|
|
target_label = _label_for_tier(tier)
|
|
applied = _claim_runtime._add_label(pr_number, target_label, cfg)
|
|
if not applied:
|
|
_logger.warning(
|
|
"apply_attempt_label: failed to add %r to PR #%s "
|
|
"(label may not be provisioned in Forgejo — run the "
|
|
"label provisioner; escalation continues without "
|
|
"per-tier label observability)",
|
|
target_label,
|
|
pr_number,
|
|
)
|
|
|
|
cleared: list[str] = []
|
|
for label in ATTEMPT_TIER_LABELS:
|
|
if label == target_label:
|
|
continue
|
|
# Best-effort: _remove_label returns False on
|
|
# label-not-provisioned. We don't log the soft fail — it's
|
|
# the symmetric case of "add" failing and the same warning
|
|
# above already surfaces the root cause.
|
|
if _claim_runtime._remove_label(pr_number, label, cfg):
|
|
cleared.append(label)
|
|
|
|
return {
|
|
"applied": target_label if applied else None,
|
|
"cleared": cleared,
|
|
"skipped_provisioning_missing": not applied,
|
|
}
|
|
|
|
|
|
def clear_attempt_labels(cfg: Any, pr_number: int) -> dict[str, Any]:
|
|
"""Remove all three ``auto/last-attempt-tier-*`` labels from the
|
|
PR.
|
|
|
|
Called on successful cycle exit (the stale labels are no longer
|
|
informative — the PR's status moves on) and as a safety sweep
|
|
on END_CYCLE / EXHAUSTED paths.
|
|
|
|
Returns ``{"cleared": [...]}`` — the list of labels actually
|
|
removed (vs. labels that were already absent). Best-effort:
|
|
failures log WARNING + proceed.
|
|
"""
|
|
cleared: list[str] = []
|
|
for label in ATTEMPT_TIER_LABELS:
|
|
if _claim_runtime._remove_label(pr_number, label, cfg):
|
|
cleared.append(label)
|
|
return {"cleared": cleared}
|