d3d66f3726
Adds a flag-gated escalation loop to the implementer dispatcher (`IMPLEMENTER_ESCALATION_ENABLED=1`, default OFF). When the worker fails in a way the predicate determines escalation can help, the dispatcher holds the claim, resets the worktree to the prefetched head_sha, refreshes the TTL via _claim_runtime.claim_pr, applies the next-tier label, and re-runs the worker at the next tier — all within the same cycle. Bounded by per-failure-class budgets in _implementer_escalation.BUDGET_PER_FAILURE_CLASS. Tier 2 (tier-kimi) is default-ON with a kill-switch flag (IMPLEMENTER_ESCALATION_TIER2_ENABLED=0). Cross-cycle resumption: the dispatcher reads auto/last-attempt-tier-N at cycle start and seeds start_tier = min(N+1, max_tier) so crash recovery skips known-failed tiers. Worker holds release across the cycle via the new release_claim_on_exit: false directive — eliminates the inter-tier claim-absent race window. Behaviour preservation: flag=0 path is byte-equivalent to the pre-feature build (worker prompt unchanged, Phase 4 row schema unchanged, status-comment fingerprint unchanged). Issue work (new_issue work group) always takes the legacy path even with the flag on. Supersedes the cross-cycle-only Phase 5c scheme in auto-agents-tier-2-3-plan.md (now updated to point at the new plan doc and the dual-role label semantics). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
155 lines
5.8 KiB
Python
155 lines
5.8 KiB
Python
"""Attempt-tier label mutation for the implementer dispatcher.
|
|
|
|
Owns the ``auto/last-attempt-tier-{0,1,2}`` label set that the
|
|
in-cycle escalation loop uses for both operator visibility AND
|
|
cross-cycle resumption seeding.
|
|
|
|
**Dual role (updated 2026-05-12).** Earlier iterations of the plan
|
|
described these labels as "observability-only" with crash recovery
|
|
restarting at Tier 0. The shipped design uses them more
|
|
ambitiously: in addition to being mutated for the Forgejo UI, the
|
|
dispatcher reads them at cycle start via
|
|
:func:`dispatch_implementer._read_start_tier_from_labels` to seed
|
|
``start_tier = min(labeled_tier + 1, max_tier)``. A PR that
|
|
exhausted Tier 1 in a previous cycle resumes at Tier 2 instead of
|
|
re-running Tier 0, saving a worker session per crash-recovery
|
|
cycle. See
|
|
``docs/development/implementer-in-cycle-escalation-plan.md`` §
|
|
"Cross-cycle resumption".
|
|
|
|
The labels are NOT a strict source of truth — a stale label
|
|
(operator-set, crash before clear) can mis-seed the next cycle by
|
|
one tier. The cost is bounded: worst case is one extra higher-tier
|
|
attempt, vs. the savings of skipping a known-failed tier.
|
|
|
|
Two operations:
|
|
|
|
- :func:`apply_attempt_label` — set the label for the just-launched
|
|
tier, clear the labels for any other tier. Called at every tier
|
|
boundary so the Forgejo UI shows "this PR is currently being
|
|
worked on at tier N" while the worker session is in flight.
|
|
- :func:`clear_attempt_labels` — remove all three labels. Called
|
|
when the cycle ends successfully (the labels are stale after
|
|
resolution) and as a safety sweep on cycle exit paths where the
|
|
dispatcher may have left a stale tier label behind.
|
|
|
|
Both are **best-effort**: if a label has not been provisioned in
|
|
Forgejo, the underlying HTTP call returns a 404 and
|
|
:func:`_claim_runtime._add_label` / ``_remove_label`` already return
|
|
``False`` (their helpers swallow the label-not-found state). We log
|
|
WARNING and proceed; escalation continues without label
|
|
observability.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
import sys
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
_TOOLS_DIR = str(Path(__file__).resolve().parent)
|
|
if _TOOLS_DIR not in sys.path:
|
|
sys.path.insert(0, _TOOLS_DIR)
|
|
from _loader import ( # noqa: E402 type: ignore[import-not-found]
|
|
load_sibling as _load_sibling,
|
|
)
|
|
|
|
_claim_runtime = _load_sibling("_claim_runtime", "_claim_runtime.py")
|
|
|
|
_logger = logging.getLogger("implementer_label_state")
|
|
|
|
|
|
# The three labels owned by the in-cycle escalation loop. Kept here
|
|
# rather than in _claim_runtime so the constant lives next to the
|
|
# code that mutates it.
|
|
ATTEMPT_TIER_LABELS: tuple[str, ...] = (
|
|
"auto/last-attempt-tier-0",
|
|
"auto/last-attempt-tier-1",
|
|
"auto/last-attempt-tier-2",
|
|
)
|
|
|
|
|
|
def _label_for_tier(tier: int) -> str:
|
|
"""Map a tier integer to its label name. Raises ``ValueError``
|
|
on out-of-range — callers should never reach here with an
|
|
invalid tier (the escalation loop's max_tier guard catches it
|
|
earlier)."""
|
|
if tier < 0 or tier >= len(ATTEMPT_TIER_LABELS):
|
|
raise ValueError(
|
|
f"tier {tier} out of range; valid range is 0..{len(ATTEMPT_TIER_LABELS) - 1}"
|
|
)
|
|
return ATTEMPT_TIER_LABELS[tier]
|
|
|
|
|
|
def apply_attempt_label(
|
|
cfg: Any, pr_number: int, tier: int
|
|
) -> dict[str, Any]:
|
|
"""Apply ``auto/last-attempt-tier-{tier}`` and clear the other
|
|
tier labels from the PR.
|
|
|
|
Side-effect-only — returns a small status dict for the cycle
|
|
archive so an operator can verify the mutation fired:
|
|
|
|
.. code-block:: python
|
|
|
|
{
|
|
"applied": "auto/last-attempt-tier-1",
|
|
"cleared": ["auto/last-attempt-tier-0"],
|
|
"skipped_provisioning_missing": False,
|
|
}
|
|
|
|
On label-not-provisioned (Forgejo returns 404 from the label
|
|
lookup), the dispatcher logs WARNING and proceeds — escalation
|
|
is unaffected, but operators won't see the per-tier label in
|
|
the Forgejo UI until the label provisioner is run. The
|
|
``skipped_provisioning_missing`` flag in the return value gives
|
|
the cycle archive a clean signal an operator can grep for.
|
|
"""
|
|
target_label = _label_for_tier(tier)
|
|
applied = _claim_runtime._add_label(pr_number, target_label, cfg)
|
|
if not applied:
|
|
_logger.warning(
|
|
"apply_attempt_label: failed to add %r to PR #%s "
|
|
"(label may not be provisioned in Forgejo — run the "
|
|
"label provisioner; escalation continues without "
|
|
"per-tier label observability)",
|
|
target_label,
|
|
pr_number,
|
|
)
|
|
|
|
cleared: list[str] = []
|
|
for label in ATTEMPT_TIER_LABELS:
|
|
if label == target_label:
|
|
continue
|
|
# Best-effort: _remove_label returns False on
|
|
# label-not-provisioned. We don't log the soft fail — it's
|
|
# the symmetric case of "add" failing and the same warning
|
|
# above already surfaces the root cause.
|
|
if _claim_runtime._remove_label(pr_number, label, cfg):
|
|
cleared.append(label)
|
|
|
|
return {
|
|
"applied": target_label if applied else None,
|
|
"cleared": cleared,
|
|
"skipped_provisioning_missing": not applied,
|
|
}
|
|
|
|
|
|
def clear_attempt_labels(cfg: Any, pr_number: int) -> dict[str, Any]:
|
|
"""Remove all three ``auto/last-attempt-tier-*`` labels from the
|
|
PR.
|
|
|
|
Called on successful cycle exit (the stale labels are no longer
|
|
informative — the PR's status moves on) and as a safety sweep
|
|
on END_CYCLE / EXHAUSTED paths.
|
|
|
|
Returns ``{"cleared": [...]}`` — the list of labels actually
|
|
removed (vs. labels that were already absent). Best-effort:
|
|
failures log WARNING + proceed.
|
|
"""
|
|
cleared: list[str] = []
|
|
for label in ATTEMPT_TIER_LABELS:
|
|
if _claim_runtime._remove_label(pr_number, label, cfg):
|
|
cleared.append(label)
|
|
return {"cleared": cleared}
|