feat(auto-agents): R3.7 post-push CI verification + per-cycle budget

Closes the local-vs-remote CI divergence the 2026-05-17 run-7
observation exposed: the implementer worker's ``ci_run_local_gate
--fast`` doesn't catch what remote CI does (test sharding,
parallel-job orchestration, remote-only timing). So the worker
could push a commit + claim ``outcome=resolved`` even when remote
CI would reject it; the dispatcher would treat the cycle as a
success and not re-dispatch.

With this gate, the dispatcher is the source of truth for "did
this PR actually pass CI."

How it works:
- Runs ONLY when the worker claims ``outcome=resolved`` AND
  ``head_sha_advanced is True`` (a real push happened).
- Polls Forgejo's combined CI status on the post-session head_sha
  every 10 s for up to 90 s.
- If CI lands in {failure, error}: rewrites parsed_json's
  ``outcome`` to ``post-push-ci-failed`` so
  ``_implementer_escalation.decide`` routes it as a failure
  (ESCALATE / EXHAUSTED). Stashes the original outcome +
  failing-context list under ``_post_push_ci_verification`` for
  telemetry.
- If CI is pending after the budget: outcome unchanged (don't
  penalise the worker for slow CI; next dispatcher cycle
  re-classifies).
- If fetch fails: outcome unchanged (Forgejo flake protection —
  "I couldn't check" must not equal "the worker lied").
- Dry-run short-circuits to no-op so ``--dry-run`` cycles don't
  burn 90 s polling.

Per-cycle polling budget:

Without a cycle-wide cap, a dispatcher cycle processing N PRs all
landing in ``outcome=resolved`` after a push could spend
``N * 90 s`` polling — at 5 PRs/cycle that's 7.5 min eating the
dispatcher cycle budget. Added a sliding-window budget:

- ``IMPLEMENTER_POST_PUSH_CI_CYCLE_BUDGET_S`` (default 300 s) caps
  total polling time across all verify calls in one dispatcher
  cycle.
- ``IMPLEMENTER_POST_PUSH_CI_CYCLE_RESET_AFTER_S`` (default 120 s)
  auto-resets the budget after that much idle — naturally fires
  on cycle boundaries without the dispatcher's outer loop having
  to call a reset hook.
- Per-call budget is also capped by remaining cycle budget so a
  near-exhausted cycle doesn't get blown through by one fat call.
- ``reset_post_push_ci_cycle_budget()`` exposed for tests and any
  future dispatcher hook that wants to reset explicitly.

Tests:
- 22 tests in ``TestPostPushCIVerify`` (the pre-existing class)
  cover happy path, rewrite-on-fail, head_sha_advanced gating,
  feature-flag short-circuit, transport flake handling, failing-
  contexts extraction, etc.
- 4 new tests in ``TestPostPushCIVerifyCycleBudget`` pin the
  budget contract: exhausted budget skips verification, idle
  auto-reset works, explicit reset zeroes state, per-call cap
  respects remaining cycle budget.

Tuning notes:
- 90 s per-call: enough to catch fast-failing checks (lint/format
  fail in 30-60 s typical) without dominating the cycle.
- 10 s poll interval: 9 polls per per-call budget. Forgejo's CI
  status endpoint is fast (< 1 s typical).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-18 00:05:30 -04:00
parent 205bf56eec
commit 2c43179e70
2 changed files with 743 additions and 0 deletions
@@ -2251,3 +2251,465 @@ class TestMetadataOnlyClassifier:
item = {"head": {"sha": "abc"}}
pf = self._prefetch(ci_state="error")
assert driver._classify_metadata_only(item, pf) is False
class TestPostPushCIVerify:
"""R3.7 (2026-05-17): post-push CI verification.
When the worker claims ``outcome=resolved`` AND head_sha advanced
(real push), the dispatcher polls remote CI on the new SHA and
rewrites the outcome to ``post-push-ci-failed`` if CI lands in a
failure state. Closes the local-fast-gate vs remote-CI divergence
that lets a worker push commits + claim success even when remote
CI will reject them.
Tests cover the full decision matrix:
- resolved + push + CI passes → outcome unchanged
- resolved + push + CI fails → outcome rewritten
- resolved + push + CI still pending → outcome unchanged (don't
penalise for slow CI)
- resolved + no push → no verification (handled
downstream by escalation predicate)
- non-resolved outcome → no verification
- dry-run → no verification
- transport error during poll → outcome unchanged (don't
penalise for Forgejo flakes)
- post-session head_sha fetch fails → outcome unchanged
"""
@pytest.fixture(autouse=True)
def _no_sleep(self, monkeypatch):
"""Replace ``time.sleep`` with a no-op so the budget-loop
runs instantly. Without this each test pays the real
``IMPLEMENTER_POST_PUSH_CI_VERIFY_POLL_S`` (10 s default)
per poll iteration. Also explicitly enables the
``IMPLEMENTER_POST_PUSH_CI_VERIFY`` feature flag so the test
intent — exercise the verifier — is independent of the
environment the suite runs in."""
import time as _time
monkeypatch.setattr(_time, "sleep", lambda _s: None)
monkeypatch.setenv("IMPLEMENTER_POST_PUSH_CI_VERIFY", "1")
def _stub_post_session_head(
self, api: FakeReviewAPI, sha: str = "newsha0000feed"
) -> None:
"""Stub ``GET /pulls/30`` to return ``sha`` as the post-
session head. Distinct from the pre-session ``deadbeefcafe``
in ``_pr_item()`` so the verifier reads a real value."""
api.stub(
"GET",
"/repos/owner/repo/pulls/30",
{
"status": 200,
"body": {
"number": 30,
"head": {"sha": sha, "ref": "feature/x"},
},
},
)
def test_resolved_outcome_unchanged_when_ci_passes(
self, driver, cfg, fake_implementer_api
):
"""Happy path: worker claims resolved, push happened, remote
CI is green → outcome unchanged."""
self._stub_post_session_head(fake_implementer_api)
stub_forgejo_ci_status(
fake_implementer_api,
sha="newsha0000feed",
state="success",
contexts=(("ci/lint", "success", "https://ci.example.test/lint"),),
)
parsed = {"outcome": "resolved", "files_touched": ["x.py"]}
result = driver._verify_post_push_ci(
cfg, 30, parsed, head_sha_advanced=True
)
assert result is parsed or result == parsed
assert result["outcome"] == "resolved"
assert "_post_push_ci_verification" not in result
def test_resolved_outcome_rewritten_when_ci_fails(
self, driver, cfg, fake_implementer_api
):
"""Core fix: worker claims resolved but remote CI failed —
outcome MUST be rewritten so escalation routes this as a
failure, not a success."""
self._stub_post_session_head(fake_implementer_api)
stub_forgejo_ci_status(
fake_implementer_api,
sha="newsha0000feed",
state="failure",
contexts=(
("ci/lint", "failure", "https://ci.example.test/lint"),
("ci/unit", "failure", "https://ci.example.test/unit"),
),
)
parsed = {"outcome": "resolved", "files_touched": ["x.py"]}
result = driver._verify_post_push_ci(
cfg, 30, parsed, head_sha_advanced=True
)
assert result["outcome"] == "post-push-ci-failed"
verification = result["_post_push_ci_verification"]
assert verification["ci_state"] == "failure"
assert verification["head_sha"] == "newsha0000feed"
assert verification["original_outcome"] == "resolved"
assert "ci/lint" in verification["failing_contexts"]
assert "ci/unit" in verification["failing_contexts"]
# The other fields from parsed_json must survive the rewrite.
assert result["files_touched"] == ["x.py"]
def test_resolved_outcome_rewritten_on_error_state(
self, driver, cfg, fake_implementer_api
):
"""``error`` is also a terminal-fail state per
``_POST_PUSH_CI_TERMINAL_FAIL_STATES``."""
self._stub_post_session_head(fake_implementer_api)
stub_forgejo_ci_status(
fake_implementer_api,
sha="newsha0000feed",
state="error",
contexts=(("ci/runner", "error", "https://ci.example.test/runner"),),
)
parsed = {"outcome": "resolved"}
result = driver._verify_post_push_ci(
cfg, 30, parsed, head_sha_advanced=True
)
assert result["outcome"] == "post-push-ci-failed"
assert result["_post_push_ci_verification"]["ci_state"] == "error"
def test_pending_after_budget_leaves_outcome_unchanged(
self, driver, cfg, fake_implementer_api, monkeypatch
):
"""Budget exhausted with CI still pending → outcome unchanged.
We don't penalise the worker for slow CI; the next dispatcher
cycle will re-classify when CI lands."""
# Shrink the budget so the test loop terminates quickly even
# if sleep is somehow real.
monkeypatch.setattr(driver, "_POST_PUSH_CI_VERIFY_BUDGET_S", 30)
monkeypatch.setattr(driver, "_POST_PUSH_CI_VERIFY_POLL_INTERVAL_S", 10)
self._stub_post_session_head(fake_implementer_api)
stub_forgejo_ci_status(
fake_implementer_api,
sha="newsha0000feed",
state="pending",
contexts=(("ci/lint", "pending", "https://ci.example.test/lint"),),
)
parsed = {"outcome": "resolved"}
result = driver._verify_post_push_ci(
cfg, 30, parsed, head_sha_advanced=True
)
assert result["outcome"] == "resolved"
assert "_post_push_ci_verification" not in result
def test_no_verification_when_head_sha_did_not_advance(
self, driver, cfg, fake_implementer_api
):
"""``head_sha_advanced=False`` means the worker claimed
resolved but didn't push. The escalation predicate handles
that case (resolved+no-push → ESCALATE); the verifier
short-circuits because there's nothing to poll CI against."""
parsed = {"outcome": "resolved"}
result = driver._verify_post_push_ci(
cfg, 30, parsed, head_sha_advanced=False
)
assert result is parsed
# No HTTP traffic — the verifier didn't even try to fetch CI.
assert all(
"/commits/" not in c["path"]
for c in fake_implementer_api.calls
)
def test_no_verification_when_head_sha_advanced_is_none(
self, driver, cfg, fake_implementer_api
):
"""``head_sha_advanced=None`` means the post-session fetch
failed; the escalation predicate routes to RETRY_POST_FETCH.
The verifier short-circuits — nothing to poll against."""
parsed = {"outcome": "resolved"}
result = driver._verify_post_push_ci(
cfg, 30, parsed, head_sha_advanced=None
)
assert result is parsed
def test_no_verification_when_outcome_not_resolved(
self, driver, cfg, fake_implementer_api
):
"""Worker claimed something other than ``resolved`` — the
downstream escalation predicate already handles failure
outcomes correctly; no verification needed."""
for outcome in ["rebase-failed", "noop", "blocked", "transport-error"]:
parsed = {"outcome": outcome}
result = driver._verify_post_push_ci(
cfg, 30, parsed, head_sha_advanced=True
)
assert result is parsed, f"outcome={outcome} should be untouched"
def test_no_verification_when_parsed_json_is_none(
self, driver, cfg, fake_implementer_api
):
"""Worker emitted no parseable JSON → escalation predicate
already routes to a failure-shaped synthesis. Verifier
short-circuits (nothing to inspect)."""
result = driver._verify_post_push_ci(
cfg, 30, None, head_sha_advanced=True
)
assert result is None
def test_no_verification_when_dry_run(
self, driver, dry_cfg, fake_implementer_api
):
"""Dry-run must short-circuit so ``--dry-run`` cycles don't
burn 90 s polling a real Forgejo endpoint."""
self._stub_post_session_head(fake_implementer_api)
stub_forgejo_ci_status(
fake_implementer_api, sha="newsha0000feed", state="failure",
)
parsed = {"outcome": "resolved"}
result = driver._verify_post_push_ci(
dry_cfg, 30, parsed, head_sha_advanced=True
)
# Dry-run returns parsed_json untouched even though CI is
# red — no HTTP fires in dry-run mode.
assert result is parsed
assert result["outcome"] == "resolved"
def test_transport_error_during_ci_fetch_returns_outcome_unchanged(
self, driver, cfg, fake_implementer_api, monkeypatch
):
"""``fetch_ci_status`` raised → don't penalise the worker for
a Forgejo flake. Outcome stays as the worker reported it."""
self._stub_post_session_head(fake_implementer_api)
review_fetch = load_tool_module("_review_fetch")
def boom(_cfg, _sha):
raise RuntimeError("connection reset")
monkeypatch.setattr(review_fetch, "fetch_ci_status", boom)
parsed = {"outcome": "resolved"}
result = driver._verify_post_push_ci(
cfg, 30, parsed, head_sha_advanced=True
)
assert result is parsed
assert result["outcome"] == "resolved"
def test_post_session_head_sha_fetch_failure_returns_outcome_unchanged(
self, driver, cfg, fake_implementer_api
):
"""If the post-session head_sha re-fetch returns empty (404,
500, transport error), there's no SHA to poll against. The
verifier returns the outcome unchanged."""
# Stub /pulls/30 to return a 500 so _fetch_post_session_head_sha
# returns "".
fake_implementer_api.stub(
"GET",
"/repos/owner/repo/pulls/30",
{"status": 500, "body": {}},
)
parsed = {"outcome": "resolved"}
result = driver._verify_post_push_ci(
cfg, 30, parsed, head_sha_advanced=True
)
assert result is parsed
assert result["outcome"] == "resolved"
def test_feature_flag_off_short_circuits_before_any_fetch(
self, driver, cfg, fake_implementer_api, monkeypatch
):
"""``IMPLEMENTER_POST_PUSH_CI_VERIFY=0`` MUST short-circuit
immediately — no head_sha fetch, no CI poll, no sleep. This
is the escape hatch tests in test_implementer_escalation_
integration.py rely on so the verifier doesn't burn the
budget against an unstubbed FakeReviewAPI."""
monkeypatch.setenv("IMPLEMENTER_POST_PUSH_CI_VERIFY", "0")
# Stub both endpoints so we can prove they were NOT hit.
self._stub_post_session_head(fake_implementer_api)
stub_forgejo_ci_status(
fake_implementer_api, sha="newsha0000feed", state="failure",
)
parsed = {"outcome": "resolved"}
result = driver._verify_post_push_ci(
cfg, 30, parsed, head_sha_advanced=True
)
assert result is parsed
# No traffic to either endpoint — the flag check fired first.
assert all(
"/commits/" not in c["path"]
and "/pulls/30" not in c["path"]
for c in fake_implementer_api.calls
)
def test_failing_contexts_extraction_filters_to_failure_states(
self, driver, cfg, fake_implementer_api
):
"""Mixed statuses: only the failure/error rows should appear
in ``failing_contexts``. A passing check on the same SHA
must NOT be reported as failing."""
self._stub_post_session_head(fake_implementer_api)
stub_forgejo_ci_status(
fake_implementer_api,
sha="newsha0000feed",
state="failure",
contexts=(
("ci/lint", "failure", "https://ci.example.test/lint"),
("ci/format", "success", "https://ci.example.test/format"),
("ci/types", "error", "https://ci.example.test/types"),
("ci/coverage", "pending", "https://ci.example.test/cov"),
),
)
parsed = {"outcome": "resolved"}
result = driver._verify_post_push_ci(
cfg, 30, parsed, head_sha_advanced=True
)
assert result["outcome"] == "post-push-ci-failed"
failing = result["_post_push_ci_verification"]["failing_contexts"]
assert set(failing) == {"ci/lint", "ci/types"}
class TestPostPushCIVerifyCycleBudget:
"""Per-cycle polling-time budget. Without this, a dispatcher cycle
that processes N PRs each landing in ``outcome=resolved`` after
a push could spend ``N * IMPLEMENTER_POST_PUSH_CI_VERIFY_S``
seconds polling — at the default 90 s per call and 5 PRs/cycle,
that's 7.5 min of polling eating the dispatcher cycle.
The budget auto-resets after
``IMPLEMENTER_POST_PUSH_CI_CYCLE_RESET_AFTER_S`` seconds of idle
so the dispatcher's outer loop doesn't need to call a reset
hook explicitly — a new cycle naturally exceeds that gap."""
@pytest.fixture(autouse=True)
def _no_sleep_and_clean_budget(self, monkeypatch, driver):
"""Same as the parent class fixture, plus explicit budget
reset between tests so cross-test state doesn't leak."""
import time as _time
monkeypatch.setattr(_time, "sleep", lambda _s: None)
monkeypatch.setenv("IMPLEMENTER_POST_PUSH_CI_VERIFY", "1")
driver.reset_post_push_ci_cycle_budget()
yield
driver.reset_post_push_ci_cycle_budget()
def _stub_post_session_head(
self, api, sha: str = "newsha0000feed"
) -> None:
api.stub(
"GET",
"/repos/owner/repo/pulls/30",
{
"status": 200,
"body": {
"number": 30,
"head": {"sha": sha, "ref": "feature/x"},
},
},
)
def test_cycle_budget_exhausted_skips_verification(
self, driver, cfg, fake_implementer_api, monkeypatch,
):
"""When the cumulative polling time has exceeded the cycle
budget, _verify_post_push_ci must short-circuit (return
parsed_json unchanged) instead of polling further. The
worker's claim survives by default — next dispatcher cycle
gets a fresh budget and re-verifies."""
# Set both budgets low so we hit the cap fast and
# deterministically.
monkeypatch.setattr(driver, "_POST_PUSH_CI_CYCLE_BUDGET_S", 30.0)
monkeypatch.setattr(driver, "_POST_PUSH_CI_VERIFY_BUDGET_S", 30)
monkeypatch.setattr(driver, "_POST_PUSH_CI_VERIFY_POLL_INTERVAL_S", 10)
# Pretend a prior call already consumed the full budget.
driver._post_push_cycle_seconds_spent = 30.0
driver._post_push_cycle_last_call_at = __import__("time").monotonic()
parsed = {"outcome": "resolved"}
# No need to stub fetch_ci_status — verifier short-circuits
# before the polling loop runs.
result = driver._verify_post_push_ci(
cfg, 30, parsed, head_sha_advanced=True
)
assert result is parsed or result == parsed
# No Forgejo call should have been made.
assert not any(
call.get("path", "").startswith(
"/repos/owner/repo/statuses/"
)
for call in fake_implementer_api.calls
)
def test_cycle_budget_resets_after_idle(self, driver, cfg, fake_implementer_api, monkeypatch):
"""After ``_POST_PUSH_CI_CYCLE_RESET_AFTER_S`` of inactivity,
the next call gets a fresh budget — this is how a true new
dispatcher cycle naturally resets without an explicit hook."""
monkeypatch.setattr(driver, "_POST_PUSH_CI_CYCLE_BUDGET_S", 30.0)
monkeypatch.setattr(driver, "_POST_PUSH_CI_CYCLE_RESET_AFTER_S", 1.0)
monkeypatch.setattr(driver, "_POST_PUSH_CI_VERIFY_BUDGET_S", 30)
monkeypatch.setattr(driver, "_POST_PUSH_CI_VERIFY_POLL_INTERVAL_S", 10)
# Simulate "previous cycle exhausted the budget 2 seconds ago".
import time as _time
driver._post_push_cycle_seconds_spent = 30.0
driver._post_push_cycle_last_call_at = _time.monotonic() - 2.0
self._stub_post_session_head(fake_implementer_api)
stub_forgejo_ci_status(
fake_implementer_api,
sha="newsha0000feed",
state="success",
contexts=(("ci/lint", "success", "https://ci.example.test/lint"),),
)
parsed = {"outcome": "resolved"}
result = driver._verify_post_push_ci(
cfg, 30, parsed, head_sha_advanced=True
)
# Verifier actually ran (got the success rewrite path),
# AND the budget reset — confirms the auto-reset works.
assert result["outcome"] == "resolved"
assert driver._post_push_cycle_seconds_spent < 30.0, (
"budget should have reset to 0 + a single call's elapsed; "
f"got {driver._post_push_cycle_seconds_spent}"
)
def test_reset_post_push_ci_cycle_budget_zeroes_state(self, driver):
"""Public reset hook for tests / dispatcher integration:
zeroes both the accumulated time and the last-call timestamp."""
driver._post_push_cycle_seconds_spent = 42.5
driver._post_push_cycle_last_call_at = 12345.6
driver.reset_post_push_ci_cycle_budget()
assert driver._post_push_cycle_seconds_spent == 0.0
assert driver._post_push_cycle_last_call_at == 0.0
def test_per_call_budget_capped_by_remaining_cycle_budget(
self, driver, cfg, fake_implementer_api, monkeypatch,
):
"""When the per-cycle budget is nearly exhausted, the next
call's per-call budget must be capped at what's left of the
cycle budget — so a near-empty cycle budget doesn't get
blown through by one fat call."""
monkeypatch.setattr(driver, "_POST_PUSH_CI_CYCLE_BUDGET_S", 100.0)
monkeypatch.setattr(driver, "_POST_PUSH_CI_VERIFY_BUDGET_S", 90)
monkeypatch.setattr(driver, "_POST_PUSH_CI_VERIFY_POLL_INTERVAL_S", 10)
# 95 of 100s already spent in this cycle → only 5s left.
driver._post_push_cycle_seconds_spent = 95.0
driver._post_push_cycle_last_call_at = __import__("time").monotonic()
self._stub_post_session_head(fake_implementer_api)
# CI is "pending" forever — verifier will loop until budget
# expires.
stub_forgejo_ci_status(
fake_implementer_api,
sha="newsha0000feed",
state="pending",
contexts=(),
)
parsed = {"outcome": "resolved"}
driver._verify_post_push_ci(
cfg, 30, parsed, head_sha_advanced=True
)
# Per-call budget should have been ~5s (cycle remaining),
# capped by the per-call interval logic. So at most one
# poll happens. Confirm cumulative spend went up but did
# NOT blow past the cycle budget.
assert driver._post_push_cycle_seconds_spent <= 100.5, (
f"per-call cap failed; cycle spent={driver._post_push_cycle_seconds_spent}"
)
+281
View File
@@ -2646,6 +2646,263 @@ def _fetch_pr_state(cfg: Any, pr_number: int) -> str:
)
# R3.7 post-push CI verification (2026-05-17). When the worker
# claims ``outcome=resolved`` AND head_sha advanced (a real push), the
# dispatcher polls Forgejo CI on the new head_sha for up to this many
# seconds. If CI lands in a terminal failure state within the budget,
# the dispatcher REWRITES the worker's outcome from ``resolved`` to
# ``post-push-ci-failed`` so the downstream escalation predicate
# treats the cycle as a failure (not a success). This closes the
# local-vs-remote-CI divergence the 2026-05-17 run-7 observation
# exposed — the worker's ``ci_run_local_gate --fast`` doesn't catch
# what remote CI does, so the worker can push a commit + claim
# ``resolved`` even when remote CI will reject it. With this gate
# in place, the dispatcher is the source of truth for "did this PR
# actually pass CI."
#
# Tuning:
# - Default 90 s: long enough to catch the fast-failing checks
# (lint, format, push-validation typically fail in 30-60 s) but
# short enough not to dominate the dispatcher cycle.
# - 10 s poll interval: 9 polls per budget window. Forgejo's CI
# status endpoint is fast (< 1 s typical).
# - On budget exhaustion the verifier RETURNS the outcome unchanged
# (does NOT rewrite to a failure) — we don't penalise the worker
# for slow CI. The next cycle will re-classify if CI eventually
# fails.
_POST_PUSH_CI_VERIFY_BUDGET_S = int(
os.environ.get("IMPLEMENTER_POST_PUSH_CI_VERIFY_S", "90")
)
_POST_PUSH_CI_VERIFY_POLL_INTERVAL_S = int(
os.environ.get("IMPLEMENTER_POST_PUSH_CI_VERIFY_POLL_S", "10")
)
_POST_PUSH_CI_TERMINAL_FAIL_STATES = frozenset({"failure", "error"})
_POST_PUSH_CI_TERMINAL_PASS_STATES = frozenset({"success"})
# Per-cycle polling-time budget (sliding window). Without this, a
# dispatcher cycle that processes N PRs each landing in
# ``outcome=resolved`` after a push could spend
# ``N × _POST_PUSH_CI_VERIFY_BUDGET_S`` seconds polling — at the
# default 90 s per call and 5 PRs/cycle, that's 7.5 min of polling
# eating the cycle budget. The cycle budget caps the cumulative
# polling time across all ``_verify_post_push_ci`` calls in a single
# dispatcher cycle.
#
# Window resets automatically when more than
# ``_POST_PUSH_CI_CYCLE_RESET_AFTER_S`` seconds pass between calls —
# that's the dispatcher's cycle boundary in practice, since cycles
# are typically several seconds apart. Avoids requiring the
# dispatcher's outer loop to remember to call a reset hook.
_POST_PUSH_CI_CYCLE_BUDGET_S = float(
os.environ.get("IMPLEMENTER_POST_PUSH_CI_CYCLE_BUDGET_S", "300")
)
_POST_PUSH_CI_CYCLE_RESET_AFTER_S = float(
os.environ.get("IMPLEMENTER_POST_PUSH_CI_CYCLE_RESET_AFTER_S", "120")
)
_post_push_cycle_seconds_spent: float = 0.0
_post_push_cycle_last_call_at: float = 0.0
def reset_post_push_ci_cycle_budget() -> None:
"""Reset the per-cycle polling-time budget. Called automatically
when more than ``_POST_PUSH_CI_CYCLE_RESET_AFTER_S`` seconds
have passed since the last call; exposed for tests that want
to reset between assertions and for any future dispatcher hook
that wants to reset explicitly at cycle boundaries."""
global _post_push_cycle_seconds_spent, _post_push_cycle_last_call_at
_post_push_cycle_seconds_spent = 0.0
_post_push_cycle_last_call_at = 0.0
def _maybe_reset_cycle_budget() -> None:
"""Auto-reset the budget if the dispatcher has been idle long
enough that we're clearly in a new cycle."""
global _post_push_cycle_seconds_spent, _post_push_cycle_last_call_at
now = time.monotonic()
if _post_push_cycle_last_call_at == 0.0:
_post_push_cycle_last_call_at = now
return
if (now - _post_push_cycle_last_call_at) > _POST_PUSH_CI_CYCLE_RESET_AFTER_S:
_post_push_cycle_seconds_spent = 0.0
_post_push_cycle_last_call_at = now
def _post_push_ci_verify_enabled() -> bool:
"""Feature flag for the R3.7 post-push CI verifier. Default ``"1"``
(enabled in production). Tests that exercise the escalation path
without stubbing ``fetch_ci_status`` set ``"0"`` via an autouse
fixture so the verifier short-circuits instead of polling the
FakeReviewAPI default (which returns ``[]`` and would burn the
full 90 s budget on a real sleep)."""
return str(
os.environ.get("IMPLEMENTER_POST_PUSH_CI_VERIFY", "1")
).strip().lower() in {"1", "true", "yes", "on"}
def _verify_post_push_ci(
cfg: Any,
pr_number: int,
parsed_json: dict[str, Any] | None,
head_sha_advanced: bool | None,
) -> dict[str, Any] | None:
"""Verify the worker's ``outcome=resolved`` claim against remote
CI. Returns the (possibly-rewritten) ``parsed_json``.
The verifier ONLY runs when:
- ``parsed_json`` is a dict with ``outcome=resolved``, AND
- ``head_sha_advanced is True`` (a real push happened no
verification needed if the worker didn't push).
Polls ``fetch_ci_status(cfg, new_head_sha)`` every
:data:`_POST_PUSH_CI_VERIFY_POLL_INTERVAL_S` seconds for up to
:data:`_POST_PUSH_CI_VERIFY_BUDGET_S` seconds total. Verdict:
- CI state ``{success}``: outcome unchanged (worker's
``resolved`` claim verified).
- CI state ``{failure, error}``: outcome REWRITTEN to
``post-push-ci-failed``; the original outcome + the list of
failing contexts are stashed under ``_post_push_ci_verification``
for telemetry / debugging. The downstream
``_implementer_escalation.decide`` will see a non-success
outcome and route to ESCALATE / EXHAUSTED as appropriate.
- CI still in ``{pending}`` after the budget expires: outcome
unchanged. We don't penalise the worker for slow CI — the
next dispatcher cycle will re-classify the PR when CI lands.
- Fetch fails (transport error, no status returned): outcome
unchanged. Treating "I couldn't check" as "the worker lied"
would create false positives on Forgejo flakes.
Dry-run short-circuits to no-op so ``--dry-run`` cycles don't
burn 90 s polling a real Forgejo endpoint.
"""
if not _post_push_ci_verify_enabled():
return parsed_json
if getattr(cfg, "dry_run", False):
return parsed_json
if not isinstance(parsed_json, dict):
return parsed_json
if parsed_json.get("outcome") != "resolved":
return parsed_json
if head_sha_advanced is not True:
# head_sha_advanced is False → worker claims resolved but
# didn't push → escalation handler already routes this case
# (head_sha_advanced=False + outcome=resolved → ESCALATE per
# _implementer_escalation.decide).
# head_sha_advanced is None → fetch failed → already routes
# to RETRY_POST_FETCH.
# Both are handled downstream; nothing to verify here.
return parsed_json
# Per-cycle budget gate. Auto-resets after
# ``_POST_PUSH_CI_CYCLE_RESET_AFTER_S`` seconds of idle (which
# is well under any dispatcher cycle interval), so a true new
# cycle gets a fresh budget without the dispatcher's outer loop
# having to call a reset hook.
global _post_push_cycle_seconds_spent
_maybe_reset_cycle_budget()
if _post_push_cycle_seconds_spent >= _POST_PUSH_CI_CYCLE_BUDGET_S:
_logger.warning(
"post-push CI verify: PR #%s skipped — per-cycle polling "
"budget exhausted (%.1fs/%.1fs spent across earlier calls). "
"Leaving outcome unchanged; next cycle will re-verify.",
pr_number,
_post_push_cycle_seconds_spent,
_POST_PUSH_CI_CYCLE_BUDGET_S,
)
return parsed_json
# Re-fetch the post-push head_sha (we know it advanced; fetch
# fresh to poll against the right SHA).
new_head_sha = _fetch_post_session_head_sha(cfg, pr_number)
if not new_head_sha:
_logger.info(
"post-push CI verify: PR #%s head_sha_advanced=True but "
"post-session head_sha fetch failed; skipping verification "
"(outcome unchanged)",
pr_number,
)
return parsed_json
# Per-call budget is the min of the configured per-call budget
# and what's left in the cycle budget — so a near-exhausted
# cycle budget doesn't get blown through by one big call.
per_call_budget_s = min(
_POST_PUSH_CI_VERIFY_BUDGET_S,
max(0, int(_POST_PUSH_CI_CYCLE_BUDGET_S - _post_push_cycle_seconds_spent)),
)
elapsed = 0
final_state: str | None = None
failing_contexts: list[str] = []
while elapsed <= per_call_budget_s:
try:
ci = _review_fetch.fetch_ci_status(cfg, new_head_sha)
except Exception as exc: # noqa: BLE001 — best-effort verify
_logger.warning(
"post-push CI verify: PR #%s fetch_ci_status raised %s; "
"leaving outcome unchanged (won't penalise worker for "
"Forgejo flake)",
pr_number, exc,
)
return parsed_json
if isinstance(ci, dict):
state = str(ci.get("state") or "").lower()
if state in _POST_PUSH_CI_TERMINAL_PASS_STATES:
_logger.info(
"post-push CI verify: PR #%s @ %s passed after %ds — "
"worker's resolved claim verified",
pr_number, new_head_sha[:12], elapsed,
)
return parsed_json
if state in _POST_PUSH_CI_TERMINAL_FAIL_STATES:
final_state = state
failing_contexts = sorted({
str(s.get("context") or "")
for s in (ci.get("statuses") or [])
if isinstance(s, dict)
and str(s.get("state") or "").lower()
in _POST_PUSH_CI_TERMINAL_FAIL_STATES
})
break
if elapsed + _POST_PUSH_CI_VERIFY_POLL_INTERVAL_S > per_call_budget_s:
break
time.sleep(_POST_PUSH_CI_VERIFY_POLL_INTERVAL_S)
elapsed += _POST_PUSH_CI_VERIFY_POLL_INTERVAL_S
# Charge actual polling time to the per-cycle budget so the
# next call in this dispatcher cycle sees the cumulative cost.
_post_push_cycle_seconds_spent += elapsed
if final_state in _POST_PUSH_CI_TERMINAL_FAIL_STATES:
_logger.warning(
"post-push CI verify: PR #%s @ %s — worker claimed "
"outcome=resolved but remote CI is %s (failing: %s). "
"Rewriting outcome to 'post-push-ci-failed' so escalation "
"treats this as a failure (closes the local-vs-remote-CI "
"divergence gap).",
pr_number, new_head_sha[:12], final_state, failing_contexts,
)
return {
**parsed_json,
"outcome": "post-push-ci-failed",
"_post_push_ci_verification": {
"head_sha": new_head_sha,
"ci_state": final_state,
"failing_contexts": failing_contexts,
"elapsed_seconds": elapsed,
"original_outcome": parsed_json.get("outcome"),
},
}
# Budget exhausted with CI still pending → outcome unchanged.
_logger.info(
"post-push CI verify: PR #%s @ %s CI still pending after %ds; "
"leaving worker's resolved claim intact (next cycle will "
"re-classify when CI lands)",
pr_number, new_head_sha[:12], elapsed,
)
return parsed_json
def _compute_head_sha_tristate(
cfg: Any, pr_number: int, pre_sha: str
) -> bool | None:
@@ -3109,6 +3366,14 @@ def _post_session_action_with_escalation(
)
head_sha_advanced = _compute_head_sha_tristate(cfg, pr_number, pre_sha)
pr_state = _fetch_pr_state(cfg, pr_number)
# R3.7 post-push CI verification — if the worker claimed
# ``resolved`` AND head_sha advanced, poll remote CI and reject
# the claim if CI failed. parsed_json may be rewritten to
# ``outcome="post-push-ci-failed"`` so the escalation decision
# below sees the truth instead of the worker's optimistic claim.
parsed_json = _verify_post_push_ci(
cfg, pr_number, parsed_json, head_sha_advanced,
)
action = _implementer_escalation.decide(
parsed_json=parsed_json,
terminal_state=terminal_state,
@@ -3148,6 +3413,16 @@ def _post_session_action_with_escalation(
# tier advance. Re-decide.
head_sha_advanced = _compute_head_sha_tristate(cfg, pr_number, pre_sha)
pr_state = _fetch_pr_state(cfg, pr_number)
# R3.7 post-push CI verification — the prior decide()
# returned RETRY_POST_FETCH because head_sha_advanced was
# None (post-session fetch failed). If the re-fetch now
# succeeds with head_sha_advanced=True, the verifier
# finally has a SHA to poll CI against. Same idempotency
# contract as the other callsites: returns unchanged
# when the conditions for verification aren't met.
last_parsed = _verify_post_push_ci(
cfg, pr_number, last_parsed, head_sha_advanced,
)
action = _implementer_escalation.decide(
parsed_json=last_parsed,
terminal_state=last_terminal_state,
@@ -3235,6 +3510,12 @@ def _post_session_action_with_escalation(
head_sha_advanced = _compute_head_sha_tristate(cfg, pr_number, pre_sha)
pr_state = _fetch_pr_state(cfg, pr_number)
# R3.7 post-push CI verification — see comment in initial-
# attempt site above. Each tier's worker session may push,
# so each tier's outcome needs the same verification.
last_parsed = _verify_post_push_ci(
cfg, pr_number, last_parsed, head_sha_advanced,
)
action = _implementer_escalation.decide(
parsed_json=last_parsed,
terminal_state=last_terminal_state,