Files
cleveragents-core/tools/controller/master/forgejo_http.py
T
drew 3e23853ffa fix(controller): batch R — wire 5 V1-contract fields the controller silently dropped
Trial run-3 (2026-05-19) surfaced the first instance of a broader bug
class: V1 contract fields existed and agents emitted them, but no
controller code wired them into state transitions. An adversarial
"walk the happy path" code review found 4 more, all listed below.

The class shape: a V1 field is "Required iff X" by contract docstring,
the worker emits it correctly, but the master reads the wrong field
(or doesn't read it at all), so a critical state transition silently
no-ops or drops to the wrong default.

FIX #0 — outcome-mapper early-return (committed earlier in this
session) — moved role dispatch before the ``outcome is None`` guard
so estimator+reviewer+summarizer (V1 contracts without an ``outcome``
field) are correctly handled. Without this fix, all estimator
attempts in trial run-3 completed successfully then were silently
discarded, stranding all 6 workflows in ANALYZING.

FIX #1 — current_tier never written from estimator's recommended_tier
File: tools/controller/master/tick.py
The ANALYZING→IMPLEMENTING UPDATE wrote only current_state /
last_transition_at / entered_state_at. recommended_tier from the
estimator payload was never extracted, so every PR ran at the
workflow's creation-time tier (typically 0) regardless of what the
estimator recommended — the entire tier-escalation ladder was
informational-only. Fix: per-event ``extra_set`` clauses; on
``estimator_done`` / ``estimator_metadata_only`` events, set
``current_tier = :rec_tier`` from the payload (with 0..2 validation).
Tests: TestEstimatorRecommendedTierWritten (3 cases).

FIX #2 — approved_at_sha never passed to merge callback
File: tools/controller/master/merging.py, forgejo_http.py
ReviewerOutputV1.approved_at_sha is the exact SHA the reviewer
signed off on. Pre-fix the MergeCallback signature was
``(owner, repo, pr_number)`` — Forgejo merged whatever HEAD currently
was. Race condition: a concurrent push (operator or another driver)
between approval and merge would silently merge unapproved code.
Fix: extended signature to ``(owner, repo, pr_number, approved_at_sha)``;
SQL SELECT now pulls the latest reviewer attempt's output_payload as
a subquery; merge_pr forwards it to Forgejo as ``head_commit_id``
(Forgejo refuses with 409 if HEAD has advanced). Defensive: still
merges when approved_at_sha is None but logs a WARNING. Tests:
TestApprovedAtShaPassedToMerge (2 cases).

FIX #3 — tier_last_succeeded column had ZERO writers
File: tools/controller/master/tick.py
The schema column existed; the merging.py 409-conflict path read it
to recover the last-known-good tier; but NOTHING ever wrote to it.
Every workflow's tier_last_succeeded was permanently NULL → the
409-recovery path transitioned to IMPLEMENTING(tier=NULL) → scheduler
silently coerced to tier 0. Fix: on ``implementer_pushed`` event,
``UPDATE workflows SET tier_last_succeeded = current_tier``. Tests:
TestTierLastSucceededWritten.

FIX #4 — outcome column NULL for estimator/reviewer/summarizer
File: tools/controller/worker/runner.py
``workflow_attempts.outcome`` is the operator-facing audit column.
Pre-fix the runner extracted ``output_payload.get("outcome")``
blindly — works for implementer/conflict_resolver but those three
roles have no ``outcome`` field. Result: ``SELECT … WHERE outcome IS
NOT NULL`` audit queries silently missed every estimator/reviewer/
summarizer attempt. Fix: new ``_derive_outcome_for_audit(role, payload)``
helper synthesizes meaningful per-role values:
  - implementer/conflict_resolver: payload['outcome'] (unchanged)
  - reviewer: payload['verdict']
  - estimator: 'metadata-only' OR f'tier-{recommended_tier}'
  - summarizer: 'summarized'
Tests: TestOutcomeAuditColumn (parametrized 8 cases).

FIX #5 — conflict_resolver new_head_sha never preferred
File: tools/controller/worker/runner.py
ConflictResolverOutputV1.new_head_sha is "Required iff outcome='resolved'"
(the canonical post-rebase branch tip). Pre-fix runner.py used
``commit_shas[-1]`` for head_sha_after — works for normal git rebase
--continue but wrong for resolvers that did force-pushed merge commits
where the last commit SHA ≠ the branch tip. CI status poll would then
poll the wrong SHA. Fix: when role=='conflict_resolver', prefer
``new_head_sha`` over commits[-1]. Tests:
TestConflictResolverNewHeadShaUsed (2 cases).

ALSO updated existing tests that papered over the original bug:
- test_master_outcomes.py: estimator tests used to inject a fake
  ``"outcome": "(implicit)"`` field; now use real V1 shape (no outcome).
  Reviewer tests now use ``verdict`` (the real V1 field) not ``outcome``.
- test_master_tick.py reviewer tests: same `verdict` switch.
- test_master_merging.py: updated all 13 ``lambda o, r, n: ...``
  merge-callback stubs to the new 4-arg signature.

CONFIRMED-CLEAN (no fix needed) by the same code review:
- outcomes.py post-fix-#0
- prefetch.py field reads
- prompts.py field accesses
- ci_status_poll.py role+outcome filter
The above were verified to handle all 5 V1 contract shapes correctly.

Total: 802 → 819 controller tests, 0 regressions.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 20:40:31 -04:00

442 lines
17 KiB
Python

"""HTTP adapters wiring the controller's callback protocols to
``tools/_claim_runtime``.
The discovery / scheduler-prefetch / forgejo_writes / merging modules
all take callback functions for Forgejo I/O. Tests inject synthetic
callbacks; production wires them via this module's
``build_callbacks(cfg)`` factory.
The factory takes a ``RuntimeContext`` (from ``_claim_runtime``) and
returns a frozen ``ForgejoCallbacks`` dataclass with every callback
the controller needs. The callbacks are thin closures over
``_claim_runtime.get / post / patch / delete``.
Path conventions match Forgejo's API:
- list PRs: GET /repos/{owner}/{repo}/pulls?state=open
- list issues: GET /repos/{owner}/{repo}/issues?state=open&type=issues
- list comments: GET /repos/{owner}/{repo}/issues/{n}/comments
- post comment: POST /repos/{owner}/{repo}/issues/{n}/comments
- get labels: GET /repos/{owner}/{repo}/issues/{n}/labels
- add label: POST /repos/{owner}/{repo}/issues/{n}/labels
- remove label: DELETE /repos/{owner}/{repo}/issues/{n}/labels/{label_id}
- merge PR: POST /repos/{owner}/{repo}/pulls/{n}/merge
Forgejo treats issues + PRs interchangeably at the timeline level
(labels + comments use issue endpoints for both).
"""
from __future__ import annotations
import logging
from dataclasses import dataclass
from typing import Any, Protocol
from . import forgejo_writes as fw
from . import merging as mg
from . import discovery as ds
from . import prefetch as pf
from . import reconciliation as rec
logger = logging.getLogger(__name__)
class _ClaimRuntime(Protocol):
"""Just the parts of ``_claim_runtime`` we use. Lets tests inject
a fake without importing the real module."""
def get(self, path: str, cfg: Any) -> dict[str, Any]: ...
def post(self, path: str, cfg: Any, body: Any) -> dict[str, Any]: ...
def delete(self, path: str, cfg: Any) -> dict[str, Any]: ...
# get_ci_status(owner, repo, head_sha) → Forgejo combined-status dict
# (``{"state": "...", "statuses": [...]}``) or None on fetch failure.
from collections.abc import Callable as _Callable
GetCIStatusCallback = _Callable[[str, str, str], dict | None]
@dataclass
class ForgejoCallbacks:
"""Bundle of every callback the controller needs. Built by
``build_callbacks``. Each callback is a thin closure over
``_claim_runtime``."""
list_prs: ds.ListPRsCallback
list_issues: ds.ListIssuesCallback
list_comments: fw.ListCommentsCallback
post_comment: fw.PostCommentCallback
get_labels: fw.GetLabelsCallback
add_label: fw.AddLabelCallback
remove_label: fw.RemoveLabelCallback
merge_pr: mg.MergeCallback
# Reconciliation callbacks (Phase 1g):
get_pr_state: rec.GetPRStateCallback
get_issue_state: rec.GetIssueStateCallback
# CI status callback (Phase 1k++++ trial — wired to ci_status_poll):
get_ci_status: "GetCIStatusCallback"
# Prefetch callbacks (Phase 1h):
get_pr_details: pf.GetPRDetailsCallback
get_pr_diff: pf.GetPRDiffCallback
list_pr_reviews: pf.ListPRReviewsCallback
list_pr_comments: pf.ListPRCommentsCallback
def build_callbacks(
cfg: Any, *, runtime: _ClaimRuntime | None = None,
) -> ForgejoCallbacks:
"""Wire callbacks for a given RuntimeContext.
Tests pass a fake ``runtime`` with ``.get / .post / .delete``
methods that return the standard ``{"status": int, "body": ...}``
shape. Production omits ``runtime`` and the wiring uses the real
module.
"""
if runtime is None:
import sys
from pathlib import Path
tools_dir = Path(__file__).resolve().parents[2]
if str(tools_dir) not in sys.path:
sys.path.insert(0, str(tools_dir))
from tools import _claim_runtime as runtime # type: ignore[no-redef]
return ForgejoCallbacks(
list_prs=_make_list_prs(cfg, runtime),
list_issues=_make_list_issues(cfg, runtime),
list_comments=_make_list_comments(cfg, runtime),
post_comment=_make_post_comment(cfg, runtime),
get_labels=_make_get_labels(cfg, runtime),
add_label=_make_add_label(cfg, runtime),
remove_label=_make_remove_label(cfg, runtime),
merge_pr=_make_merge_pr(cfg, runtime),
get_pr_state=_make_get_pr_state(cfg, runtime),
get_issue_state=_make_get_issue_state(cfg, runtime),
get_pr_details=_make_get_pr_details(cfg, runtime),
get_pr_diff=_make_get_pr_diff(cfg, runtime),
list_pr_reviews=_make_list_pr_reviews(cfg, runtime),
list_pr_comments=_make_list_pr_comments(cfg, runtime),
get_ci_status=_make_get_ci_status(cfg, runtime),
)
# ─── discovery ───────────────────────────────────────────────────────
def _make_list_prs(cfg, runtime):
def list_prs(owner: str, repo: str) -> list[dict]:
path = f"/repos/{owner}/{repo}/pulls?state=open"
resp = runtime.get(path, cfg)
return _list_or_empty(resp, "list_prs")
return list_prs
def _make_list_issues(cfg, runtime):
def list_issues(owner: str, repo: str) -> list[dict]:
# Forgejo's /issues endpoint returns both issues + PRs by
# default; type=issues filters out PRs (which list_prs handles).
path = f"/repos/{owner}/{repo}/issues?state=open&type=issues"
resp = runtime.get(path, cfg)
return _list_or_empty(resp, "list_issues")
return list_issues
# ─── status comments ─────────────────────────────────────────────────
def _make_list_comments(cfg, runtime):
def list_comments(owner: str, repo: str, pr_number: int) -> list[dict]:
path = f"/repos/{owner}/{repo}/issues/{int(pr_number)}/comments"
resp = runtime.get(path, cfg)
return _list_or_empty(resp, "list_comments")
return list_comments
def _make_post_comment(cfg, runtime):
def post_comment(
owner: str, repo: str, pr_number: int, body: str,
) -> dict:
path = f"/repos/{owner}/{repo}/issues/{int(pr_number)}/comments"
resp = runtime.post(path, cfg, {"body": body})
if int(resp.get("status") or 0) not in (200, 201):
raise RuntimeError(
f"post_comment HTTP {resp.get('status')}: "
f"{_summarise_body(resp.get('body'))}"
)
body_obj = resp.get("body") or {}
if not isinstance(body_obj, dict):
return {"id": None}
return body_obj
return post_comment
# ─── labels ──────────────────────────────────────────────────────────
def _make_get_labels(cfg, runtime):
def get_labels(owner: str, repo: str, pr_number: int) -> list[dict]:
path = f"/repos/{owner}/{repo}/issues/{int(pr_number)}/labels"
resp = runtime.get(path, cfg)
return _list_or_empty(resp, "get_labels")
return get_labels
def _make_add_label(cfg, runtime):
def add_label(
owner: str, repo: str, pr_number: int, label_name: str,
) -> bool:
path = f"/repos/{owner}/{repo}/issues/{int(pr_number)}/labels"
# Forgejo accepts {"labels": ["label_name"]} on POST.
resp = runtime.post(path, cfg, {"labels": [label_name]})
return int(resp.get("status") or 0) in (200, 201)
return add_label
def _make_remove_label(cfg, runtime):
def remove_label(
owner: str, repo: str, pr_number: int, label_name: str,
) -> bool:
# Forgejo's DELETE-by-name endpoint:
# DELETE /repos/{owner}/{repo}/issues/{n}/labels/{label_name}
# (some Forgejo deployments use label_id; if so, the caller
# would need to look it up first via get_labels)
from urllib.parse import quote
path = (
f"/repos/{owner}/{repo}/issues/{int(pr_number)}/"
f"labels/{quote(label_name, safe='')}"
)
resp = runtime.delete(path, cfg)
return int(resp.get("status") or 0) in (200, 204, 404)
return remove_label
# ─── merge ───────────────────────────────────────────────────────────
def _make_merge_pr(cfg, runtime):
def merge_pr(
owner: str, repo: str, pr_number: int,
approved_at_sha: str | None = None,
) -> mg.MergeResponse:
path = f"/repos/{owner}/{repo}/pulls/{int(pr_number)}/merge"
# Forgejo's merge endpoint accepts ``head_commit_id``: when set,
# Forgejo refuses the merge (returns 409) if the PR's HEAD has
# advanced past that SHA since approval. This is the race-
# protection invariant the reviewer's approved_at_sha exists to
# enforce. Defensive: only include when caller provides a SHA;
# legacy callers / missing-reviewer-payload paths still merge
# whatever HEAD currently is.
body: dict = {"Do": "merge"}
if approved_at_sha:
body["head_commit_id"] = approved_at_sha
try:
resp = runtime.post(path, cfg, body)
except Exception as exc: # noqa: BLE001 — bubble as transient
# Synthetic 503 — the merging handler treats this as a
# retryable failure.
return mg.MergeResponse(
status_code=503,
error_message=f"merge HTTP call raised: {exc}",
)
status = int(resp.get("status") or 0)
body = resp.get("body")
err = _summarise_body(body) if status >= 400 else None
# For 404, try to fetch the PR state to distinguish
# externally-merged vs externally-closed. If the PR fetch
# itself fails (5xx / 404 / other), leave pr_state=None
# so the merging handler defaults to ABANDONED conservatively.
pr_state: str | None = None
if status == 404:
try:
pr_resp = runtime.get(
f"/repos/{owner}/{repo}/pulls/{int(pr_number)}", cfg,
)
if int(pr_resp.get("status") or 0) == 200:
pr_body = pr_resp.get("body") or {}
if isinstance(pr_body, dict):
if pr_body.get("merged") is True:
pr_state = "merged"
elif pr_body.get("state") == "closed":
pr_state = "closed"
else:
pr_state = "open"
except Exception:
# Couldn't determine; leave None → handler defaults
# to ABANDONED per its conservative policy.
pass
return mg.MergeResponse(
status_code=status, pr_state=pr_state, error_message=err,
)
return merge_pr
# ─── helpers ─────────────────────────────────────────────────────────
def _list_or_empty(resp: dict, action: str) -> list[dict]:
"""Normalize a list-GET response into a list of dicts."""
status = int(resp.get("status") or 0)
if status != 200:
logger.warning("%s: HTTP %s; treating as empty", action, status)
return []
body = resp.get("body")
if not isinstance(body, list):
return []
return [item for item in body if isinstance(item, dict)]
def _summarise_body(body: Any) -> str:
"""Truncate a Forgejo error body to fit in a one-line error."""
if body is None:
return ""
s = str(body)
if len(s) <= 200:
return s
return s[:200] + f"… ({len(s)} chars)"
# ─── reconciliation (Phase 1g) ───────────────────────────────────────
def _make_get_pr_state(cfg, runtime):
def get_pr_state(owner: str, repo: str, pr_number: int) -> dict | None:
path = f"/repos/{owner}/{repo}/pulls/{int(pr_number)}"
resp = runtime.get(path, cfg)
status = int(resp.get("status") or 0)
if status == 404:
return None
if status != 200:
logger.warning(
"get_pr_state: HTTP %s for %s/%s #%d",
status, owner, repo, pr_number,
)
# Treat non-200/404 as "couldn't determine" — raise so the
# reconciliation tick records fetch-failed (rather than
# silently transitioning to STUCK).
raise RuntimeError(f"get_pr_state HTTP {status}")
body = resp.get("body")
if not isinstance(body, dict):
raise RuntimeError("get_pr_state: non-dict body")
return body
return get_pr_state
def _make_get_issue_state(cfg, runtime):
def get_issue_state(owner: str, repo: str, issue_number: int) -> dict | None:
path = f"/repos/{owner}/{repo}/issues/{int(issue_number)}"
resp = runtime.get(path, cfg)
status = int(resp.get("status") or 0)
if status == 404:
return None
if status != 200:
raise RuntimeError(f"get_issue_state HTTP {status}")
body = resp.get("body")
if not isinstance(body, dict):
raise RuntimeError("get_issue_state: non-dict body")
return body
return get_issue_state
# ─── prefetch (Phase 1h) ─────────────────────────────────────────────
def _make_get_pr_details(cfg, runtime):
def get_pr_details(owner: str, repo: str, pr_number: int) -> dict | None:
path = f"/repos/{owner}/{repo}/pulls/{int(pr_number)}"
try:
resp = runtime.get(path, cfg)
except Exception as exc: # noqa: BLE001 — transport failure
logger.warning(
"get_pr_details: transport error for %s/%s #%d: %s",
owner, repo, pr_number, exc,
)
return None
status = int(resp.get("status") or 0)
if status == 404:
return None
if status != 200:
logger.warning(
"get_pr_details: HTTP %s for %s/%s #%d",
status, owner, repo, pr_number,
)
return None
body = resp.get("body")
return body if isinstance(body, dict) else None
return get_pr_details
def _make_get_pr_diff(cfg, runtime):
def get_pr_diff(owner: str, repo: str, pr_number: int) -> str | None:
# Forgejo: GET /repos/{owner}/{repo}/pulls/{n}.diff returns
# the raw unified diff as text/plain.
path = f"/repos/{owner}/{repo}/pulls/{int(pr_number)}.diff"
try:
resp = runtime.get(path, cfg)
except Exception as exc: # noqa: BLE001
logger.warning(
"get_pr_diff: transport error for %s/%s #%d: %s",
owner, repo, pr_number, exc,
)
return None
status = int(resp.get("status") or 0)
if status != 200:
return None
body = resp.get("body")
if isinstance(body, str):
return body
if isinstance(body, (bytes, bytearray)):
try:
return bytes(body).decode("utf-8", errors="replace")
except Exception:
return None
return None
return get_pr_diff
def _make_list_pr_reviews(cfg, runtime):
def list_pr_reviews(owner: str, repo: str, pr_number: int) -> list[dict]:
path = f"/repos/{owner}/{repo}/pulls/{int(pr_number)}/reviews"
resp = runtime.get(path, cfg)
return _list_or_empty(resp, "list_pr_reviews")
return list_pr_reviews
def _make_list_pr_comments(cfg, runtime):
def list_pr_comments(owner: str, repo: str, pr_number: int) -> list[dict]:
# Forgejo: PR comments use the issues endpoint (PRs ARE issues
# at the comment/label level).
path = f"/repos/{owner}/{repo}/issues/{int(pr_number)}/comments"
resp = runtime.get(path, cfg)
return _list_or_empty(resp, "list_pr_comments")
return list_pr_comments
# ─── CI status (Phase 1k++++ trial — wired to ci_status_poll) ───────
def _make_get_ci_status(cfg, runtime):
def get_ci_status(owner: str, repo: str, head_sha: str) -> dict | None:
if not head_sha:
return None
path = f"/repos/{owner}/{repo}/commits/{head_sha}/status"
try:
resp = runtime.get(path, cfg)
except Exception as exc: # noqa: BLE001 — transient
logger.warning(
"get_ci_status: transport error for %s/%s @%s: %s",
owner, repo, head_sha[:12], exc,
)
return None
status = int(resp.get("status") or 0)
if status != 200:
logger.warning(
"get_ci_status: HTTP %s for %s/%s @%s",
status, owner, repo, head_sha[:12],
)
return None
body = resp.get("body")
return body if isinstance(body, dict) else None
return get_ci_status
__all__ = ["ForgejoCallbacks", "GetCIStatusCallback", "build_callbacks"]