test(integration): workflow example 7 — CI/CD integration, automated PR review and fix (ci profile)
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 23s
CI / lint (pull_request) Successful in 3m20s
CI / typecheck (pull_request) Successful in 3m54s
CI / security (pull_request) Successful in 4m13s
CI / quality (pull_request) Successful in 4m17s
CI / integration_tests (pull_request) Successful in 9m21s
CI / unit_tests (pull_request) Successful in 9m42s
CI / docker (pull_request) Successful in 14s
CI / e2e_tests (pull_request) Successful in 12m1s
CI / coverage (pull_request) Successful in 11m41s
CI / status-check (pull_request) Successful in 1s
CI / build (push) Successful in 26s
CI / lint (push) Successful in 3m19s
CI / quality (push) Successful in 3m41s
CI / typecheck (push) Successful in 3m56s
CI / benchmark-regression (push) Has been skipped
CI / security (push) Successful in 4m2s
CI / integration_tests (push) Successful in 9m3s
CI / unit_tests (push) Successful in 9m24s
CI / docker (push) Successful in 1m9s
CI / e2e_tests (push) Successful in 13m51s
CI / coverage (push) Successful in 11m25s
CI / status-check (push) Successful in 1s
CI / benchmark-publish (push) Failing after 24m52s
CI / benchmark-regression (pull_request) Failing after 37m6s
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 23s
CI / lint (pull_request) Successful in 3m20s
CI / typecheck (pull_request) Successful in 3m54s
CI / security (pull_request) Successful in 4m13s
CI / quality (pull_request) Successful in 4m17s
CI / integration_tests (pull_request) Successful in 9m21s
CI / unit_tests (pull_request) Successful in 9m42s
CI / docker (pull_request) Successful in 14s
CI / e2e_tests (pull_request) Successful in 12m1s
CI / coverage (pull_request) Successful in 11m41s
CI / status-check (pull_request) Successful in 1s
CI / build (push) Successful in 26s
CI / lint (push) Successful in 3m19s
CI / quality (push) Successful in 3m41s
CI / typecheck (push) Successful in 3m56s
CI / benchmark-regression (push) Has been skipped
CI / security (push) Successful in 4m2s
CI / integration_tests (push) Successful in 9m3s
CI / unit_tests (push) Successful in 9m24s
CI / docker (push) Successful in 1m9s
CI / e2e_tests (push) Successful in 13m51s
CI / coverage (push) Successful in 11m25s
CI / status-check (push) Successful in 1s
CI / benchmark-publish (push) Failing after 24m52s
CI / benchmark-regression (pull_request) Failing after 37m6s
Implemented Robot Framework integration test suite validating the CI/CD workflow (Specification Example 7). Tests cover ci-profile configuration (automation-profile, format, log level), idempotent resource and project registration with duplicate-detection assertions, three validation tools (ci-lint, ci-typecheck, ci-tests) registration and resource attachment via ToolRegistryService, action creation with typed arguments and invariants per spec Step 2, plan lifecycle with phase-by-phase completion through all phases (strategize, execute, apply) until terminal applied state, and JSON output structure verification including plan_id, phase, state, action, projects, and arguments fields. Post-review fixes applied (round 1): - H1: Fix truncated invariant #2 text to match spec line 39051 - H2: Fix definition_of_done to match spec's 5-bullet-point format - M1: Register all 3 spec validations (ci-lint, ci-typecheck, ci-tests) - M3: Add branch property to resource registration per spec Step 3 - M4: Replace phantom resource_id with real registered resource - M5: Add projects and arguments field assertions to JSON output test - M7: Replace 'polling-based' with 'phase-by-phase' in docs/comments - L1: Use timezone-aware datetime (timezone.utc) - L2: Use tempfile for resource path instead of hardcoded /tmp - L3: Clarify json_output() docstring to reflect as_cli_dict() scope - L4: Tighten attachment count assertion from >= 1 to == 1 (now == 3) Post-review fixes applied (round 2): - M2: Use Setup Test Environment With Database Isolation for pabot safety - L1: Replace remaining hardcoded /tmp paths with tempfile (validation_attach, resource_idempotent duplicate registration) - L2: Fix stale 'Polling-based' comment missed in round 1 - L5: Align action description with spec line 39027 - M1/L3/L6: Document automation_profile propagation gap in use_action() with TODO for when production code wires the field onto Plan - L4: Add comment explaining local actor name deviation from spec Includes CHANGELOG update describing the integration test scope. ISSUES CLOSED: #771
This commit was merged in pull request #806.
This commit is contained in:
@@ -106,7 +106,7 @@ class CorrectionServiceSuite:
|
||||
"""Benchmark creating a correction request."""
|
||||
self.service.request_correction(
|
||||
plan_id="plan-bench",
|
||||
decision_id="DEC-001",
|
||||
target_decision_id="DEC-001",
|
||||
mode=CorrectionMode.REVERT,
|
||||
guidance="Benchmark correction",
|
||||
)
|
||||
@@ -115,7 +115,7 @@ class CorrectionServiceSuite:
|
||||
"""Benchmark impact analysis (stub)."""
|
||||
req = self.service.request_correction(
|
||||
plan_id="plan-bench",
|
||||
decision_id="DEC-001",
|
||||
target_decision_id="DEC-001",
|
||||
mode=CorrectionMode.REVERT,
|
||||
guidance="Benchmark analysis",
|
||||
)
|
||||
@@ -125,7 +125,7 @@ class CorrectionServiceSuite:
|
||||
"""Benchmark correction execution (stub)."""
|
||||
req = self.service.request_correction(
|
||||
plan_id="plan-bench",
|
||||
decision_id="DEC-001",
|
||||
target_decision_id="DEC-001",
|
||||
mode=CorrectionMode.REVERT,
|
||||
guidance="Benchmark execution",
|
||||
)
|
||||
|
||||
@@ -44,7 +44,7 @@ except ModuleNotFoundError:
|
||||
from cleveragents.infrastructure.database.unit_of_work import UnitOfWork
|
||||
|
||||
|
||||
_PLAN_ID = "01HV00000000000000DIBENCH1"
|
||||
_PLAN_ID = "01HV00000000000000DJBENCH1"
|
||||
|
||||
|
||||
def _make_uow() -> UnitOfWork:
|
||||
|
||||
@@ -119,16 +119,16 @@ class AutoRevertSuite:
|
||||
)
|
||||
plan_id = plan.identity.plan_id
|
||||
self.service.start_strategize(plan_id)
|
||||
# complete_strategize auto-progresses to EXECUTE with "ci" profile
|
||||
self.service.complete_strategize(plan_id)
|
||||
plan = self.service.get_plan(plan_id)
|
||||
if plan.phase != PlanPhase.EXECUTE:
|
||||
self.service.execute_plan(plan_id)
|
||||
self.service.start_execute(plan_id)
|
||||
# complete_execute auto-progresses to APPLY with "ci" profile
|
||||
self.service.complete_execute(plan_id)
|
||||
self.service.apply_plan(plan_id)
|
||||
self.service.start_apply(plan_id)
|
||||
plan = self.service.get_plan(plan_id)
|
||||
plan.processing_state = ProcessingState.CONSTRAINED
|
||||
self.service.constrain_apply(plan_id, "benchmark constraint")
|
||||
self.service.try_auto_revert_from_apply(plan_id, "benchmark")
|
||||
|
||||
|
||||
|
||||
@@ -75,10 +75,10 @@ class ResourceTreeSuite:
|
||||
"""Set up mock service."""
|
||||
root = _mock_resource()
|
||||
child1 = _mock_resource(
|
||||
"01HBENCH0000000000CHILD001", "local/child-1", "fs-directory"
|
||||
"01HBENCH0000000000CHJKD001", "local/child-1", "fs-directory"
|
||||
)
|
||||
child2 = _mock_resource(
|
||||
"01HBENCH0000000000CHILD002", "local/child-2", "fs-directory"
|
||||
"01HBENCH0000000000CHJKD002", "local/child-2", "fs-directory"
|
||||
)
|
||||
tree = [
|
||||
_mock_tree_node(
|
||||
|
||||
@@ -61,6 +61,11 @@ class SessionListDISuite:
|
||||
bind=self._engine,
|
||||
expire_on_commit=False,
|
||||
)
|
||||
# Pre-initialise attributes used by per-method setup hooks so that
|
||||
# benchmarks still work even when the ASV runner skips per-method
|
||||
# setup_<name>() callbacks (e.g. in fork-server mode).
|
||||
self._empty_svc = self._make_service()
|
||||
self._fresh_engine()
|
||||
|
||||
def teardown(self) -> None:
|
||||
self._engine.dispose()
|
||||
|
||||
@@ -84,7 +84,7 @@ class TimeUnifiedModelCreation:
|
||||
def time_context_payload(self) -> None:
|
||||
"""Create core ContextPayload (inherits CRP AssembledContext)."""
|
||||
for _ in range(1000):
|
||||
ContextPayload(plan_id="01JQBENCHPN00000000000000AA")
|
||||
ContextPayload(plan_id="01JQBENCHPN0000000000000AA")
|
||||
|
||||
|
||||
class TimeIsinstanceChecks:
|
||||
@@ -104,7 +104,7 @@ class TimeIsinstanceChecks:
|
||||
)
|
||||
self.budget = ContextBudget(max_tokens=4096, reserved_tokens=512)
|
||||
self.payload = ContextPayload(
|
||||
plan_id="01JQBENCHPN00000000000000AA",
|
||||
plan_id="01JQBENCHPN0000000000000AA",
|
||||
)
|
||||
|
||||
def time_isinstance_provenance(self) -> None:
|
||||
|
||||
Reference in New Issue
Block a user