7f078f75a5
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 28s
CI / security (pull_request) Successful in 1m8s
CI / build (pull_request) Successful in 20s
CI / helm (pull_request) Successful in 34s
CI / quality (pull_request) Successful in 3m53s
CI / integration_tests (pull_request) Successful in 4m2s
CI / typecheck (pull_request) Successful in 4m9s
CI / unit_tests (pull_request) Successful in 4m48s
CI / docker (pull_request) Successful in 1m19s
CI / e2e_tests (pull_request) Successful in 9m47s
CI / coverage (pull_request) Successful in 11m41s
CI / status-check (pull_request) Successful in 1s
CI / quality (push) Successful in 52s
CI / lint (push) Successful in 3m18s
CI / build (push) Successful in 20s
CI / typecheck (push) Successful in 3m56s
CI / helm (push) Successful in 22s
CI / security (push) Successful in 4m15s
CI / unit_tests (push) Successful in 4m39s
CI / docker (push) Successful in 18s
CI / integration_tests (push) Successful in 6m56s
CI / e2e_tests (push) Successful in 12m34s
CI / coverage (push) Successful in 11m39s
CI / status-check (push) Successful in 1s
CI / benchmark-regression (push) Has been skipped
CI / benchmark-publish (push) Successful in 30m10s
CI / benchmark-regression (pull_request) Successful in 57m33s
Changed SandboxManager.commit_all() from partial-commit semantics to all-or-nothing atomic operation per specification requirement. On partial failure, already-committed sandboxes are rolled back. Error reporting indicates which sandbox failed and what was rolled back. Added Behave scenarios verifying atomicity guarantee. Hardened atomicity guarantees after code review: - commit_all and _rollback_committed catch Exception (not just SandboxError) so unexpected errors cannot bypass rollback. Non-SandboxError exceptions are wrapped in a new AtomicCommitError (chaining the original as __cause__) that carries rolled_back_ids and failed_rollback_ids attributes so callers can programmatically determine rollback outcomes. - _rollback_committed returns both rolled_back_ids and failed_rollback_ids; the error result metadata now carries both "rolled_back" and "rollback_failed" keys. - _rollback_committed iterates in reverse (LIFO) order following the standard transaction-log undo pattern. Clarified in docstring that this is distinct from the specification DAG-based "top-down" rollback ordering (line 24632). - TransactionSandbox.rollback() from COMMITTED now raises SandboxRollbackError (database commits are irreversible) instead of silently transitioning to ROLLED_BACK. - TransactionSandbox is now classified as non-rollbackable in commit_all alongside NoSandbox and committed last in the batch, since database COMMIT is irreversible. Docstring corrected to match the raising behavior. - CopyOnWriteSandbox and OverlaySandbox rollback-from-COMMITTED uses rename-based safe_restore() to prevent data loss when the copytree step fails after the original was removed. - CopyOnWriteSandbox and OverlaySandbox rollback() now catches Exception (not just OSError), matching the broader catch used in _rollback_committed, so non-OSError exceptions from safe_restore set the status to ERRORED correctly. - Extracted shared _fs_utils module (backup_directory, safe_restore, compute_diff) with symlink, permission, and timestamp preservation (including directory timestamps), replacing duplicated per-class _backup_directory and _compute_diff methods. - backup_directory defers directory permissions and timestamps to a bottom-up post-walk pass, fixing incorrect mtime preservation (POSIX file creation inside a directory overwrites its mtime) and preventing restrictive source permissions from blocking backup writes. - backup_directory skips non-regular files (FIFOs, sockets, device files) with a warning to prevent hangs on special files. - CopyOnWriteSandbox and OverlaySandbox commit() now attempts to restore the original from the pre-commit backup when the file-copy phase fails midway, preventing partial corruption. If the restore itself fails the backup is preserved for manual recovery (cleanup() still removes it). - CopyOnWriteSandbox and OverlaySandbox commit() error handler now catches Exception (not just OSError) so that unexpected errors during the file-copy phase also trigger pre-commit backup restoration, preventing partial corruption of the original directory. - Pre-commit backup exception handler catches Exception (not just OSError) preventing temp directory leaks on non-OSError failures from backup_directory. - Fixed _pre_commit_backup assignment timing: the backup reference is now assigned only AFTER backup_directory() succeeds, preventing safe_restore() from corrupting an intact original with a partial backup when backup_directory() fails (e.g. disk full). - Rollback from COMMITTED with no pre-commit backup (no changes were applied) is now a no-op instead of raising SandboxRollbackError, preventing false rollback-failure reports in commit_all error metadata. - commit_all logs a warning when NoSandbox or TransactionSandbox instances are present in the batch since their changes cannot be rolled back, which breaks the atomicity guarantee. - Pre-commit backup is skipped when compute_diff returns no changes, avoiding a full directory copy for no-op commits. - GitWorktreeSandbox clears _pre_merge_commit on commit failure so the stale value cannot be used by future code. - commit_all docstring documents Raises clause for AtomicCommitError exception wrapping behavior. - GitWorktreeSandbox.rollback() docstring warns about multi-worktree safety when rolling back from COMMITTED. - Updated SandboxStatus transition diagram in protocol.py to clearly show the COMMITTED -> ROLLED_BACK path. - Added spec-contradiction note (line 45938 vs 19193) in commit_all docstring. - OverlaySandbox rollback from COMMITTED now properly remounts OverlayFS for real overlay (unmount, clean upper/work dirs, remount) and uses dirs_exist_ok=True for userspace fallback to prevent FileExistsError if rmtree silently fails. The merged directory is reset from the restored original, preventing stale pre-rollback data from being exposed on re-activation via get_path() (which allows ROLLED_BACK status). - OverlaySandbox rollback from COMMITTED now raises SandboxRollbackError if the OverlayFS unmount fails, preventing a double-mount attempt that would leave the sandbox in an inconsistent state. - OverlaySandbox rollback from ACTIVE now uses dirs_exist_ok=True for userspace fallback to prevent FileExistsError when rmtree with ignore_errors=True silently fails. - CopyOnWriteSandbox rollback from ACTIVE now uses dirs_exist_ok=True in copytree to prevent FileExistsError when rmtree with ignore_errors=True silently fails, matching the fix already applied to OverlaySandbox. - Non-rollbackable sandboxes (NoSandbox, TransactionSandbox) are committed last in the batch so that all rollbackable sandboxes commit first; if any rollbackable sandbox fails, none of the non-rollbackable sandboxes will have committed yet. - Moved NoSandbox and TransactionSandbox imports to module level in manager.py (no circular dependency exists). - Pre-commit backups are now created on the same filesystem as the original directory (using dir= argument to mkdtemp), avoiding cross-device copy overhead and ensuring os.rename compatibility. - safe_restore now renames the target into the mkdtemp directory instead of removing the mkdtemp dir first, eliminating the residual TOCTOU window between rmdir and rename. - safe_restore catches BaseException (not just OSError) to ensure the original directory is always renamed back on unexpected errors, preventing the original from being left in the renamed-aside state. - Added AtomicCommitError exception class to protocol.py carrying rolled_back_ids and failed_rollback_ids attributes. - Exported AtomicCommitError from sandbox package __init__.py so callers can import it from the public API. - Added BDD scenarios: LIFO rollback order, AtomicCommitError wrapping with RuntimeError cause and rollback metadata, _fs_utils backup/restore coverage, no-change commit rollback success, directory timestamp preservation, OverlaySandbox merged dir reset after COMMITTED rollback, CopyOnWriteSandbox rollback from COMMITTED restores original, GitWorktreeSandbox rollback from COMMITTED undoes merge, TransactionSandbox rollback from COMMITTED raises SandboxRollbackError about irreversible commit. ISSUES CLOSED: #925 Post-review hardening (PR #1146 review findings): - OverlaySandbox rollback from COMMITTED with no backup (no-op) now skips the merged directory reset entirely, preventing unnecessary unmount/remount or re-copy that could fail and turn a harmless no-op rollback into a SandboxRollbackError during commit_all atomic recovery. - OverlaySandbox rollback no longer double-wraps SandboxRollbackError: the outer except Exception handler now has a preceding except SandboxRollbackError clause that re-raises directly, avoiding a confusing double-wrapped error chain. - CopyOnWriteSandbox.get_path() now accepts ROLLED_BACK status for consistency with OverlaySandbox and the protocol status transition table (ROLLED_BACK -> ACTIVE). - CopyOnWriteSandbox rollback from COMMITTED now resets the sandbox copy from the restored original via rmtree+copytree, preventing stale pre-rollback modifications from being exposed on re-activation. - rollback_all now catches Exception (not just SandboxError) so that unexpected rollback errors do not prevent remaining sandboxes from being rolled back, consistent with the pattern already used in _rollback_committed. - commit_all docstring now documents a thread-safety warning: the method is not safe for concurrent calls on the same plan_id since sandbox commit/rollback runs outside the lock. - Fixed CHANGELOG.md whitespace inconsistencies (double leading spaces on two lines). Post-review hardening round 2 (PR #1146 automated review): - safe_restore now uses os.rename (O(1) atomic rename) instead of shutil.copytree (O(n) recursive copy) for the main restore path, since backup and target are always on the same filesystem. This eliminates the ENOTEMPTY bug where a partial copytree failure left target_path partially populated, causing the recovery os.rename to fail and strand the original in the stale temp directory. - OverlaySandbox.get_path() now transitions ROLLED_BACK to ACTIVE, matching CopyOnWriteSandbox and the protocol transition table (ROLLED_BACK -> ACTIVE). - GitWorktreeSandbox.get_path() now accepts ROLLED_BACK status for consistency with all other sandbox implementations and the protocol transition table (ROLLED_BACK -> ACTIVE). - rollback_all now also handles sandboxes in COMMITTED status (not just ACTIVE), consistent with the state machine allowing COMMITTED -> ROLLED_BACK. - cleanup_all now catches Exception (not just SandboxError) so a single unexpected error does not abort cleanup of remaining sandboxes, consistent with _rollback_committed and rollback_all. - Restructured CHANGELOG entry from a single ~90-line paragraph into structured sub-bullets for readability. - Added BDD scenarios: no-op rollback from COMMITTED for CopyOnWriteSandbox and OverlaySandbox (zero-change commit), commit ordering verification (rollbackable before non-rollbackable). Post-review hardening round 3 (PR #1146 deep automated review): - cleanup_abandoned now catches Exception (not just SandboxError) so that unexpected errors (e.g. raw OSError, PermissionError) do not crash the loop and prevent remaining abandoned sandboxes from being cleaned up, consistent with cleanup_all, rollback_all, and _rollback_committed. - OverlaySandbox._mount_overlay() now catches subprocess.TimeoutExpired (in addition to CalledProcessError and OSError), preventing create() from leaving the sandbox in PENDING status when mount hangs beyond the timeout. - OverlaySandbox._unmount_overlay() now catches subprocess.TimeoutExpired (in addition to CalledProcessError and OSError), preventing cleanup() from leaving the sandbox in a zombie state when umount hangs beyond the timeout. - OverlaySandbox._mount_overlay() validates that overlay paths do not contain commas, which would corrupt the OverlayFS mount options string (comma is the mount option delimiter). - GitWorktreeSandbox.commit() now checks git diff return code so that a failed diff command raises CalledProcessError instead of silently concluding there are no changes and skipping the merge. - safe_restore cleanup of the temporary rollback container now runs in a finally block, preventing a temp directory leak when the rename fails and the exception is re-raised. - Fixed misleading BDD step name: "backup path that will cause copytree to fail" renamed to "backup path that will cause rename to fail" since safe_restore now uses os.rename.
831 lines
31 KiB
Python
831 lines
31 KiB
Python
"""Step definitions for sandbox manager BDD coverage tests."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from datetime import datetime
|
|
from typing import Any
|
|
from unittest.mock import MagicMock, PropertyMock
|
|
|
|
from behave import given, then, when
|
|
|
|
from cleveragents.infrastructure.sandbox.factory import SandboxFactory
|
|
from cleveragents.infrastructure.sandbox.manager import SandboxManager
|
|
from cleveragents.infrastructure.sandbox.no_sandbox import NoSandbox
|
|
from cleveragents.infrastructure.sandbox.protocol import (
|
|
AtomicCommitError,
|
|
CommitResult,
|
|
SandboxContext,
|
|
SandboxError,
|
|
SandboxStatus,
|
|
)
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Helpers
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _make_mock_sandbox(
|
|
sandbox_id: str = "sb-mock-001",
|
|
status: SandboxStatus = SandboxStatus.CREATED,
|
|
resource_id: str = "res-mock",
|
|
original_path: str = "/tmp/mock",
|
|
plan_id: str = "plan-mock",
|
|
) -> MagicMock:
|
|
"""Create a mock sandbox satisfying the Sandbox protocol."""
|
|
sb = MagicMock()
|
|
sb.sandbox_id = sandbox_id
|
|
type(sb).status = PropertyMock(return_value=status)
|
|
sb.context = SandboxContext(
|
|
sandbox_id=sandbox_id,
|
|
sandbox_path=original_path,
|
|
original_path=original_path,
|
|
resource_id=resource_id,
|
|
plan_id=plan_id,
|
|
created_at=datetime.now(),
|
|
)
|
|
sb.create.return_value = sb.context
|
|
sb.commit.return_value = CommitResult(
|
|
sandbox_id=sandbox_id,
|
|
success=True,
|
|
timestamp=datetime.now(),
|
|
)
|
|
sb.rollback.return_value = None
|
|
sb.cleanup.return_value = None
|
|
return sb
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Background / Given steps
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@given("a sandbox factory instance")
|
|
def step_given_sandbox_factory(context: Any) -> None:
|
|
context.factory = SandboxFactory()
|
|
|
|
|
|
@given("a sandbox manager with the factory")
|
|
def step_given_sandbox_manager(context: Any) -> None:
|
|
context.manager = SandboxManager(factory=context.factory, cleanup_on_exit=False)
|
|
context.error = None
|
|
context.result = None
|
|
context.sandbox_result = None
|
|
context.commit_results = None
|
|
context.abandoned_count = None
|
|
context.first_sandbox = None
|
|
context._sandbox_refs = {}
|
|
context._rollback_call_log = []
|
|
context._committable_mock_keys = []
|
|
context.cov_error = None
|
|
context.sandbox_list = None
|
|
|
|
|
|
@given("a sandbox manager with cleanup_on_exit disabled")
|
|
def step_given_sandbox_manager_no_cleanup(context: Any) -> None:
|
|
context.manager = SandboxManager(factory=context.factory, cleanup_on_exit=False)
|
|
|
|
|
|
@given(
|
|
'a sandbox exists for plan "{plan_id}" resource "{res_id}" path "{path}" strategy "{strategy}"'
|
|
)
|
|
def step_given_sandbox_exists(
|
|
context: Any, plan_id: str, res_id: str, path: str, strategy: str
|
|
) -> None:
|
|
sandbox = context.manager.get_or_create_sandbox(
|
|
plan_id=plan_id,
|
|
resource_id=res_id,
|
|
original_path=path,
|
|
sandbox_strategy=strategy,
|
|
)
|
|
# Store first sandbox for identity comparison
|
|
key = f"{plan_id}:{res_id}"
|
|
try:
|
|
refs = context._sandbox_refs
|
|
except (KeyError, AttributeError):
|
|
refs = {}
|
|
context._sandbox_refs = refs
|
|
refs[key] = sandbox
|
|
|
|
|
|
@given("cleanup_all is patched to raise a RuntimeError")
|
|
def step_given_cleanup_all_raises(context: Any) -> None:
|
|
"""Patch cleanup_all to raise a RuntimeError to exercise the except block in atexit handler."""
|
|
|
|
def _raising_cleanup_all(plan_id: str) -> None:
|
|
raise RuntimeError("simulated cleanup_all failure")
|
|
|
|
context.manager.cleanup_all = _raising_cleanup_all
|
|
|
|
|
|
@given('the sandbox for plan "{plan_id}" resource "{res_id}" is activated')
|
|
def step_given_sandbox_activated(context: Any, plan_id: str, res_id: str) -> None:
|
|
sandbox = context.manager.get_sandbox(plan_id, res_id)
|
|
if sandbox is not None:
|
|
# Activating a NoSandbox: call get_path to transition CREATED -> ACTIVE
|
|
sandbox.get_path("dummy_file.txt")
|
|
|
|
|
|
@given('the sandbox for plan "{plan_id}" resource "{res_id}" is committed')
|
|
def step_given_sandbox_committed(context: Any, plan_id: str, res_id: str) -> None:
|
|
sandbox = context.manager.get_sandbox(plan_id, res_id)
|
|
if sandbox is not None:
|
|
# Ensure it's in a committable state
|
|
if sandbox.status == SandboxStatus.CREATED:
|
|
pass # CREATED can commit directly
|
|
sandbox.commit()
|
|
|
|
|
|
@given('the sandbox for plan "{plan_id}" resource "{res_id}" is cleaned up')
|
|
def step_given_sandbox_cleaned_up(context: Any, plan_id: str, res_id: str) -> None:
|
|
sandbox = context.manager.get_sandbox(plan_id, res_id)
|
|
if sandbox is not None:
|
|
sandbox.cleanup()
|
|
|
|
|
|
@given('the sandbox for plan "{plan_id}" resource "{res_id}" is rolled back')
|
|
def step_given_sandbox_rolled_back(context: Any, plan_id: str, res_id: str) -> None:
|
|
"""Put sandbox into ROLLED_BACK state using mock injection."""
|
|
sandbox = context.manager.get_sandbox(plan_id, res_id)
|
|
if sandbox is not None:
|
|
# NoSandbox.rollback always raises, so we need to inject a mock
|
|
# that has ROLLED_BACK status into the manager's tracking dict
|
|
mock_sb = _make_mock_sandbox(
|
|
sandbox_id=sandbox.sandbox_id,
|
|
status=SandboxStatus.ROLLED_BACK,
|
|
resource_id=res_id,
|
|
original_path="/tmp/repo",
|
|
plan_id=plan_id,
|
|
)
|
|
context.manager._active_sandboxes[plan_id][res_id] = mock_sb
|
|
key = f"{plan_id}:{res_id}"
|
|
try:
|
|
context._sandbox_refs[key] = mock_sb
|
|
except (KeyError, AttributeError):
|
|
context._sandbox_refs = {key: mock_sb}
|
|
|
|
|
|
@given('the sandbox for plan "{plan_id}" resource "{res_id}" is in errored state')
|
|
def step_given_sandbox_errored(context: Any, plan_id: str, res_id: str) -> None:
|
|
"""Put sandbox into ERRORED state by injecting a mock."""
|
|
sandbox = context.manager.get_sandbox(plan_id, res_id)
|
|
if sandbox is not None:
|
|
mock_sb = _make_mock_sandbox(
|
|
sandbox_id=sandbox.sandbox_id,
|
|
status=SandboxStatus.ERRORED,
|
|
resource_id=res_id,
|
|
original_path="/tmp/repo",
|
|
plan_id=plan_id,
|
|
)
|
|
context.manager._active_sandboxes[plan_id][res_id] = mock_sb
|
|
|
|
|
|
@given(
|
|
'the sandbox for plan "{plan_id}" resource "{res_id}" is replaced with a committable mock'
|
|
)
|
|
def step_given_sandbox_replaced_with_committable_mock(
|
|
context: Any, plan_id: str, res_id: str
|
|
) -> None:
|
|
"""Replace sandbox with a mock that commits successfully and supports rollback."""
|
|
sandbox = context.manager.get_sandbox(plan_id, res_id)
|
|
assert sandbox is not None, (
|
|
f"Sandbox for plan '{plan_id}' resource '{res_id}' must exist "
|
|
f"before it can be replaced with a committable mock"
|
|
)
|
|
if sandbox is not None:
|
|
mock_sb = _make_mock_sandbox(
|
|
sandbox_id=sandbox.sandbox_id,
|
|
status=SandboxStatus.ACTIVE,
|
|
resource_id=res_id,
|
|
)
|
|
key = f"{plan_id}:{res_id}"
|
|
|
|
# Add a side_effect to rollback that logs the call order
|
|
context._rollback_call_log = getattr(context, "_rollback_call_log", [])
|
|
committable_keys: list[str] = getattr(context, "_committable_mock_keys", [])
|
|
if key not in committable_keys:
|
|
committable_keys.append(key)
|
|
context._committable_mock_keys = committable_keys
|
|
|
|
def _log_rollback(k: str = key) -> None:
|
|
context._rollback_call_log.append(k)
|
|
|
|
mock_sb.rollback.side_effect = _log_rollback
|
|
|
|
context.manager._active_sandboxes[plan_id][res_id] = mock_sb
|
|
if not hasattr(context, "_sandbox_refs"):
|
|
context._sandbox_refs = {}
|
|
context._sandbox_refs[key] = mock_sb
|
|
|
|
|
|
@given('the sandbox for plan "{plan_id}" resource "{res_id}" will fail on commit')
|
|
def step_given_sandbox_commit_fails(context: Any, plan_id: str, res_id: str) -> None:
|
|
"""Make the sandbox's commit method raise a SandboxError."""
|
|
sandbox = context.manager.get_sandbox(plan_id, res_id)
|
|
if sandbox is not None:
|
|
# Replace with a mock that raises on commit
|
|
mock_sb = _make_mock_sandbox(
|
|
sandbox_id=sandbox.sandbox_id,
|
|
status=SandboxStatus.ACTIVE,
|
|
resource_id=res_id,
|
|
)
|
|
mock_sb.commit.side_effect = SandboxError("commit failed in test")
|
|
context.manager._active_sandboxes[plan_id][res_id] = mock_sb
|
|
|
|
|
|
@given(
|
|
'the sandbox for plan "{plan_id}" resource "{res_id}" will succeed commit then fail rollback'
|
|
)
|
|
def step_given_sandbox_commit_succeeds_rollback_fails(
|
|
context: Any, plan_id: str, res_id: str
|
|
) -> None:
|
|
"""Make a mock sandbox that commits successfully but fails on rollback."""
|
|
sandbox = context.manager.get_sandbox(plan_id, res_id)
|
|
assert sandbox is not None, (
|
|
f"Sandbox for plan '{plan_id}' resource '{res_id}' must exist "
|
|
f"before it can be configured to fail on rollback"
|
|
)
|
|
if sandbox is not None:
|
|
mock_sb = _make_mock_sandbox(
|
|
sandbox_id=sandbox.sandbox_id,
|
|
status=SandboxStatus.ACTIVE,
|
|
resource_id=res_id,
|
|
)
|
|
# commit succeeds (default mock behaviour)
|
|
# rollback fails
|
|
mock_sb.rollback.side_effect = SandboxError("rollback failed in test")
|
|
context.manager._active_sandboxes[plan_id][res_id] = mock_sb
|
|
|
|
|
|
@given(
|
|
'the sandbox for plan "{plan_id}" resource "{res_id}" is replaced with a non-rollbackable mock'
|
|
)
|
|
def step_given_sandbox_replaced_with_non_rollbackable(
|
|
context: Any, plan_id: str, res_id: str
|
|
) -> None:
|
|
"""Replace sandbox with a mock that passes isinstance(sb, NoSandbox).
|
|
|
|
This uses ``spec=NoSandbox`` so that ``isinstance`` checks in
|
|
``commit_all`` classify it as non-rollbackable, causing it to be
|
|
committed last in the batch.
|
|
"""
|
|
sandbox = context.manager.get_sandbox(plan_id, res_id)
|
|
assert sandbox is not None, (
|
|
f"Sandbox for plan '{plan_id}' resource '{res_id}' must exist "
|
|
f"before it can be replaced with a non-rollbackable mock"
|
|
)
|
|
mock_sb = MagicMock(spec=NoSandbox)
|
|
mock_sb.sandbox_id = sandbox.sandbox_id
|
|
type(mock_sb).status = PropertyMock(return_value=SandboxStatus.ACTIVE)
|
|
mock_sb.context = SandboxContext(
|
|
sandbox_id=sandbox.sandbox_id,
|
|
sandbox_path="/tmp/mock",
|
|
original_path="/tmp/mock",
|
|
resource_id=res_id,
|
|
plan_id=plan_id,
|
|
created_at=datetime.now(),
|
|
)
|
|
mock_sb.commit.return_value = CommitResult(
|
|
sandbox_id=sandbox.sandbox_id,
|
|
success=True,
|
|
timestamp=datetime.now(),
|
|
)
|
|
key = f"{plan_id}:{res_id}"
|
|
if not hasattr(context, "_non_rollbackable_refs"):
|
|
context._non_rollbackable_refs = {}
|
|
context._non_rollbackable_refs[key] = mock_sb
|
|
context.manager._active_sandboxes[plan_id][res_id] = mock_sb
|
|
|
|
|
|
@given('the sandbox for plan "{plan_id}" resource "{res_id}" will fail on rollback')
|
|
def step_given_sandbox_rollback_fails(context: Any, plan_id: str, res_id: str) -> None:
|
|
"""Make the sandbox's rollback method raise a SandboxError."""
|
|
sandbox = context.manager.get_sandbox(plan_id, res_id)
|
|
if sandbox is not None:
|
|
mock_sb = _make_mock_sandbox(
|
|
sandbox_id=sandbox.sandbox_id,
|
|
status=SandboxStatus.ACTIVE,
|
|
resource_id=res_id,
|
|
)
|
|
mock_sb.rollback.side_effect = SandboxError("rollback failed in test")
|
|
context.manager._active_sandboxes[plan_id][res_id] = mock_sb
|
|
|
|
|
|
@given('the sandbox for plan "{plan_id}" resource "{res_id}" will fail on cleanup')
|
|
def step_given_sandbox_cleanup_fails(context: Any, plan_id: str, res_id: str) -> None:
|
|
"""Make the sandbox's cleanup method raise a SandboxError."""
|
|
sandbox = context.manager.get_sandbox(plan_id, res_id)
|
|
if sandbox is not None:
|
|
# Preserve current status
|
|
current_status = sandbox.status
|
|
mock_sb = _make_mock_sandbox(
|
|
sandbox_id=sandbox.sandbox_id
|
|
if hasattr(sandbox, "_sandbox_id")
|
|
else "sb-mock",
|
|
status=current_status,
|
|
resource_id=res_id,
|
|
)
|
|
mock_sb.cleanup.side_effect = SandboxError("cleanup failed in test")
|
|
context.manager._active_sandboxes[plan_id][res_id] = mock_sb
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# When steps
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@when("I create a sandbox manager with None factory")
|
|
def step_when_create_manager_none_factory(context: Any) -> None:
|
|
try:
|
|
context.manager = SandboxManager(factory=None, cleanup_on_exit=False)
|
|
except ValueError as exc:
|
|
context.error = exc
|
|
|
|
|
|
@when(
|
|
'I get or create a sandbox for plan "{plan_id}" resource "{res_id}" path "{path}" strategy "{strategy}"'
|
|
)
|
|
def step_when_get_or_create_sandbox(
|
|
context: Any, plan_id: str, res_id: str, path: str, strategy: str
|
|
) -> None:
|
|
try:
|
|
context.sandbox_result = context.manager.get_or_create_sandbox(
|
|
plan_id=plan_id,
|
|
resource_id=res_id,
|
|
original_path=path,
|
|
sandbox_strategy=strategy,
|
|
)
|
|
except (ValueError, SandboxError) as exc:
|
|
context.error = exc
|
|
|
|
|
|
@when("I get or create a sandbox with empty plan_id")
|
|
def step_when_get_or_create_empty_plan(context: Any) -> None:
|
|
try:
|
|
context.sandbox_result = context.manager.get_or_create_sandbox(
|
|
plan_id="",
|
|
resource_id="res-001",
|
|
original_path="/tmp/repo",
|
|
sandbox_strategy="none",
|
|
)
|
|
except (ValueError, SandboxError) as exc:
|
|
context.error = exc
|
|
|
|
|
|
@when("I get or create a sandbox with empty resource_id")
|
|
def step_when_get_or_create_empty_resource(context: Any) -> None:
|
|
try:
|
|
context.sandbox_result = context.manager.get_or_create_sandbox(
|
|
plan_id="plan-001",
|
|
resource_id="",
|
|
original_path="/tmp/repo",
|
|
sandbox_strategy="none",
|
|
)
|
|
except (ValueError, SandboxError) as exc:
|
|
context.error = exc
|
|
|
|
|
|
@when("I get or create a sandbox with empty original_path")
|
|
def step_when_get_or_create_empty_path(context: Any) -> None:
|
|
try:
|
|
context.sandbox_result = context.manager.get_or_create_sandbox(
|
|
plan_id="plan-001",
|
|
resource_id="res-001",
|
|
original_path="",
|
|
sandbox_strategy="none",
|
|
)
|
|
except (ValueError, SandboxError) as exc:
|
|
context.error = exc
|
|
|
|
|
|
@when("I commit all sandboxes with empty plan_id")
|
|
def step_when_commit_all_empty_plan(context: Any) -> None:
|
|
try:
|
|
context.commit_results = context.manager.commit_all("")
|
|
except ValueError as exc:
|
|
context.error = exc
|
|
|
|
|
|
@when("I rollback all sandboxes with empty plan_id")
|
|
def step_when_rollback_all_empty_plan(context: Any) -> None:
|
|
try:
|
|
context.manager.rollback_all("")
|
|
except ValueError as exc:
|
|
context.error = exc
|
|
|
|
|
|
@when("I cleanup all sandboxes with empty plan_id")
|
|
def step_when_cleanup_all_empty_plan(context: Any) -> None:
|
|
try:
|
|
context.manager.cleanup_all("")
|
|
except ValueError as exc:
|
|
context.error = exc
|
|
|
|
|
|
@when('I get sandbox for plan "{plan_id}" resource "{res_id}"')
|
|
def step_when_get_sandbox(context: Any, plan_id: str, res_id: str) -> None:
|
|
context.sandbox_result = context.manager.get_sandbox(plan_id, res_id)
|
|
|
|
|
|
@when('I list sandboxes for plan "{plan_id}"')
|
|
def step_when_list_sandboxes(context: Any, plan_id: str) -> None:
|
|
context.sandbox_list = context.manager.list_sandboxes(plan_id)
|
|
|
|
|
|
@when('I commit all sandboxes for plan "{plan_id}"')
|
|
def step_when_commit_all(context: Any, plan_id: str) -> None:
|
|
try:
|
|
context.commit_results = context.manager.commit_all(plan_id)
|
|
except ValueError as exc:
|
|
context.error = exc
|
|
|
|
|
|
@when('I rollback all sandboxes for plan "{plan_id}"')
|
|
def step_when_rollback_all(context: Any, plan_id: str) -> None:
|
|
try:
|
|
context.manager.rollback_all(plan_id)
|
|
except ValueError as exc:
|
|
context.error = exc
|
|
|
|
|
|
@when('I cleanup all sandboxes for plan "{plan_id}"')
|
|
def step_when_cleanup_all(context: Any, plan_id: str) -> None:
|
|
try:
|
|
context.manager.cleanup_all(plan_id)
|
|
except ValueError as exc:
|
|
context.error = exc
|
|
|
|
|
|
@when("I cleanup abandoned sandboxes")
|
|
def step_when_cleanup_abandoned(context: Any) -> None:
|
|
context.abandoned_count = context.manager.cleanup_abandoned()
|
|
|
|
|
|
@when("the atexit cleanup handler runs")
|
|
def step_when_atexit_handler(context: Any) -> None:
|
|
try:
|
|
context.manager._cleanup_on_exit_handler()
|
|
context.error = None
|
|
except Exception as exc:
|
|
context.error = exc
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Then steps
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@then("the manager should have no active sandboxes")
|
|
def step_then_no_active_sandboxes(context: Any) -> None:
|
|
assert len(context.manager._active_sandboxes) == 0, (
|
|
f"Expected no active sandboxes, got {len(context.manager._active_sandboxes)}"
|
|
)
|
|
|
|
|
|
@then("the manager cleanup_on_exit flag should be true")
|
|
def step_then_cleanup_on_exit_true(context: Any) -> None:
|
|
manager = SandboxManager(factory=context.factory, cleanup_on_exit=True)
|
|
assert manager._cleanup_on_exit is True
|
|
|
|
|
|
@then("the manager cleanup_on_exit flag should be false")
|
|
def step_then_cleanup_on_exit_false(context: Any) -> None:
|
|
assert context.manager._cleanup_on_exit is False
|
|
|
|
|
|
@then('a sandbox ValueError should be raised with message "{message}"')
|
|
def step_then_sandbox_value_error(context: Any, message: str) -> None:
|
|
assert context.error is not None, "Expected a ValueError but no error was raised"
|
|
assert isinstance(context.error, ValueError), (
|
|
f"Expected ValueError, got {type(context.error).__name__}: {context.error}"
|
|
)
|
|
assert message in str(context.error), (
|
|
f"Expected message '{message}' in '{context.error}'"
|
|
)
|
|
|
|
|
|
@then("a sandbox should be returned")
|
|
def step_then_sandbox_returned(context: Any) -> None:
|
|
assert context.sandbox_result is not None, "Expected a sandbox but got None"
|
|
|
|
|
|
@then('the sandbox should have status "{status}"')
|
|
def step_then_sandbox_status(context: Any, status: str) -> None:
|
|
assert context.sandbox_result.status.value == status, (
|
|
f"Expected status '{status}', got '{context.sandbox_result.status.value}'"
|
|
)
|
|
|
|
|
|
@then('the sandbox should be tracked for plan "{plan_id}"')
|
|
def step_then_sandbox_tracked(context: Any, plan_id: str) -> None:
|
|
assert plan_id in context.manager._active_sandboxes, (
|
|
f"Plan '{plan_id}' not found in active sandboxes"
|
|
)
|
|
|
|
|
|
@then("the same sandbox instance should be returned")
|
|
def step_then_same_sandbox(context: Any) -> None:
|
|
# Compare with the first stored reference
|
|
for key, ref_sandbox in context._sandbox_refs.items():
|
|
plan_id, res_id = key.split(":")
|
|
current = context.manager.get_sandbox(plan_id, res_id)
|
|
if current is context.sandbox_result:
|
|
assert context.sandbox_result is ref_sandbox, (
|
|
"Expected the same sandbox instance but got a different one"
|
|
)
|
|
return
|
|
# If we got here with a result, check directly
|
|
assert context.sandbox_result is not None
|
|
|
|
|
|
@then("a different sandbox instance should be returned")
|
|
def step_then_different_sandbox(context: Any) -> None:
|
|
for key, ref_sandbox in context._sandbox_refs.items():
|
|
_plan_id, _res_id = key.split(":")
|
|
if (
|
|
context.sandbox_result is not None
|
|
and context.sandbox_result is not ref_sandbox
|
|
):
|
|
return # Found a different instance
|
|
raise AssertionError("Expected a different sandbox instance")
|
|
|
|
|
|
@then('{count:d} sandboxes should be tracked for plan "{plan_id}"')
|
|
def step_then_sandbox_count(context: Any, count: int, plan_id: str) -> None:
|
|
tracked = context.manager._active_sandboxes.get(plan_id, {})
|
|
assert len(tracked) == count, (
|
|
f"Expected {count} sandboxes for plan '{plan_id}', got {len(tracked)}"
|
|
)
|
|
|
|
|
|
@then("the sandbox result should be None")
|
|
def step_then_sandbox_result_none(context: Any) -> None:
|
|
assert context.sandbox_result is None, (
|
|
f"Expected None, got {context.sandbox_result}"
|
|
)
|
|
|
|
|
|
@then("the sandbox list should contain {count:d} sandboxes")
|
|
def step_then_sandbox_list_count(context: Any, count: int) -> None:
|
|
assert len(context.sandbox_list) == count, (
|
|
f"Expected {count} sandboxes in list, got {len(context.sandbox_list)}"
|
|
)
|
|
|
|
|
|
@then("{count:d} commit results should be returned")
|
|
def step_then_commit_results_count(context: Any, count: int) -> None:
|
|
assert context.commit_results is not None, "No commit results available"
|
|
assert len(context.commit_results) == count, (
|
|
f"Expected {count} commit results, got {len(context.commit_results)}"
|
|
)
|
|
|
|
|
|
@then("all commit results should be successful")
|
|
def step_then_all_commits_successful(context: Any) -> None:
|
|
for result in context.commit_results:
|
|
assert result.success is True, (
|
|
f"Expected successful commit, got error: {result.error}"
|
|
)
|
|
|
|
|
|
@then("the first commit result should have success false")
|
|
def step_then_first_commit_failed(context: Any) -> None:
|
|
assert context.commit_results is not None and len(context.commit_results) > 0
|
|
assert context.commit_results[0].success is False, (
|
|
"Expected first commit result to be unsuccessful"
|
|
)
|
|
|
|
|
|
@then("no sandbox error should be raised")
|
|
def step_then_no_sandbox_error(context: Any) -> None:
|
|
assert context.error is None, f"Unexpected error: {context.error}"
|
|
|
|
|
|
@then('plan "{plan_id}" should no longer be tracked')
|
|
def step_then_plan_not_tracked(context: Any, plan_id: str) -> None:
|
|
assert plan_id not in context.manager._active_sandboxes, (
|
|
f"Plan '{plan_id}' is still tracked in active sandboxes"
|
|
)
|
|
|
|
|
|
@then("the first commit result error should mention the failed sandbox")
|
|
def step_then_first_result_mentions_failed(context: Any) -> None:
|
|
assert context.commit_results is not None and len(context.commit_results) > 0
|
|
error = context.commit_results[0].error or ""
|
|
assert "Atomic commit failed" in error, (
|
|
f"Expected 'Atomic commit failed' in error: {error}"
|
|
)
|
|
|
|
|
|
@then("the first commit result metadata should list rolled back sandboxes")
|
|
def step_then_first_result_has_rolled_back(context: Any) -> None:
|
|
assert context.commit_results is not None and len(context.commit_results) > 0
|
|
metadata = context.commit_results[0].metadata
|
|
assert "rolled_back" in metadata, (
|
|
f"Expected 'rolled_back' key in metadata: {metadata}"
|
|
)
|
|
rolled_back = metadata["rolled_back"]
|
|
assert isinstance(rolled_back, list), (
|
|
f"Expected rolled_back to be a list, got {type(rolled_back)}"
|
|
)
|
|
assert len(rolled_back) > 0, "Expected at least one rolled-back sandbox ID"
|
|
|
|
|
|
@then("the first commit result metadata should list failed rollback sandboxes")
|
|
def step_then_first_result_has_failed_rollbacks(context: Any) -> None:
|
|
assert context.commit_results is not None and len(context.commit_results) > 0
|
|
metadata = context.commit_results[0].metadata
|
|
assert "rollback_failed" in metadata, (
|
|
f"Expected 'rollback_failed' key in metadata: {metadata}"
|
|
)
|
|
failed = metadata["rollback_failed"]
|
|
assert isinstance(failed, list), (
|
|
f"Expected rollback_failed to be a list, got {type(failed)}"
|
|
)
|
|
assert len(failed) > 0, "Expected at least one failed-rollback sandbox ID"
|
|
|
|
|
|
@then('the first commit result error should contain "{text}"')
|
|
def step_then_first_result_error_contains(context: Any, text: str) -> None:
|
|
assert context.commit_results is not None and len(context.commit_results) > 0
|
|
error = context.commit_results[0].error or ""
|
|
assert text in error, f"Expected '{text}' in error: {error}"
|
|
|
|
|
|
@given(
|
|
'the sandbox for plan "{plan_id}" resource "{res_id}" will raise RuntimeError on commit'
|
|
)
|
|
def step_given_sandbox_commit_raises_runtime_error(
|
|
context: Any, plan_id: str, res_id: str
|
|
) -> None:
|
|
"""Make a mock sandbox that raises RuntimeError (not SandboxError) on commit."""
|
|
sandbox = context.manager.get_sandbox(plan_id, res_id)
|
|
if sandbox is not None:
|
|
mock_sb = _make_mock_sandbox(
|
|
sandbox_id=sandbox.sandbox_id,
|
|
status=SandboxStatus.ACTIVE,
|
|
resource_id=res_id,
|
|
)
|
|
mock_sb.commit.side_effect = RuntimeError("unexpected non-sandbox error")
|
|
context.manager._active_sandboxes[plan_id][res_id] = mock_sb
|
|
|
|
|
|
@when('I commit all sandboxes for plan "{plan_id}" expecting a non-sandbox error')
|
|
def step_when_commit_all_expecting_non_sandbox_error(
|
|
context: Any, plan_id: str
|
|
) -> None:
|
|
"""Call commit_all and capture an AtomicCommitError wrapping a non-SandboxError."""
|
|
context.cov_error = None
|
|
try:
|
|
context.commit_results = context.manager.commit_all(plan_id)
|
|
except AtomicCommitError as exc:
|
|
# Non-SandboxError exceptions are now wrapped in AtomicCommitError
|
|
# with the original exception chained as __cause__.
|
|
context.cov_error = exc
|
|
except Exception as exc:
|
|
context.cov_error = exc
|
|
|
|
|
|
@then('the rollback order should be reverse of commit order for plan "{plan_id}"')
|
|
def step_then_rollback_order_is_reversed(context: Any, plan_id: str) -> None:
|
|
"""Verify that rollbacks were called in reverse order of commits.
|
|
|
|
Uses the rollback_call_log list populated by side_effect callbacks
|
|
on the committable mocks (set up in the "replaced with a committable
|
|
mock" step).
|
|
"""
|
|
rollback_log: list[str] = getattr(context, "_rollback_call_log", [])
|
|
committable: list[str] = getattr(context, "_committable_mock_keys", [])
|
|
assert len(rollback_log) >= 2, (
|
|
f"Need >= 2 rollback calls, got {len(rollback_log)}: {rollback_log}; "
|
|
f"committable_keys: {committable}"
|
|
)
|
|
# Verify that the rollback log is the exact reverse of the commit
|
|
# order (LIFO), using the known committable keys list as the
|
|
# commit-order reference rather than relying on lexicographic
|
|
# string comparison.
|
|
expected_order = list(reversed(committable))
|
|
assert rollback_log == expected_order, (
|
|
f"Rollback order {rollback_log} does not match expected "
|
|
f"reverse commit order {expected_order}"
|
|
)
|
|
|
|
|
|
@then("an AtomicCommitError should have been raised with RuntimeError as cause")
|
|
def step_then_atomic_commit_error_raised(context: Any) -> None:
|
|
"""Verify that an AtomicCommitError was raised wrapping a RuntimeError."""
|
|
assert context.cov_error is not None, "Expected an error to be raised"
|
|
assert isinstance(context.cov_error, AtomicCommitError), (
|
|
f"Expected AtomicCommitError, got {type(context.cov_error).__name__}"
|
|
)
|
|
assert context.cov_error.__cause__ is not None, (
|
|
"Expected AtomicCommitError to chain the original exception as __cause__"
|
|
)
|
|
assert isinstance(context.cov_error.__cause__, RuntimeError), (
|
|
f"Expected __cause__ to be RuntimeError, got "
|
|
f"{type(context.cov_error.__cause__).__name__}"
|
|
)
|
|
|
|
|
|
@then("the AtomicCommitError should carry rollback metadata")
|
|
def step_then_atomic_commit_error_has_metadata(context: Any) -> None:
|
|
"""Verify that the AtomicCommitError carries rollback metadata."""
|
|
assert isinstance(context.cov_error, AtomicCommitError)
|
|
assert isinstance(context.cov_error.rolled_back_ids, list), (
|
|
f"Expected rolled_back_ids to be a list, got "
|
|
f"{type(context.cov_error.rolled_back_ids)}"
|
|
)
|
|
assert isinstance(context.cov_error.failed_rollback_ids, list), (
|
|
f"Expected failed_rollback_ids to be a list, got "
|
|
f"{type(context.cov_error.failed_rollback_ids)}"
|
|
)
|
|
|
|
|
|
@then(
|
|
'the committable mock for plan "{plan_id}" resource "{res_id}" should have been rolled back'
|
|
)
|
|
def step_then_committable_mock_rolled_back(
|
|
context: Any, plan_id: str, res_id: str
|
|
) -> None:
|
|
key = f"{plan_id}:{res_id}"
|
|
rollback_log: list[str] = getattr(context, "_rollback_call_log", [])
|
|
assert key in rollback_log, f"Expected rollback for {key} in log: {rollback_log}"
|
|
|
|
|
|
@given(
|
|
'the sandbox for plan "{plan_id}" resource "{res_id}" is replaced with a no-change committable mock'
|
|
)
|
|
def step_given_sandbox_replaced_with_no_change_committable(
|
|
context: Any, plan_id: str, res_id: str
|
|
) -> None:
|
|
"""Replace sandbox with a mock that commits with no changes (no backup).
|
|
|
|
Simulates the case where ``commit()`` detects no diff and skips the
|
|
pre-commit backup. The mock's ``rollback()`` logs the call so we can
|
|
verify it was invoked and did not raise.
|
|
"""
|
|
sandbox = context.manager.get_sandbox(plan_id, res_id)
|
|
if sandbox is not None:
|
|
mock_sb = _make_mock_sandbox(
|
|
sandbox_id=sandbox.sandbox_id,
|
|
status=SandboxStatus.ACTIVE,
|
|
resource_id=res_id,
|
|
)
|
|
key = f"{plan_id}:{res_id}"
|
|
context._rollback_call_log = getattr(context, "_rollback_call_log", [])
|
|
|
|
def _log_rollback(k: str = key) -> None:
|
|
context._rollback_call_log.append(k)
|
|
|
|
mock_sb.rollback.side_effect = _log_rollback
|
|
context.manager._active_sandboxes[plan_id][res_id] = mock_sb
|
|
if not hasattr(context, "_sandbox_refs"):
|
|
context._sandbox_refs = {}
|
|
context._sandbox_refs[key] = mock_sb
|
|
|
|
|
|
@then(
|
|
'the no-change sandbox rollback should have succeeded for plan "{plan_id}" resource "{res_id}"'
|
|
)
|
|
def step_then_no_change_rollback_succeeded(
|
|
context: Any, plan_id: str, res_id: str
|
|
) -> None:
|
|
"""Verify that the no-change sandbox was rolled back (not failed)."""
|
|
key = f"{plan_id}:{res_id}"
|
|
rollback_log: list[str] = getattr(context, "_rollback_call_log", [])
|
|
assert key in rollback_log, (
|
|
f"Expected rollback for no-change sandbox {key} in log: {rollback_log}"
|
|
)
|
|
# Verify the sandbox is NOT in the failed_rollback_ids
|
|
if context.commit_results:
|
|
metadata = context.commit_results[0].metadata
|
|
failed = metadata.get("rollback_failed", [])
|
|
assert key.split(":")[1] not in [sid for sid in failed], (
|
|
f"No-change sandbox should not appear in failed_rollback_ids: {failed}"
|
|
)
|
|
|
|
|
|
@then(
|
|
'the non-rollbackable mock for plan "{plan_id}" resource "{res_id}" should not have been committed'
|
|
)
|
|
def step_then_non_rollbackable_not_committed(
|
|
context: Any, plan_id: str, res_id: str
|
|
) -> None:
|
|
"""Verify that the non-rollbackable mock was never committed.
|
|
|
|
Because rollbackable sandboxes are committed first and one of them
|
|
failed, ``commit_all`` should not have reached the non-rollbackable
|
|
sandbox.
|
|
"""
|
|
key = f"{plan_id}:{res_id}"
|
|
refs = getattr(context, "_non_rollbackable_refs", {})
|
|
mock_sb = refs.get(key)
|
|
assert mock_sb is not None, f"No non-rollbackable ref found for {key}"
|
|
mock_sb.commit.assert_not_called()
|
|
|
|
|
|
@then("{count:d} abandoned sandboxes should be cleaned up")
|
|
def step_then_abandoned_count(context: Any, count: int) -> None:
|
|
assert context.abandoned_count == count, (
|
|
f"Expected {count} abandoned sandboxes cleaned, got {context.abandoned_count}"
|
|
)
|