freemo
b122ec7ed5
CI / lint (pull_request) Successful in 23s
CI / quality (pull_request) Successful in 51s
CI / typecheck (pull_request) Successful in 1m0s
CI / security (pull_request) Successful in 55s
CI / build (pull_request) Successful in 25s
CI / helm (pull_request) Successful in 32s
CI / push-validation (pull_request) Successful in 26s
CI / e2e_tests (pull_request) Successful in 3m38s
CI / integration_tests (pull_request) Successful in 6m42s
CI / unit_tests (pull_request) Successful in 8m19s
CI / docker (pull_request) Successful in 13s
CI / coverage (pull_request) Successful in 15m23s
CI / status-check (pull_request) Successful in 2s
Remove the local ${PYTHON} python (and python3) variable definitions from
the *** Variables *** sections of all affected robot files. These local
definitions were overriding the pabot-injected venv Python path passed via
--variable PYTHON:/path/to/venv/python, causing tests to use the system
Python (which lacks required packages like structlog, sqlalchemy, etc.)
instead of the nox venv Python.
The correct ${PYTHON} value is already set by Setup Test Environment in
common.resource via sys.executable, and pabot passes it via --variable.
The local fallback definitions are redundant and harmful in parallel runs.
Audit found 56 robot files with the pattern (more than the 9 originally
identified in the issue). All occurrences have been removed.
ISSUES CLOSED: #1309
530 lines
26 KiB
Plaintext
530 lines
26 KiB
Plaintext
*** Settings ***
|
|
Documentation Integration tests for retry policy wiring into services
|
|
... Tests the ServiceRetryWiring, RetryPolicyConfig, CircuitBreakerConfig,
|
|
... and per-service retry policies with circuit breaker protection.
|
|
|
|
Library OperatingSystem
|
|
Library Process
|
|
Library String
|
|
Library Collections
|
|
Library BuiltIn
|
|
Resource ${CURDIR}/common.resource
|
|
|
|
Default Tags retry wiring integration
|
|
|
|
Test Setup Setup Retry Policy Test Environment
|
|
Test Teardown Cleanup Retry Policy Test Environment
|
|
|
|
*** Variables ***
|
|
${WORKSPACE_ROOT} ${CURDIR}/..
|
|
${TEST_FILE} ${EMPTY}
|
|
${TEST_OUTPUT} ${EMPTY}
|
|
${TIMEOUT} 30s
|
|
|
|
*** Test Cases ***
|
|
Test RetryPolicyConfig Default Values
|
|
[Documentation] Verify RetryPolicyConfig default construction
|
|
${output} = Run Retry Test Script retry_policy_defaults
|
|
Should Match Regexp ${output} \\bmax_attempts=3\\b
|
|
Should Contain ${output} base_delay=1.0
|
|
Should Contain ${output} max_delay=60.0
|
|
Should Contain ${output} jitter=True
|
|
Should Contain ${output} backoff_strategy=exponential
|
|
|
|
Test CircuitBreakerConfig Default Values
|
|
[Documentation] Verify CircuitBreakerConfig default construction
|
|
${output} = Run Retry Test Script cb_config_defaults
|
|
Should Match Regexp ${output} \\bfailure_threshold=5\\b
|
|
Should Contain ${output} recovery_timeout=60.0
|
|
Should Contain ${output} half_open_max_successes=2
|
|
Should Contain ${output} enabled=True
|
|
|
|
Test ServiceRetryPolicyRegistry Known Service
|
|
[Documentation] Verify registry returns policy for known service
|
|
${output} = Run Retry Test Script registry_known
|
|
Should Contain ${output} service_name=plan_service
|
|
Should Contain ${output} has_description=True
|
|
|
|
Test ServiceRetryPolicyRegistry Override
|
|
[Documentation] Verify registry applies config overrides
|
|
${output} = Run Retry Test Script registry_override
|
|
Should Match Regexp ${output} \\bmax_attempts=7\\b
|
|
|
|
Test ServiceRetryWiring Idempotent Retry
|
|
[Documentation] Verify wiring retries idempotent operations
|
|
${output} = Run Retry Test Script wiring_idempotent
|
|
Should Contain ${output} result=success
|
|
Should Contain ${output} call_count=2
|
|
|
|
Test ServiceRetryWiring Non-Idempotent Skips Retry
|
|
[Documentation] Verify wiring does not retry non-idempotent operations
|
|
${output} = Run Retry Test Script wiring_non_idempotent
|
|
Should Contain ${output} call_count=1
|
|
Should Contain ${output} failed=True
|
|
|
|
Test ServiceRetryWiring Circuit Breaker Open
|
|
[Documentation] Verify wiring raises CircuitBreakerOpen when circuit is open
|
|
${output} = Run Retry Test Script wiring_cb_open
|
|
Should Contain ${output} CircuitBreakerOpen raised
|
|
|
|
Test Retry Policy Validation
|
|
[Documentation] Verify max_delay >= base_delay validation
|
|
${output} = Run Retry Test Script validation
|
|
Should Contain ${output} ValidationError raised
|
|
|
|
Test Settings Retry Config Keys
|
|
[Documentation] Verify Settings has retry configuration fields
|
|
${output} = Run Retry Test Script settings_keys
|
|
Should Match Regexp ${output} \\bretry_max_attempts=3\\b
|
|
Should Match Regexp ${output} \\bcircuit_breaker_failure_threshold=5\\b
|
|
|
|
Test Category Defaults Coverage
|
|
[Documentation] Verify CATEGORY_DEFAULTS covers all categories
|
|
${output} = Run Retry Test Script category_defaults
|
|
Should Contain ${output} all_categories_covered=True
|
|
|
|
Test Services Package Exports
|
|
[Documentation] Verify star-import from application.services works and
|
|
... ServiceRetryWiring is importable via the public package API
|
|
${output} = Run Retry Test Script services_package_exports
|
|
Should Contain ${output} star_import_ok=True
|
|
Should Contain ${output} ServiceRetryWiring_importable=True
|
|
Should Contain ${output} no_ContextFragment=True
|
|
|
|
Test CircuitBreakerState Enum Importable And Usable
|
|
[Documentation] Verify CircuitBreakerState StrEnum is importable, has expected
|
|
... members, and string comparisons still work
|
|
${output} = Run Retry Test Script circuit_breaker_state_enum
|
|
Should Contain ${output} CLOSED=closed
|
|
Should Contain ${output} OPEN=open
|
|
Should Contain ${output} HALF_OPEN=half-open
|
|
Should Contain ${output} is_strenum=True
|
|
Should Contain ${output} str_compare_ok=True
|
|
|
|
Test Structured Log Emission On Retry
|
|
[Documentation] Verify that retry attempts emit structured log output
|
|
... including service name, operation, attempt number, and error
|
|
${output} = Run Retry Test Script structured_log_emission
|
|
Should Contain ${output} has_retry_log=True
|
|
Should Contain ${output} has_attempt_field=True
|
|
Should Contain ${output} has_service_field=True
|
|
|
|
Test Unicode Control Char Rejection In ServiceRetryPolicy
|
|
[Documentation] Verify ServiceRetryPolicy.service_name rejects Unicode control
|
|
... characters while accepting valid service names
|
|
${output} = Run Retry Test Script unicode_control_char_rejection
|
|
Should Contain ${output} valid_name_ok=True
|
|
Should Contain ${output} null_byte_rejected=True
|
|
Should Contain ${output} bel_char_rejected=True
|
|
Should Contain ${output} zero_width_rejected=True
|
|
|
|
Test Async Wiring Execute With Retry
|
|
[Documentation] Verify ServiceRetryWiring.async_execute retries idempotent
|
|
... async operations correctly
|
|
${output} = Run Retry Test Script async_wiring_execute
|
|
Should Contain ${output} async_result=success
|
|
Should Contain ${output} async_call_count=2
|
|
|
|
Test Circuit Breaker Half Open Cooldown
|
|
[Documentation] Verify circuit breaker cooldown prevents premature half-open
|
|
... transition and allows it after cooldown expires
|
|
${output} = Run Retry Test Script cb_half_open_cooldown
|
|
Should Contain ${output} cooldown_blocks=True
|
|
Should Contain ${output} after_cooldown_allows=True
|
|
|
|
*** Keywords ***
|
|
Setup Retry Policy Test Environment
|
|
[Documentation] Create temporary directory for test files
|
|
${temp_dir}= Evaluate tempfile.mkdtemp() modules=tempfile
|
|
${python_exec}= Evaluate sys.executable modules=sys
|
|
Set Test Variable ${PYTHON} ${python_exec}
|
|
Set Test Variable ${TEMP_DIR} ${temp_dir}
|
|
Set Test Variable ${TEST_FILE} ${temp_dir}/retry_policy_test.py
|
|
Set Test Variable ${TEST_OUTPUT} ${temp_dir}/retry_policy_output.txt
|
|
|
|
Cleanup Retry Policy Test Environment
|
|
[Documentation] Clean up temporary test files
|
|
Run Keyword And Ignore Error Remove File ${TEST_FILE}
|
|
Run Keyword And Ignore Error Remove File ${TEST_OUTPUT}
|
|
Run Keyword And Ignore Error Remove Directory ${TEMP_DIR} recursive=True
|
|
|
|
Run Retry Test Script
|
|
[Arguments] ${test_type}
|
|
[Documentation] Create and run a Python test script, returning its output
|
|
Create Test Script ${test_type}
|
|
${result} = Run Process ${PYTHON} ${TEST_FILE}
|
|
... stdout=${TEST_OUTPUT} stderr=STDOUT timeout=${TIMEOUT} on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0 msg=Script ${test_type} failed:\n${result.stdout}
|
|
${output} = Get File ${TEST_OUTPUT}
|
|
RETURN ${output}
|
|
|
|
Create Test Script
|
|
[Arguments] ${test_type}
|
|
[Documentation] Create a Python test script for the given test type
|
|
${script} = Run Keyword Get ${test_type} Script
|
|
Create File ${TEST_FILE} ${script}
|
|
|
|
Get retry_policy_defaults Script
|
|
${script} = Catenate SEPARATOR=\n
|
|
... import sys
|
|
... sys.path.insert(0, '${WORKSPACE_ROOT}/src')
|
|
... from cleveragents.domain.models.core.retry_policy import RetryPolicyConfig
|
|
... p = RetryPolicyConfig()
|
|
... print(f"max_attempts={p.max_attempts}", flush=True)
|
|
... print(f"base_delay={p.base_delay}", flush=True)
|
|
... print(f"max_delay={p.max_delay}", flush=True)
|
|
... print(f"jitter={p.jitter}", flush=True)
|
|
... print(f"backoff_strategy={p.backoff_strategy.value}", flush=True)
|
|
... sys.exit(0)
|
|
RETURN ${script}
|
|
|
|
Get cb_config_defaults Script
|
|
${script} = Catenate SEPARATOR=\n
|
|
... import sys
|
|
... sys.path.insert(0, '${WORKSPACE_ROOT}/src')
|
|
... from cleveragents.domain.models.core.retry_policy import CircuitBreakerConfig
|
|
... c = CircuitBreakerConfig()
|
|
... print(f"failure_threshold={c.failure_threshold}", flush=True)
|
|
... print(f"recovery_timeout={c.recovery_timeout}", flush=True)
|
|
... print(f"half_open_max_successes={c.half_open_max_successes}", flush=True)
|
|
... print(f"enabled={c.enabled}", flush=True)
|
|
... sys.exit(0)
|
|
RETURN ${script}
|
|
|
|
Get registry_known Script
|
|
${script} = Catenate SEPARATOR=\n
|
|
... import sys
|
|
... sys.path.insert(0, '${WORKSPACE_ROOT}/src')
|
|
... from cleveragents.domain.models.core.retry_policy import ServiceRetryPolicyRegistry
|
|
... reg = ServiceRetryPolicyRegistry()
|
|
... p = reg.get("plan_service")
|
|
... print(f"service_name={p.service_name}", flush=True)
|
|
... print(f"has_description={bool(p.description)}", flush=True)
|
|
... sys.exit(0)
|
|
RETURN ${script}
|
|
|
|
Get registry_override Script
|
|
${script} = Catenate SEPARATOR=\n
|
|
... import sys
|
|
... sys.path.insert(0, '${WORKSPACE_ROOT}/src')
|
|
... from cleveragents.domain.models.core.retry_policy import ServiceRetryPolicyRegistry
|
|
... reg = ServiceRetryPolicyRegistry()
|
|
... reg.apply_overrides({"plan_service": {"retry": {"max_attempts": 7}}})
|
|
... p = reg.get("plan_service")
|
|
... print(f"max_attempts={p.retry.max_attempts}", flush=True)
|
|
... sys.exit(0)
|
|
RETURN ${script}
|
|
|
|
Get wiring_idempotent Script
|
|
${script} = Catenate SEPARATOR=\n
|
|
... import sys
|
|
... import os
|
|
... sys.path.insert(0, '${WORKSPACE_ROOT}/src')
|
|
... import logging
|
|
... logging.getLogger().setLevel(logging.ERROR)
|
|
... os.environ['PYTHONUNBUFFERED'] = '1'
|
|
... from cleveragents.config.settings import Settings
|
|
... from cleveragents.application.services.service_retry_wiring import ServiceRetryWiring
|
|
... settings = Settings(database_url="sqlite:///${TEMP_DIR}/test.db", mock_providers=True, env="test")
|
|
... wiring = ServiceRetryWiring(settings)
|
|
... call_count = 0
|
|
... def fail_once():
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}global call_count
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}call_count += 1
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}if call_count <= 1:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}raise RuntimeError("transient")
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}return "success"
|
|
... result = wiring.execute("plan_service", "test_op", fail_once, idempotent=True)
|
|
... print(f"result={result}", flush=True)
|
|
... print(f"call_count={call_count}", flush=True)
|
|
... sys.exit(0)
|
|
RETURN ${script}
|
|
|
|
Get wiring_non_idempotent Script
|
|
${script} = Catenate SEPARATOR=\n
|
|
... import sys
|
|
... import os
|
|
... sys.path.insert(0, '${WORKSPACE_ROOT}/src')
|
|
... import logging
|
|
... logging.getLogger().setLevel(logging.ERROR)
|
|
... os.environ['PYTHONUNBUFFERED'] = '1'
|
|
... from cleveragents.config.settings import Settings
|
|
... from cleveragents.application.services.service_retry_wiring import ServiceRetryWiring
|
|
... settings = Settings(database_url="sqlite:///${TEMP_DIR}/test.db", mock_providers=True, env="test")
|
|
... wiring = ServiceRetryWiring(settings)
|
|
... call_count = 0
|
|
... def always_fail():
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}global call_count
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}call_count += 1
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}raise RuntimeError("fail")
|
|
... failed = False
|
|
... try:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}wiring.execute("plan_service", "test_write", always_fail, idempotent=False)
|
|
... except RuntimeError:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}failed = True
|
|
... print(f"call_count={call_count}", flush=True)
|
|
... print(f"failed={failed}", flush=True)
|
|
... sys.exit(0)
|
|
RETURN ${script}
|
|
|
|
Get wiring_cb_open Script
|
|
${script} = Catenate SEPARATOR=\n
|
|
... import sys
|
|
... import os
|
|
... sys.path.insert(0, '${WORKSPACE_ROOT}/src')
|
|
... import logging
|
|
... logging.getLogger().setLevel(logging.ERROR)
|
|
... os.environ['PYTHONUNBUFFERED'] = '1'
|
|
... from cleveragents.config.settings import Settings
|
|
... from cleveragents.application.services.service_retry_wiring import ServiceRetryWiring
|
|
... from cleveragents.core.retry_patterns import CircuitBreakerOpen
|
|
... from cleveragents.core.circuit_breaker import CircuitBreakerState
|
|
... settings = Settings(database_url="sqlite:///${TEMP_DIR}/test.db", mock_providers=True, env="test")
|
|
... wiring = ServiceRetryWiring(settings)
|
|
... cb = wiring.get_circuit_breaker("plan_service")
|
|
... # Drive the circuit breaker open through repeated failures
|
|
... for i in range(cb.failure_threshold):
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}try:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}cb.call(lambda: (_ for _ in ()).throw(Exception("fail")))
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}except Exception:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}pass
|
|
... assert cb.state == CircuitBreakerState.OPEN, f"Expected OPEN, got {cb.state}"
|
|
... try:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}wiring.execute("plan_service", "test", lambda: "ok", idempotent=True)
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}print("No exception raised", flush=True)
|
|
... except CircuitBreakerOpen:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}print("CircuitBreakerOpen raised", flush=True)
|
|
... sys.exit(0)
|
|
RETURN ${script}
|
|
|
|
Get validation Script
|
|
${script} = Catenate SEPARATOR=\n
|
|
... import sys
|
|
... sys.path.insert(0, '${WORKSPACE_ROOT}/src')
|
|
... from pydantic import ValidationError
|
|
... from cleveragents.domain.models.core.retry_policy import RetryPolicyConfig
|
|
... try:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}RetryPolicyConfig(base_delay=10.0, max_delay=5.0)
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}print("No error", flush=True)
|
|
... except ValidationError:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}print("ValidationError raised", flush=True)
|
|
... sys.exit(0)
|
|
RETURN ${script}
|
|
|
|
Get settings_keys Script
|
|
${script} = Catenate SEPARATOR=\n
|
|
... import sys
|
|
... sys.path.insert(0, '${WORKSPACE_ROOT}/src')
|
|
... from cleveragents.config.settings import Settings
|
|
... s = Settings(database_url="sqlite:///${TEMP_DIR}/test.db", mock_providers=True, env="test")
|
|
... print(f"retry_max_attempts={s.retry_max_attempts}", flush=True)
|
|
... print(f"circuit_breaker_failure_threshold={s.circuit_breaker_failure_threshold}", flush=True)
|
|
... sys.exit(0)
|
|
RETURN ${script}
|
|
|
|
Get category_defaults Script
|
|
${script} = Catenate SEPARATOR=\n
|
|
... import sys
|
|
... sys.path.insert(0, '${WORKSPACE_ROOT}/src')
|
|
... from cleveragents.domain.models.core.retry_policy import CATEGORY_DEFAULTS, RetryCategory
|
|
... all_covered = all(cat in CATEGORY_DEFAULTS for cat in RetryCategory)
|
|
... print(f"all_categories_covered={all_covered}", flush=True)
|
|
... sys.exit(0)
|
|
RETURN ${script}
|
|
|
|
Get services_package_exports Script
|
|
${script} = Catenate SEPARATOR=\n
|
|
... import sys
|
|
... sys.path.insert(0, '${WORKSPACE_ROOT}/src')
|
|
... # Test 1: star-import must not raise ImportError (U1 removed phantom ContextFragment)
|
|
... try:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}exec("from cleveragents.application.services import *")
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}print("star_import_ok=True", flush=True)
|
|
... except ImportError as e:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}print(f"star_import_ok=False error={e}", flush=True)
|
|
... # Test 2: ServiceRetryWiring is importable via the public package API (U3)
|
|
... try:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}from cleveragents.application.services import ServiceRetryWiring
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}print(f"ServiceRetryWiring_importable={ServiceRetryWiring is not None}", flush=True)
|
|
... except ImportError as e:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}print(f"ServiceRetryWiring_importable=False error={e}", flush=True)
|
|
... # Test 3: ContextFragment must NOT be in __all__ (U1)
|
|
... from cleveragents.application.services import __all__ as exports
|
|
... print(f"no_ContextFragment={'ContextFragment' not in exports}", flush=True)
|
|
... sys.exit(0)
|
|
RETURN ${script}
|
|
|
|
Get circuit_breaker_state_enum Script
|
|
${script} = Catenate SEPARATOR=\n
|
|
... import sys
|
|
... import enum
|
|
... sys.path.insert(0, '${WORKSPACE_ROOT}/src')
|
|
... from cleveragents.core.circuit_breaker import CircuitBreakerState
|
|
... # Verify enum members
|
|
... print(f"CLOSED={CircuitBreakerState.CLOSED.value}", flush=True)
|
|
... print(f"OPEN={CircuitBreakerState.OPEN.value}", flush=True)
|
|
... print(f"HALF_OPEN={CircuitBreakerState.HALF_OPEN.value}", flush=True)
|
|
... # Verify it is a StrEnum
|
|
... print(f"is_strenum={issubclass(CircuitBreakerState, (str, enum.Enum))}", flush=True)
|
|
... # Verify string comparison still works
|
|
... ok = (CircuitBreakerState.CLOSED == "closed"
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}and CircuitBreakerState.OPEN == "open"
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}and CircuitBreakerState.HALF_OPEN == "half-open")
|
|
... print(f"str_compare_ok={ok}", flush=True)
|
|
... sys.exit(0)
|
|
RETURN ${script}
|
|
|
|
Get structured_log_emission Script
|
|
${script} = Catenate SEPARATOR=\n
|
|
... import sys
|
|
... import os
|
|
... import json
|
|
... sys.path.insert(0, '${WORKSPACE_ROOT}/src')
|
|
... import logging
|
|
... import structlog
|
|
... os.environ['PYTHONUNBUFFERED'] = '1'
|
|
... # Install a structlog capture processor to intercept log events
|
|
... captured = []
|
|
... def capture_processor(logger, method_name, event_dict):
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}captured.append(dict(event_dict))
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}raise structlog.DropEvent
|
|
... structlog.configure(
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}processors=[capture_processor],
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}wrapper_class=structlog.make_filtering_bound_logger(logging.DEBUG),
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}logger_factory=structlog.PrintLoggerFactory(),
|
|
... )
|
|
... from cleveragents.config.settings import Settings
|
|
... from cleveragents.application.services.service_retry_wiring import ServiceRetryWiring
|
|
... settings = Settings(database_url="sqlite:///${TEMP_DIR}/test.db", mock_providers=True, env="test")
|
|
... wiring = ServiceRetryWiring(settings)
|
|
... call_count = 0
|
|
... def fail_once():
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}global call_count
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}call_count += 1
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}if call_count <= 1:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}raise RuntimeError("transient")
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}return "ok"
|
|
... wiring.execute("plan_service", "read_plan", fail_once, idempotent=True)
|
|
... # Inspect captured logs for retry-related events
|
|
... retry_logs = [e for e in captured if "retry" in str(e.get("event", "")).lower()
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}or "attempt" in str(e).lower()]
|
|
... has_retry = len(retry_logs) > 0
|
|
... has_attempt = any("attempt" in str(e) for e in retry_logs)
|
|
... has_service = any("service" in str(e) for e in retry_logs)
|
|
... print(f"has_retry_log={has_retry}", flush=True)
|
|
... print(f"has_attempt_field={has_attempt}", flush=True)
|
|
... print(f"has_service_field={has_service}", flush=True)
|
|
... sys.exit(0)
|
|
RETURN ${script}
|
|
|
|
Get unicode_control_char_rejection Script
|
|
${script} = Catenate SEPARATOR=\n
|
|
... import sys
|
|
... sys.path.insert(0, '${WORKSPACE_ROOT}/src')
|
|
... from pydantic import ValidationError
|
|
... from cleveragents.domain.models.core.retry_policy import ServiceRetryPolicy
|
|
... # Valid name should succeed
|
|
... try:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}ServiceRetryPolicy(service_name="plan_service")
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}print("valid_name_ok=True", flush=True)
|
|
... except ValidationError:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}print("valid_name_ok=False", flush=True)
|
|
... # Null byte should be rejected
|
|
... try:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}ServiceRetryPolicy(service_name="plan\\x00service")
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}print("null_byte_rejected=False", flush=True)
|
|
... except ValidationError:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}print("null_byte_rejected=True", flush=True)
|
|
... # BEL character should be rejected
|
|
... try:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}ServiceRetryPolicy(service_name="plan\\x07service")
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}print("bel_char_rejected=False", flush=True)
|
|
... except ValidationError:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}print("bel_char_rejected=True", flush=True)
|
|
... # Zero-width space (U+200B) should be rejected
|
|
... try:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}ServiceRetryPolicy(service_name="plan\\u200bservice")
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}print("zero_width_rejected=False", flush=True)
|
|
... except ValidationError:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}print("zero_width_rejected=True", flush=True)
|
|
... sys.exit(0)
|
|
RETURN ${script}
|
|
|
|
Get async_wiring_execute Script
|
|
${script} = Catenate SEPARATOR=\n
|
|
... import sys
|
|
... import os
|
|
... import asyncio
|
|
... sys.path.insert(0, '${WORKSPACE_ROOT}/src')
|
|
... import logging
|
|
... logging.getLogger().setLevel(logging.ERROR)
|
|
... os.environ['PYTHONUNBUFFERED'] = '1'
|
|
... from cleveragents.config.settings import Settings
|
|
... from cleveragents.application.services.service_retry_wiring import ServiceRetryWiring
|
|
... settings = Settings(database_url="sqlite:///${TEMP_DIR}/test.db", mock_providers=True, env="test")
|
|
... wiring = ServiceRetryWiring(settings)
|
|
... call_count = 0
|
|
... async def async_fail_once():
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}global call_count
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}call_count += 1
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}if call_count <= 1:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}raise RuntimeError("transient async")
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}return "success"
|
|
... async def main():
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}result = await wiring.async_execute(
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}"plan_service", "async_read", async_fail_once, idempotent=True)
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}return result
|
|
... result = asyncio.run(main())
|
|
... print(f"async_result={result}", flush=True)
|
|
... print(f"async_call_count={call_count}", flush=True)
|
|
... sys.exit(0)
|
|
RETURN ${script}
|
|
|
|
Get cb_half_open_cooldown Script
|
|
${script} = Catenate SEPARATOR=\n
|
|
... import sys
|
|
... import os
|
|
... import time
|
|
... sys.path.insert(0, '${WORKSPACE_ROOT}/src')
|
|
... import logging
|
|
... import structlog
|
|
... logging.getLogger().setLevel(logging.ERROR)
|
|
... structlog.configure(wrapper_class=structlog.make_filtering_bound_logger(logging.ERROR))
|
|
... os.environ['PYTHONUNBUFFERED'] = '1'
|
|
... from cleveragents.core.retry_patterns import CircuitBreaker, CircuitBreakerOpen
|
|
... from cleveragents.core.circuit_breaker import CircuitBreakerState
|
|
... # Create CB with short recovery and moderate cooldown
|
|
... cb = CircuitBreaker(failure_threshold=1, recovery_timeout=0.1, cooldown_seconds=0.5)
|
|
... # Phase 1: Open the circuit
|
|
... try:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}cb.call(lambda: (_ for _ in ()).throw(Exception("fail")))
|
|
... except Exception:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}pass
|
|
... assert cb.state == CircuitBreakerState.OPEN
|
|
... # Phase 2: Wait past recovery_timeout, enter half-open, then fail again to re-open
|
|
... time.sleep(0.15)
|
|
... try:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}cb.call(lambda: (_ for _ in ()).throw(Exception("fail again")))
|
|
... except (Exception, CircuitBreakerOpen):
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}pass
|
|
... # Now the CB is open again and _last_half_open_time is set
|
|
... assert cb.state == CircuitBreakerState.OPEN
|
|
... # Phase 3: Wait past recovery_timeout but NOT past cooldown
|
|
... time.sleep(0.15)
|
|
... try:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}cb.call(lambda: "probe")
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}blocks = False
|
|
... except CircuitBreakerOpen:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}blocks = True
|
|
... print(f"cooldown_blocks={blocks}", flush=True)
|
|
... # Phase 4: Wait past cooldown
|
|
... time.sleep(0.5)
|
|
... try:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}cb.call(lambda: "probe")
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}allows = True
|
|
... except CircuitBreakerOpen:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}allows = False
|
|
... print(f"after_cooldown_allows={allows}", flush=True)
|
|
... sys.exit(0)
|
|
RETURN ${script}
|