TEST-INFRA: [missing-test-levels] Add ASV benchmarks for core module #1921

Closed
opened 2026-04-03 00:13:24 +00:00 by freemo · 2 comments
Owner

Metadata

  • Branch: test/core-asv-benchmarks
  • Commit Message: test(core): add ASV performance benchmarks for core module
  • Milestone: v3.8.0
  • Parent Epic: #1678

Description

The core module (cleveragents/core/) is missing dedicated ASV performance benchmarks for several of its key components. While a handful of adjacent benchmark files incidentally import from core (e.g. security_exception_bench.py covers error_handling and exceptions, security_async_cleanup_bench.py covers async_cleanup), the following source files have no direct ASV benchmark coverage at all:

  • circuit_breaker.pyCircuitBreaker, CircuitBreakerState, CircuitBreakerOpen
  • retry_patterns.pyretry_with_exponential_backoff, async_retry_with_exponential_backoff, retry_network_operation, retry_provider_operation, retry_database_operation, retry_file_operation, retry_with_jitter, retry_with_timeout, retry_on_result, get_retry_decorator, configure_retry_logging
  • retry_service_patterns.pyretry_service_operation, retry_auto_debug, RetryContext, is_read_only_plan_operation

The multi-level testing mandate in CONTRIBUTING.md requires every module to have performance benchmarks (ASV) covering its key operations. These gaps must be closed to ensure latency regressions in the core resilience primitives are caught early.

Subtasks

  • Audit circuit_breaker.py to identify all public behaviors requiring ASV benchmark suites
  • Create benchmarks/core_circuit_breaker_bench.py with suites for CircuitBreaker (closed-state call overhead, state transitions, open-state fast-fail latency)
  • Audit retry_patterns.py to identify key decorator-construction and invocation paths requiring benchmarks
  • Create benchmarks/core_retry_patterns_bench.py with suites for decorator construction (retry_with_exponential_backoff, retry_with_jitter, retry_with_timeout, retry_on_result) and happy-path invocation overhead
  • Audit retry_service_patterns.py to identify key paths requiring benchmarks
  • Create benchmarks/core_retry_service_patterns_bench.py with suites for retry_service_operation decorator overhead and retry_auto_debug happy-path latency
  • Verify all new benchmark suites are discovered by ASV (asv run or nox -s benchmarks)
  • Run nox (all default sessions), fix any errors
  • Verify coverage >= 97% via nox -s coverage_report

Definition of Done

  • All subtasks above are completed and checked off.
  • benchmarks/core_circuit_breaker_bench.py exists and benchmarks CircuitBreaker in closed state, open state (fast-fail), and state-transition paths.
  • benchmarks/core_retry_patterns_bench.py exists and benchmarks decorator construction and happy-path invocation for the public retry factory functions in retry_patterns.py.
  • benchmarks/core_retry_service_patterns_bench.py exists and benchmarks retry_service_operation decorator overhead and retry_auto_debug happy-path latency.
  • All new benchmark suites pass asv run without errors.
  • All nox stages pass.
  • Coverage >= 97%.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.

Automated by CleverAgents Bot
Supervisor: Unknown | Agent: ca-new-issue-creator

## Metadata - **Branch**: `test/core-asv-benchmarks` - **Commit Message**: `test(core): add ASV performance benchmarks for core module` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Description The `core` module (`cleveragents/core/`) is missing dedicated ASV performance benchmarks for several of its key components. While a handful of adjacent benchmark files incidentally import from `core` (e.g. `security_exception_bench.py` covers `error_handling` and `exceptions`, `security_async_cleanup_bench.py` covers `async_cleanup`), the following source files have **no direct ASV benchmark coverage** at all: - `circuit_breaker.py` — `CircuitBreaker`, `CircuitBreakerState`, `CircuitBreakerOpen` - `retry_patterns.py` — `retry_with_exponential_backoff`, `async_retry_with_exponential_backoff`, `retry_network_operation`, `retry_provider_operation`, `retry_database_operation`, `retry_file_operation`, `retry_with_jitter`, `retry_with_timeout`, `retry_on_result`, `get_retry_decorator`, `configure_retry_logging` - `retry_service_patterns.py` — `retry_service_operation`, `retry_auto_debug`, `RetryContext`, `is_read_only_plan_operation` The multi-level testing mandate in CONTRIBUTING.md requires every module to have performance benchmarks (ASV) covering its key operations. These gaps must be closed to ensure latency regressions in the core resilience primitives are caught early. ## Subtasks - [ ] Audit `circuit_breaker.py` to identify all public behaviors requiring ASV benchmark suites - [ ] Create `benchmarks/core_circuit_breaker_bench.py` with suites for `CircuitBreaker` (closed-state call overhead, state transitions, open-state fast-fail latency) - [ ] Audit `retry_patterns.py` to identify key decorator-construction and invocation paths requiring benchmarks - [ ] Create `benchmarks/core_retry_patterns_bench.py` with suites for decorator construction (`retry_with_exponential_backoff`, `retry_with_jitter`, `retry_with_timeout`, `retry_on_result`) and happy-path invocation overhead - [ ] Audit `retry_service_patterns.py` to identify key paths requiring benchmarks - [ ] Create `benchmarks/core_retry_service_patterns_bench.py` with suites for `retry_service_operation` decorator overhead and `retry_auto_debug` happy-path latency - [ ] Verify all new benchmark suites are discovered by ASV (`asv run` or `nox -s benchmarks`) - [ ] Run `nox` (all default sessions), fix any errors - [ ] Verify coverage >= 97% via `nox -s coverage_report` ## Definition of Done - [ ] All subtasks above are completed and checked off. - [ ] `benchmarks/core_circuit_breaker_bench.py` exists and benchmarks `CircuitBreaker` in closed state, open state (fast-fail), and state-transition paths. - [ ] `benchmarks/core_retry_patterns_bench.py` exists and benchmarks decorator construction and happy-path invocation for the public retry factory functions in `retry_patterns.py`. - [ ] `benchmarks/core_retry_service_patterns_bench.py` exists and benchmarks `retry_service_operation` decorator overhead and `retry_auto_debug` happy-path latency. - [ ] All new benchmark suites pass `asv run` without errors. - [ ] All nox stages pass. - [ ] Coverage >= 97%. - [ ] A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - [ ] The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - [ ] The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. --- **Automated by CleverAgents Bot** Supervisor: Unknown | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-03 00:13:29 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • MoSCoW: MoSCoW/Could Have — CI/test infrastructure improvement.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: MoSCoW/Could Have — CI/test infrastructure improvement. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Owner

Implementation Attempt — Tier 1: haiku — Success

Implemented three new ASV benchmark suites for the core module:

  1. core_circuit_breaker_bench.py - Comprehensive benchmarks for CircuitBreaker class covering:

    • Closed state call overhead
    • Open state fast-fail latency
    • State transitions (closed→open, open→half-open, half-open→closed)
    • Async operations
    • Initialization overhead
  2. core_retry_patterns_bench.py - Benchmarks for retry decorators including:

    • Decorator construction overhead
    • Happy-path invocation overhead
    • Async decorator operations
    • Function arguments handling
    • Factory function performance
    • Various configuration scenarios
  3. core_retry_service_patterns_bench.py - Service-level retry benchmarks covering:

    • retry_service_operation decorator performance
    • retry_auto_debug decorator overhead
    • RetryContext operations
    • is_read_only_plan_operation utility
    • Different wait strategies (exponential, linear, fixed, jitter)

All benchmark suites follow ASV conventions and are discoverable by the ASV framework. The implementation ensures latency regressions in core resilience primitives are caught early.

Pull Request: #10961


Automated by CleverAgents Bot
Supervisor: Implementation | Agent: implementation-worker

**Implementation Attempt** — Tier 1: haiku — Success Implemented three new ASV benchmark suites for the core module: 1. **core_circuit_breaker_bench.py** - Comprehensive benchmarks for CircuitBreaker class covering: - Closed state call overhead - Open state fast-fail latency - State transitions (closed→open, open→half-open, half-open→closed) - Async operations - Initialization overhead 2. **core_retry_patterns_bench.py** - Benchmarks for retry decorators including: - Decorator construction overhead - Happy-path invocation overhead - Async decorator operations - Function arguments handling - Factory function performance - Various configuration scenarios 3. **core_retry_service_patterns_bench.py** - Service-level retry benchmarks covering: - retry_service_operation decorator performance - retry_auto_debug decorator overhead - RetryContext operations - is_read_only_plan_operation utility - Different wait strategies (exponential, linear, fixed, jitter) All benchmark suites follow ASV conventions and are discoverable by the ASV framework. The implementation ensures latency regressions in core resilience primitives are caught early. Pull Request: #10961 --- Automated by CleverAgents Bot Supervisor: Implementation | Agent: implementation-worker
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#1921
No description provided.