TEST-INFRA: [ci-execution-time] Investigate and report on the slowest running tests #2309

Closed
opened 2026-04-03 13:46:02 +00:00 by freemo · 1 comment
Owner

Metadata

  • Branch: chore/ci-execution-time-slowest-test-report
  • Commit Message: chore(ci): investigate and report on the slowest running tests
  • Milestone: v3.2.0
  • Parent Epic: #1678

Summary

The current test infrastructure does not provide a clear report of the slowest running tests. This makes it difficult to identify and prioritize optimization efforts. The integration_tests session already has a @slow tag, which is a good start, but a more systematic approach is needed.

Proposed Solution

  1. Generate a test execution time report: Configure behave and robot to generate a report of the execution time for each test scenario. This can be done using existing flags or plugins (e.g., --junit for behave, which includes timing information).
  2. Publish the report as a CI artifact: The generated report should be published as a CI artifact for every test run.
  3. Analyze the report to identify the slowest tests: The report should be analyzed to identify the top 10-20 slowest tests.
  4. Create issues for optimizing the slowest tests: For each of the slowest tests, a new issue should be created to investigate and implement performance improvements.

Subtasks

  • Update the unit_tests nox session to generate a JUnit-style test execution time report (e.g., via --junit flag for behave)
  • Update the integration_tests nox session to generate a JUnit-style test execution time report (e.g., via --outputdir and timing options for robot)
  • Update the CI configuration (.forgejo/workflows/) to publish the generated test execution time reports as downloadable CI artifacts
  • Create a new nox session (e.g., slowest_tests) that parses the generated reports and prints a ranked summary of the top 10-20 slowest test scenarios
  • Run the new session locally to produce an initial report of the slowest tests
  • Create individual issues for optimizing the top 10 slowest tests identified in the report
  • Run nox (all default sessions) and fix any errors
  • Verify coverage >= 97% via nox -s coverage_report

Definition of Done

  • The unit_tests and integration_tests nox sessions are updated to generate a test execution time report.
  • The CI configuration is updated to publish the test execution time report as a CI artifact.
  • A new nox session is created to analyze the report and print a summary of the slowest tests.
  • Issues are created for optimizing the top 10 slowest tests.
  • All nox stages pass.
  • Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `chore/ci-execution-time-slowest-test-report` - **Commit Message**: `chore(ci): investigate and report on the slowest running tests` - **Milestone**: v3.2.0 - **Parent Epic**: #1678 ## Summary The current test infrastructure does not provide a clear report of the slowest running tests. This makes it difficult to identify and prioritize optimization efforts. The `integration_tests` session already has a `@slow` tag, which is a good start, but a more systematic approach is needed. ## Proposed Solution 1. **Generate a test execution time report:** Configure `behave` and `robot` to generate a report of the execution time for each test scenario. This can be done using existing flags or plugins (e.g., `--junit` for behave, which includes timing information). 2. **Publish the report as a CI artifact:** The generated report should be published as a CI artifact for every test run. 3. **Analyze the report to identify the slowest tests:** The report should be analyzed to identify the top 10-20 slowest tests. 4. **Create issues for optimizing the slowest tests:** For each of the slowest tests, a new issue should be created to investigate and implement performance improvements. ## Subtasks - [ ] Update the `unit_tests` nox session to generate a JUnit-style test execution time report (e.g., via `--junit` flag for `behave`) - [ ] Update the `integration_tests` nox session to generate a JUnit-style test execution time report (e.g., via `--outputdir` and timing options for `robot`) - [ ] Update the CI configuration (`.forgejo/workflows/`) to publish the generated test execution time reports as downloadable CI artifacts - [ ] Create a new nox session (e.g., `slowest_tests`) that parses the generated reports and prints a ranked summary of the top 10-20 slowest test scenarios - [ ] Run the new session locally to produce an initial report of the slowest tests - [ ] Create individual issues for optimizing the top 10 slowest tests identified in the report - [ ] Run `nox` (all default sessions) and fix any errors - [ ] Verify coverage >= 97% via `nox -s coverage_report` ## Definition of Done - [ ] The `unit_tests` and `integration_tests` nox sessions are updated to generate a test execution time report. - [ ] The CI configuration is updated to publish the test execution time report as a CI artifact. - [ ] A new nox session is created to analyze the report and print a summary of the slowest tests. - [ ] Issues are created for optimizing the top 10 slowest tests. - [ ] All nox stages pass. - [ ] Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator
freemo added this to the v3.2.0 milestone 2026-04-03 13:46:13 +00:00
Author
Owner

Closing as duplicate of #2315.

This issue ("Investigate and report on the slowest running tests") is a subtask of #2315 ("Investigate and optimize slow integration_tests suite (22m+ runtime)"), which already includes profiling as its first subtask. Consolidating to avoid duplicate work.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Closing as duplicate of #2315. This issue ("Investigate and report on the slowest running tests") is a subtask of #2315 ("Investigate and optimize slow integration_tests suite (22m+ runtime)"), which already includes profiling as its first subtask. Consolidating to avoid duplicate work. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#2309
No description provided.