Proposal: improve ca-test-infra-improver — strengthen duplicate avoidance to prevent near-identical issues #2370

Closed
opened 2026-04-03 17:23:59 +00:00 by freemo · 5 comments
Owner

Agent Improvement Proposal

Pattern Detected

Type: Duplicate work / issue tracker noise
Affected Agent: ca-test-infra-improver (Worker Mode)
Evidence:

The test infrastructure improver workers are filing large numbers of near-duplicate issues. The current duplicate avoidance section (lines 338-343) instructs workers to "Search Forgejo for existing issues with 'TEST-INFRA:' prefix" and "Check for similar titles/descriptions", but this is clearly insufficient.

Specific duplicate clusters found in the open issue tracker:

Topic Duplicate Issues Count
Parallelize E2E tests #2278, #2298, #2319, #2322, #2336, #2341, #2344 7
Increase default parallelism #2209, #2277, #2283, #2285, #2296 5
Parallelize Nox sessions #2251, #2293, #2294 3
Custom Docker image for CI #2290, #2324, #2325 3
Parallelize integration tests #2328, #2330, #2332, #2333 4
Optimize coverage report #2263, #2286, #2299 3
Cache nox environments #2248, #2297 2
Matrix strategy for Python versions #2239, #2270, #2223 3

That's approximately 30 duplicate issues across 8 topic clusters. Each worker instance appears to independently identify the same improvement opportunities and file new issues without recognizing that another worker already filed an equivalent issue.

Proposed Change

Strengthen the Duplicate Avoidance section with more rigorous matching:

  1. Require keyword-based search, not just title prefix search. Instead of searching only for "TEST-INFRA:" prefix, search for the key technical terms in the proposed issue (e.g., "parallelize", "E2E", "nox", "Docker image", "coverage").

  2. Add a mandatory keyword extraction step before filing:

    Before filing any issue:
    1. Extract 3-5 key technical terms from your proposed title
    2. Search Forgejo for open issues containing ANY of those terms
    3. For each search result, compare the proposed improvement with
       the existing issue's description
    4. If ANY existing issue proposes the same or substantially similar
       improvement, SKIP — do not file
    
  3. Add a broader similarity threshold. The current guidance says "Check for similar titles/descriptions" but doesn't define "similar." Add: "Two issues are duplicates if they propose the same optimization for the same component, even if the wording differs. For example, 'Parallelize E2E tests' and 'Run E2E tests in parallel' are duplicates."

  4. Add a cross-worker coordination note. Since multiple workers run in parallel, add: "Other workers in this pool may be filing issues simultaneously. After filing an issue, wait 5 seconds and re-check for duplicates. If a duplicate appeared in that window, close your issue as a duplicate of the earlier one."

Expected Impact

  • Dramatically fewer duplicate TEST-INFRA issues (estimated 60-70% reduction).
  • Cleaner issue tracker with unique, actionable improvement proposals.
  • Less grooming work needed to close duplicates.

Risk Assessment

  • Low risk: Stricter duplicate checking may occasionally cause a worker to skip a genuinely unique issue that has superficial keyword overlap with an existing one. This is acceptable — missing one improvement suggestion is far less costly than filing 7 duplicates.
  • Slight performance impact: Additional Forgejo API calls for keyword searches, but this is negligible compared to the analysis work.

This is a proposal from the agent evolver. A human must approve this issue before the change will be implemented. To approve: remove the needs feedback label, add State/Verified, or comment with approval.


Automated by CleverAgents Bot
Supervisor: Agent Evolver | Agent: ca-agent-evolver

## Agent Improvement Proposal ### Pattern Detected **Type**: Duplicate work / issue tracker noise **Affected Agent**: `ca-test-infra-improver` (Worker Mode) **Evidence**: The test infrastructure improver workers are filing large numbers of near-duplicate issues. The current duplicate avoidance section (lines 338-343) instructs workers to "Search Forgejo for existing issues with 'TEST-INFRA:' prefix" and "Check for similar titles/descriptions", but this is clearly insufficient. Specific duplicate clusters found in the open issue tracker: | Topic | Duplicate Issues | Count | |---|---|---| | Parallelize E2E tests | #2278, #2298, #2319, #2322, #2336, #2341, #2344 | **7** | | Increase default parallelism | #2209, #2277, #2283, #2285, #2296 | **5** | | Parallelize Nox sessions | #2251, #2293, #2294 | **3** | | Custom Docker image for CI | #2290, #2324, #2325 | **3** | | Parallelize integration tests | #2328, #2330, #2332, #2333 | **4** | | Optimize coverage report | #2263, #2286, #2299 | **3** | | Cache nox environments | #2248, #2297 | **2** | | Matrix strategy for Python versions | #2239, #2270, #2223 | **3** | That's approximately **30 duplicate issues** across 8 topic clusters. Each worker instance appears to independently identify the same improvement opportunities and file new issues without recognizing that another worker already filed an equivalent issue. ### Proposed Change Strengthen the Duplicate Avoidance section with more rigorous matching: 1. **Require keyword-based search, not just title prefix search.** Instead of searching only for "TEST-INFRA:" prefix, search for the key technical terms in the proposed issue (e.g., "parallelize", "E2E", "nox", "Docker image", "coverage"). 2. **Add a mandatory keyword extraction step** before filing: ``` Before filing any issue: 1. Extract 3-5 key technical terms from your proposed title 2. Search Forgejo for open issues containing ANY of those terms 3. For each search result, compare the proposed improvement with the existing issue's description 4. If ANY existing issue proposes the same or substantially similar improvement, SKIP — do not file ``` 3. **Add a broader similarity threshold.** The current guidance says "Check for similar titles/descriptions" but doesn't define "similar." Add: "Two issues are duplicates if they propose the same optimization for the same component, even if the wording differs. For example, 'Parallelize E2E tests' and 'Run E2E tests in parallel' are duplicates." 4. **Add a cross-worker coordination note.** Since multiple workers run in parallel, add: "Other workers in this pool may be filing issues simultaneously. After filing an issue, wait 5 seconds and re-check for duplicates. If a duplicate appeared in that window, close your issue as a duplicate of the earlier one." ### Expected Impact - Dramatically fewer duplicate TEST-INFRA issues (estimated 60-70% reduction). - Cleaner issue tracker with unique, actionable improvement proposals. - Less grooming work needed to close duplicates. ### Risk Assessment - **Low risk**: Stricter duplicate checking may occasionally cause a worker to skip a genuinely unique issue that has superficial keyword overlap with an existing one. This is acceptable — missing one improvement suggestion is far less costly than filing 7 duplicates. - **Slight performance impact**: Additional Forgejo API calls for keyword searches, but this is negligible compared to the analysis work. --- *This is a proposal from the agent evolver. A human must approve this issue before the change will be implemented. To approve: remove the `needs feedback` label, add `State/Verified`, or comment with approval.* --- **Automated by CleverAgents Bot** Supervisor: Agent Evolver | Agent: ca-agent-evolver
freemo added this to the v3.7.0 milestone 2026-04-03 17:24:50 +00:00
Author
Owner

approved

approved
Author
Owner

Label compliance fix applied:

  • Removed conflicting label: Priority/Backlog
  • Kept: Priority/High
  • Reason: Issue had 2 conflicting Priority/* labels simultaneously. Per CONTRIBUTING.md, only one Priority/* label is permitted. Priority/High is the more specific/actionable priority and was retained.

Automated by CleverAgents Bot
Supervisor: Backlog Grooming | Agent: ca-backlog-groomer

Label compliance fix applied: - Removed conflicting label: `Priority/Backlog` - Kept: `Priority/High` - Reason: Issue had 2 conflicting `Priority/*` labels simultaneously. Per CONTRIBUTING.md, only one `Priority/*` label is permitted. `Priority/High` is the more specific/actionable priority and was retained. --- **Automated by CleverAgents Bot** Supervisor: Backlog Grooming | Agent: ca-backlog-groomer
Author
Owner

Label compliance fix applied:

  • Added missing labels: Priority/Backlog, State/Unverified, Type/Task
  • Reason: Issue was missing all required labels per CONTRIBUTING.md. Inferred Type/Task from the "Proposal:" prefix in the title. Applied Priority/Backlog and State/Unverified as safe defaults.

Automated by CleverAgents Bot
Supervisor: Backlog Grooming | Agent: ca-backlog-groomer

Label compliance fix applied: - Added missing labels: `Priority/Backlog`, `State/Unverified`, `Type/Task` - Reason: Issue was missing all required labels per CONTRIBUTING.md. Inferred `Type/Task` from the "Proposal:" prefix in the title. Applied `Priority/Backlog` and `State/Unverified` as safe defaults. --- **Automated by CleverAgents Bot** Supervisor: Backlog Grooming | Agent: ca-backlog-groomer
Author
Owner

PR #2798 reviewed, approved, and merged.

The duplicate avoidance section in ca-test-infra-improver.md has been expanded from a 3-line checklist to a comprehensive 6-step process including keyword extraction, semantic comparison, cross-agent prefix checking, and post-filing verification. This should reduce duplicate TEST-INFRA issues by an estimated 60-70%.


Automated by CleverAgents Bot
Supervisor: PR Review | Agent: ca-pr-self-reviewer

PR #2798 reviewed, approved, and merged. The duplicate avoidance section in `ca-test-infra-improver.md` has been expanded from a 3-line checklist to a comprehensive 6-step process including keyword extraction, semantic comparison, cross-agent prefix checking, and post-filing verification. This should reduce duplicate TEST-INFRA issues by an estimated 60-70%. --- **Automated by CleverAgents Bot** Supervisor: PR Review | Agent: ca-pr-self-reviewer
Author
Owner

PR #2798 reviewed, approved, and merged.

The duplicate avoidance section in ca-test-infra-improver.md has been expanded from a 3-step checklist to a comprehensive 6-step process with keyword extraction, semantic comparison, cross-agent prefix checking, and post-filing verification. All four proposed changes from this issue were faithfully implemented.


Automated by CleverAgents Bot
Supervisor: PR Review | Agent: ca-pr-self-reviewer

PR #2798 reviewed, approved, and merged. The duplicate avoidance section in `ca-test-infra-improver.md` has been expanded from a 3-step checklist to a comprehensive 6-step process with keyword extraction, semantic comparison, cross-agent prefix checking, and post-filing verification. All four proposed changes from this issue were faithfully implemented. --- **Automated by CleverAgents Bot** Supervisor: PR Review | Agent: ca-pr-self-reviewer
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cleveragents/cleveragents-core#2370
No description provided.