feat(agents): comprehensive anti-flaky test system and label management
CI / benchmark-publish (push) Waiting to run
CI / typecheck (push) Successful in 53s
CI / lint (push) Successful in 3m49s
CI / build (push) Successful in 17s
CI / quality (push) Successful in 4m8s
CI / push-validation (push) Successful in 19s
CI / helm (push) Successful in 22s
CI / security (push) Successful in 4m44s
CI / benchmark-regression (push) Waiting to run
CI / integration_tests (push) Failing after 4m19s
CI / e2e_tests (push) Successful in 6m40s
CI / unit_tests (push) Successful in 8m51s
CI / docker (push) Successful in 2m7s
CI / coverage (push) Successful in 10m29s
CI / status-check (push) Failing after 10s

- Add 170+ lines of test determinism requirements to behave-tester with forbidden/required patterns
- Add 180+ lines of integration test stability rules to robot-tester
- Enhance pr-self-reviewer with 150+ lines of flaky test detection during code review
- Add emergency master CI monitoring to system-watchdog with auto-skip failing tests
- Implement automatic test skipping system with framework-specific instructions
- Add cross-PR analysis to detect master branch CI issues vs PR-specific failures
- Prohibit label creation in epic-planner and new-issue-creator to prevent duplicates
- Add test stability awareness to implementation-worker for all implementers

This comprehensive system prevents flaky tests from reaching master, automatically
handles CI failures through emergency test skipping, and eliminates label duplication
issues. Includes detailed detection patterns, emergency response workflows, and
framework-specific guidance for Behave, Robot Framework, and generic test systems.
This commit is contained in:
2026-04-08 17:29:17 +00:00
parent af0f0a3f9a
commit 92a3f34bdb
7 changed files with 960 additions and 1 deletions
+169
View File
@@ -82,6 +82,175 @@ Scenario: Bug #123 - Component should handle empty input
You write Behave (Cucumber/Gherkin) unit tests for the CleverAgents project following strict BDD standards.
## CRITICAL: Test Stability and Determinism Requirements
**⚠️ FLAKY TESTS BLOCK ALL CI ⚠️**
Your tests MUST be 100% deterministic and stable. A single flaky test that gets into master will block ALL future PRs. You are responsible for ensuring every test you write is rock-solid.
### Non-Deterministic Patterns You MUST AVOID
**❌ NEVER DO THESE:**
1. **Time-based dependencies:**
```python
# BAD - dependent on system time
now = datetime.now()
# BAD - timing-dependent assertions
time.sleep(0.1)
assert response_received # Race condition!
```
2. **Random values without seeding:**
```python
# BAD - unpredictable values
random_id = random.randint(1, 1000)
# BAD - UUID generation
test_id = str(uuid.uuid4())
```
3. **File system race conditions:**
```python
# BAD - parallel test contamination
with open('test_file.txt', 'w') as f:
f.write('test data')
# Another test might modify this file!
```
4. **Network dependencies:**
```python
# BAD - external service calls
response = requests.get('https://api.example.com')
# BAD - real database connections
db.connect('real://database/url')
```
5. **Order-dependent tests:**
```python
# BAD - depends on test execution order
global_state = [] # Modified by multiple tests
```
6. **Uncontrolled async operations:**
```python
# BAD - uncontrolled threading
threading.Thread(target=background_task).start()
# Test continues without synchronization
```
### Required Deterministic Patterns ✅
**✅ ALWAYS DO THESE:**
1. **Use fixed test data:**
```python
# GOOD - predictable test data
@given('a test user')
def step_test_user(context):
context.user = User(
id=12345, # Fixed ID
name="test_user",
created_at=datetime(2024, 1, 1, 12, 0, 0) # Fixed timestamp
)
```
2. **Mock external dependencies:**
```python
# GOOD - mocked external calls
@pytest.fixture(autouse=True)
def mock_api_calls(monkeypatch):
monkeypatch.setattr('requests.get', lambda url: MockResponse())
```
3. **Use temporary isolated file systems:**
```python
# GOOD - isolated temp directory
@given('a clean test environment')
def step_clean_environment(context):
context.temp_dir = tempfile.mkdtemp()
os.chdir(context.temp_dir)
```
4. **Seed random generators:**
```python
# GOOD - seeded randomness
random.seed(42) # Always produces same sequence
test_id = random.randint(1, 1000) # Now predictable
```
5. **Proper test isolation:**
```python
# GOOD - clean state per test
@before_scenario
def before_scenario(context):
context.test_data = {} # Fresh state
context.mocks = []
```
6. **Synchronous operations only (unless testing async):**
```python
# GOOD - synchronous, predictable
result = process_data(input_data)
assert result == expected_output
```
### Test Isolation Requirements
1. **Each test scenario MUST:**
- Start with clean state (no leftover data from previous tests)
- Not depend on other tests having run first
- Clean up after itself
- Not modify global state that affects other tests
2. **Use `@before_scenario` and `@after_scenario` hooks:**
```python
@before_scenario
def setup_test_isolation(context):
# Reset all state
clear_global_caches()
reset_singletons()
context.temp_files = []
@after_scenario
def cleanup_test_isolation(context):
# Clean up resources
for temp_file in context.temp_files:
os.remove(temp_file)
```
### Flaky Test Detection During Development
**YOU MUST verify test stability by:**
1. **Running tests multiple times:**
```bash
for i in {1..10}; do
echo "Test run $i"
nox -e unit_tests
if [ $? -ne 0 ]; then
echo "FLAKY TEST DETECTED on run $i"
exit 1
fi
done
```
2. **Testing with different orders:**
```bash
# Behave runs tests in file order, but verify scenarios are independent
nox -e unit_tests --random-order # If available in config
```
3. **Monitor for intermittent failures:**
- If a test passes sometimes and fails other times WITH THE SAME CODE
- If failures mention timing, "connection refused", "file not found" randomly
- If error messages contain random elements (UUIDs, timestamps, etc.)
### E2E Test Exception
**ONLY End-to-End (E2E) tests** may have some non-determinism due to real system integration. Unit and integration tests MUST be 100% deterministic.
If you're writing unit tests (which you are), there are **NO EXCEPTIONS** to the determinism requirement.
## Setup
You will be given:
+14
View File
@@ -49,6 +49,20 @@ If these are not provided in your context, invoke `ref-reader` IMMEDIATELY to ob
- **Milestone Assignment**: Every issue must belong to a milestone
- **Dependencies**: Properly link blocking/blocked by relationships
#### ⚠️ CRITICAL: Label Creation PROHIBITED ⚠️
**YOU MUST NEVER CREATE NEW LABELS.** All required labels already exist in the system.
**Label Usage Rules:**
- **ONLY use existing labels** - Check current labels before applying any
- **State/ labels**: State/Unverified, State/Verified, State/In Progress, State/In Review, State/Completed, State/Wont Do, State/Paused
- **Type/ labels**: Type/Epic, Type/Legendary, Type/Bug, Type/Feature, Type/Task, Type/Documentation, Type/Testing, Type/Refactor, Type/Automation
- **Priority/ labels**: Priority/Critical, Priority/High, Priority/Medium, Priority/Low, Priority/Backlog
- **MoSCoW/ labels**: MoSCoW/Must Have, MoSCoW/Should Have, MoSCoW/Could Have
- **If you need a label that doesn't exist**: Create an issue requesting it, DO NOT create it yourself
**Consequence**: Creating new labels causes confusion and duplicates. Use ONLY the labels listed above.
#### Creating Issues (CONTRIBUTING.md Section: Creating Issues)
- **Title Format**: Clear, imperative statements
- **Metadata Section**: Must include all required fields
@@ -44,6 +44,13 @@ permission:
4. **If CI is failing, fix it** - Never assume checks are passing
5. **The all_checks_passing() function MUST query actual CI status** - Never skip this
**🧪 CRITICAL TEST STABILITY RULES 🧪**
1. **ALL tests must be DETERMINISTIC** - No random behavior, timing dependencies, or external calls
2. **Watch for flaky test patterns** in CI failures - they indicate non-deterministic tests
3. **Monitor cross-PR test failures** - same tests failing in multiple PRs = master branch issue
4. **Report flaky tests immediately** via issue comments if detected
5. **Never ignore intermittent test failures** - they grow into CI-blocking problems
You are a dual-mode implementation worker that handles BOTH:
1. **PR Fixing** (pr-fix mode): Fix existing PRs with failing CI, review feedback, or merge conflicts
2. **Issue Implementation** (issue-impl mode): Implement new issues from scratch through PR merge
+15
View File
@@ -38,6 +38,21 @@ You will be given:
If you need project rules for issue formatting, invoke `ref-reader`.
## ⚠️ CRITICAL: Label Creation PROHIBITED ⚠️
**YOU MUST NEVER CREATE NEW LABELS.** All required labels already exist in the system.
**Label Usage Rules:**
- **ONLY use existing labels** when creating issues
- **State/ labels**: State/Unverified (default for new issues), State/Verified, State/In Progress, State/In Review, State/Completed, State/Wont Do, State/Paused
- **Type/ labels**: Type/Epic, Type/Legendary, Type/Bug, Type/Feature, Type/Task, Type/Documentation, Type/Testing, Type/Refactor, Type/Automation
- **Priority/ labels**: Priority/Critical, Priority/High, Priority/Medium, Priority/Low, Priority/Backlog
- **MoSCoW/ labels**: MoSCoW/Must Have, MoSCoW/Should Have, MoSCoW/Could Have
- **Use `forgejo_add_issue_labels`** ONLY with existing labels from the lists above
- **If you need a label that doesn't exist**: Create an issue requesting it, DO NOT create it yourself
**Consequence**: Creating duplicate labels causes system confusion and conflicts.
## Required Reading
All work must strictly adhere to **`CONTRIBUTING.md`**'s full "Creating
+143 -1
View File
@@ -197,11 +197,12 @@ If not provided in your prompt, invoke `ref-reader` to load specification conten
- No forbidden patterns (type: ignore, etc.)?
- File size limits respected?
**Test Quality**
**Test Quality and Stability**
- Do tests verify meaningful behavior (not just coverage padding)?
- Are edge cases and error paths tested?
- Are test names descriptive and scenarios well-structured?
- Is coverage adequate and meaningful?
- **CRITICAL**: Are tests deterministic and stable (see Flaky Test Detection below)?
**TDD Tag Compliance (for Bug Fix PRs)**
- If PR closes a bug issue #N, verify ALL `@tdd_issue_N` tests have `@tdd_expected_fail` removed
@@ -251,6 +252,147 @@ Based on the focus areas assigned for this review, perform deeper analysis:
(Add similar deep-dive sections for other focus areas)
### 4.5. CRITICAL: Flaky Test Detection
**⚠️ FLAKY TESTS BLOCK ALL CI - TOP PRIORITY ⚠️**
Flaky tests that reach master will block ALL future PRs. You MUST actively detect and flag potential flaky tests:
#### During Code Review - Examine Tests for Non-Deterministic Patterns
**❌ RED FLAGS in Unit Tests (Behave):**
```python
# BAD - time dependencies
time.sleep(0.1)
now = datetime.now()
# BAD - unseeded randomness
random_id = random.randint(1, 1000)
test_uuid = str(uuid.uuid4())
# BAD - external dependencies
requests.get("https://api.example.com")
# BAD - race conditions
threading.Thread(target=worker).start()
# Test continues immediately
# BAD - file system contamination
with open('shared_file.txt', 'w') as f:
f.write('test data')
```
**❌ RED FLAGS in Integration Tests (Robot):**
```robot
# BAD - timing assumptions
Start Service
Sleep 0.1s
Service Should Be Ready
# BAD - shared resources
Create File /tmp/shared_file.txt test data
# BAD - random test data
${port}= Generate Random Port
```
**✅ GOOD PATTERNS to verify are used:**
```python
# GOOD - fixed test data
test_user = User(id=12345, name="test_user", created_at=FIXED_DATE)
# GOOD - proper mocking
@mock.patch('requests.get')
def test_api_call(mock_get):
mock_get.return_value = MockResponse()
# GOOD - seeded randomness
random.seed(42)
test_value = random.randint(1, 100) # Now deterministic
# GOOD - test isolation
@before_scenario
def setup_clean_state(context):
context.temp_dir = tempfile.mkdtemp()
```
#### CI Pattern Analysis - Look for Flaky Test Symptoms
**When fetching CI logs, watch for these patterns:**
1. **Intermittent Failures**:
- Same test passing and failing across different runs
- Failures mentioning "timeout", "connection refused", "file not found"
- Error messages with random elements (timestamps, UUIDs)
2. **Timing-Related Issues**:
- Tests that fail with "expected X but got Y" where values suggest timing
- "Process not ready" or "Service unavailable" errors
- Different results based on system load
3. **Cross-Test Contamination**:
- Test failures that only occur when run with other specific tests
- Different results when tests run in different orders
- "File already exists" or "Port already in use" errors
#### Multi-PR Analysis - Check for Master Branch Issues
**CRITICAL**: Look for patterns across multiple PRs that suggest master has failing tests:
1. **Same Tests Failing Across Multiple PRs**:
- If the same test/scenario is failing in multiple open PRs
- Especially if the PRs don't touch related code
- This indicates a master branch test that got flaky after merging
2. **Recent Master Failures** (when possible to check):
- If you can check master CI status, look for any failures
- Master should NEVER have failing tests
- Any master failures require immediate action
#### Required Actions for Flaky Test Detection
**If you detect NON-DETERMINISTIC test patterns:**
1. **REQUEST CHANGES** immediately:
```
## ⚠️ CRITICAL: Flaky Test Patterns Detected
Found non-deterministic patterns in tests that will cause CI instability:
**Location**: `features/test_user_service.py:45-52`
**Issue**: Test uses `datetime.now()` creating time-dependent behavior
**Required**: Use fixed timestamp: `datetime(2024, 1, 1, 12, 0, 0)`
**Risk**: This will cause random CI failures blocking all future PRs
**Location**: `robot/integration_tests.robot:23`
**Issue**: Uses `Sleep 0.1s` instead of proper synchronization
**Required**: Replace with `Wait Until Keyword Succeeds` pattern
FLAKY TESTS BLOCK ALL CI - THESE MUST BE FIXED BEFORE MERGE
```
2. **If you suspect master branch failures:**
- Add note in review about cross-PR test failure patterns
- Recommend checking master CI status
- Flag for system monitoring
#### Stable Test Verification Requirements
**For any PR touching tests, verify the author:**
1. **Ran tests multiple times** to ensure stability:
```bash
for i in {1..10}; do nox -e unit_tests || exit 1; done
```
2. **Used deterministic patterns** for all test data
3. **Properly isolated** test environment (temp dirs, unique ports, etc.)
4. **Avoided timing dependencies** in favor of condition-based waits
**Remember**: Unit and Integration tests must be 100% deterministic. Only E2E tests may have some non-determinism.
### 5. Make a Decision
Based on your review, you will either APPROVE or REQUEST CHANGES.
+172
View File
@@ -81,6 +81,178 @@ Bug 123 - Component Should Handle Empty Input
You write Robot Framework integration tests for the CleverAgents project.
## CRITICAL: Integration Test Stability Requirements
**⚠️ FLAKY TESTS BLOCK ALL CI ⚠️**
Integration tests MUST be deterministic and stable. Unlike E2E tests, integration tests should have minimal non-determinism even when testing real components.
### Non-Deterministic Patterns You MUST AVOID
**❌ NEVER DO THESE:**
1. **Uncontrolled timing dependencies:**
```robot
# BAD - race conditions
Start Background Process
Sleep 0.1s # Hope it's ready!
Check Process Result
```
2. **Unseeded randomness:**
```robot
# BAD - unpredictable test data
${random_port}= Generate Random Port
Start Service On Port ${random_port}
```
3. **External service dependencies:**
```robot
# BAD - real external APIs (unless that's what you're testing)
Send HTTP Request https://api.github.com/users/test
Should Be Equal ${response.status} 200
```
4. **Shared test resources without isolation:**
```robot
# BAD - multiple tests using same file/port/database
Create File /tmp/shared_test_file.txt test data
# Another test might modify this!
```
5. **Current time dependencies:**
```robot
# BAD - dependent on system clock
${current_time}= Get Current Date
Process Data With Timestamp ${current_time}
```
### Required Stable Patterns ✅
**✅ ALWAYS DO THESE:**
1. **Proper synchronization:**
```robot
# GOOD - wait for actual conditions
Start Background Process
Wait Until Keyword Succeeds 10s 1s
... Process Should Be Ready
Check Process Result
```
2. **Isolated test environments:**
```robot
# GOOD - unique test workspace per test
${test_dir}= Create Temporary Directory
Set Test Variable ${TEST_WORKSPACE} ${test_dir}
Create File ${TEST_WORKSPACE}/test_file.txt test data
```
3. **Fixed test data:**
```robot
# GOOD - predictable, deterministic data
${test_user}= Create Dictionary
... id=12345
... name=test_user
... created_at=2024-01-01T12:00:00Z
```
4. **Controlled service configurations:**
```robot
# GOOD - use test-specific ports/configs
${test_port}= Set Variable ${BASE_TEST_PORT + ${TEST_NUMBER}}
Start Test Service port=${test_port} config=${test_config}
```
5. **Test isolation hooks:**
```robot
*** Keywords ***
Test Setup
[Documentation] Clean isolated environment for each test
Create Test Database
Start Clean Service Instance
Test Teardown
[Documentation] Clean up test resources
Stop Service Instance
Delete Test Database
```
### Integration Test Determinism Rules
1. **Use test-specific instances:**
- Each test should have its own database/service instance
- Use different ports/paths/namespaces per test
- Never share stateful resources between tests
2. **Proper service lifecycle:**
```robot
*** Test Cases ***
My Integration Test
[Setup] Setup Clean Test Environment
[Teardown] Cleanup Test Environment
# Test implementation here
```
3. **Wait for actual conditions, not time:**
```robot
# GOOD - condition-based waiting
Wait Until Keyword Succeeds 30s 1s
... Service Health Check Should Pass
# BAD - time-based assumptions
Sleep 5s # Hope service is ready
```
4. **Idempotent test operations:**
- Tests should work regardless of previous test state
- Clean up completely in teardown
- Don't assume clean starting state without setup
### Cross-Test Contamination Prevention
**CRITICAL**: Integration tests often share services. Prevent contamination:
1. **Database isolation:**
```robot
*** Keywords ***
Create Test Database
${db_name}= Set Variable test_db_${TEST_NAME}
Execute SQL CREATE DATABASE ${db_name}
Set Test Variable ${TEST_DATABASE} ${db_name}
```
2. **Service isolation:**
```robot
Start Isolated Service
${service_port}= Get Unique Test Port
${config_file}= Create Test Config port=${service_port}
Start Service config=${config_file}
```
3. **Filesystem isolation:**
```robot
Setup Test Workspace
${workspace}= Create Temporary Directory
Set Test Variable ${WORKSPACE} ${workspace}
```
### When Non-Determinism is Acceptable
**ONLY in these specific cases:**
1. **Testing actual timing behavior** (rare):
- When the timing IS the feature being tested
- Must include multiple attempts with statistical validation
2. **Testing real external integrations** (rare):
- When you're actually testing external API integration
- Must include proper error handling for external failures
- Should be clearly marked as potentially flaky
**For 99% of integration tests, full determinism is required.**
## Setup
You will be given:
+440
View File
@@ -137,6 +137,11 @@ LOOP FOREVER:
cycle += 1
findings = []
# ── Audit 0: CRITICAL - Master CI Health Monitoring ──────────
# ⚠️ HIGHEST PRIORITY: Master should NEVER have failing tests
# If ANY test fails on master, immediately skip it and create tickets
findings += audit_master_ci_health()
# ── Audit 1: Quality Gate Compliance ─────────────────────────
# This is the MOST CRITICAL audit. CONTRIBUTING.md requires ALL
# CI checks to pass before merge. Violations mean broken code
@@ -236,6 +241,192 @@ LOOP FOREVER:
## Audit Implementations
### Audit 0: CRITICAL - Master CI Health Monitoring
**Purpose:** Detect and immediately fix ANY test failures on master branch.
Master should NEVER have failing CI. Any failure blocks all future PRs.
```
function audit_master_ci_health():
findings = []
# ⚠️ CRITICAL: Check latest master commit CI status
master_commits = GET /repos/{owner}/{repo}/commits?sha=master&limit=3
for commit in master_commits[:1]: # Check only latest commit
commit_sha = commit.sha
statuses = GET /repos/{owner}/{repo}/statuses/{commit_sha}
# Look for CI status checks
ci_statuses = [s for s in statuses if s.context in [
"status-check", "ci", "CI", "tests",
"unit_tests", "integration_tests", "lint",
"typecheck", "security", "coverage"
]]
failing_checks = [s for s in ci_statuses if s.state == "failure"]
if failing_checks:
# IMMEDIATE ACTION REQUIRED
for check in failing_checks:
findings.append({
severity: "CRITICAL",
type: "master_ci_failure",
detail: f"MASTER CI FAILING: {check.context} failed on commit {commit_sha[:8]}. This blocks ALL future PRs!",
commit: commit_sha,
check: check.context,
action: "immediate_test_skip_and_tickets",
priority: "EMERGENCY"
})
# Investigate which specific tests are failing
findings += investigate_and_skip_failing_tests(commit_sha, failing_checks)
return findings
def investigate_and_skip_failing_tests(commit_sha, failing_checks):
findings = []
# For each failing CI check, try to get detailed logs
for check in failing_checks:
try:
# Use web authentication to get CI logs
forgejo_web_login()
# Get the workflow run details for this commit
actions_page = curl_with_cookies(
"https://git.cleverthis.com/cleveragents/cleveragents-core/actions"
)
# Parse for workflow run ID matching this commit
run_id = extract_workflow_run_id(actions_page, commit_sha)
if run_id:
# Get job logs
job_logs = get_workflow_job_logs(run_id, check.context)
# Parse logs to identify specific failing tests
failing_tests = parse_failing_tests_from_logs(job_logs, check.context)
if failing_tests:
# Create immediate skip actions for each failing test
for test in failing_tests:
findings.append({
severity: "CRITICAL",
type: "immediate_test_skip_required",
detail: f"Test '{test.name}' failing on master - MUST skip immediately",
test_name: test.name,
test_file: test.file,
check_type: check.context,
commit: commit_sha,
action: "skip_test_and_create_tickets",
priority: "EMERGENCY"
})
else:
# Generic CI failure - may need manual investigation
findings.append({
severity: "CRITICAL",
type: "master_ci_failure_needs_investigation",
detail: f"Master CI check '{check.context}' failing but specific tests not identified",
check: check.context,
commit: commit_sha,
action: "manual_investigation_required"
})
except Exception as e:
findings.append({
severity: "HIGH",
type: "ci_investigation_failed",
detail: f"Could not investigate master CI failure for {check.context}: {str(e)}",
check: check.context,
commit: commit_sha
})
return findings
def parse_failing_tests_from_logs(logs, check_type):
"""
Parse CI logs to identify specific failing tests based on test framework
"""
failing_tests = []
if not logs:
return failing_tests
log_text = logs.lower()
# Parse Behave (unit test) failures
if check_type in ["unit_tests", "tests"]:
# Look for Behave failure patterns
import re
behave_failures = re.findall(
r'FAILED.*?(features/[^\s]+\.feature).*?line (\d+)',
logs
)
for file, line in behave_failures:
failing_tests.append({
"name": f"Scenario at line {line}",
"file": file,
"type": "behave",
"framework": "unit"
})
# Parse Robot Framework (integration test) failures
elif check_type in ["integration_tests"]:
robot_failures = re.findall(
r'FAIL.*?(robot/[^\s]+\.robot).*?([^\n]+)',
logs
)
for file, test_name in robot_failures:
failing_tests.append({
"name": test_name.strip(),
"file": file,
"type": "robot",
"framework": "integration"
})
# Parse general test failures (pytest, etc.)
else:
# Generic failure pattern matching
test_failures = re.findall(
r'FAILED (test_[^\s]+|.*test.*\.py::[^\s]+)',
logs
)
for test in test_failures:
failing_tests.append({
"name": test,
"file": "unknown",
"type": "generic",
"framework": "unknown"
})
return failing_tests
def get_workflow_job_logs(run_id, job_name):
"""
Get logs for a specific workflow job using web authentication
"""
try:
# Construct the job logs URL
logs_url = f"https://git.cleverthis.com/cleveragents/cleveragents-core/actions/runs/{run_id}/jobs"
# Get the job list page
jobs_page = curl_with_cookies(logs_url)
# Find the specific job ID for the failing check
job_id = extract_job_id_for_check(jobs_page, job_name)
if job_id:
# Get the actual log content
log_url = f"https://git.cleverthis.com/cleveragents/cleveragents-core/actions/runs/{run_id}/jobs/{job_id}/logs"
return curl_with_cookies(log_url)
except Exception as e:
echo f"Failed to get job logs: {str(e)}"
return None
return None
```
### Audit 1: Quality Gate Compliance
**Purpose:** Ensure NO code reaches master without passing ALL CI checks.
@@ -1409,6 +1600,255 @@ function audit_system_health_monitoring():
```
function take_action(finding):
# ⚠️ CRITICAL: Immediate test skipping for master CI failures
if finding.type == "immediate_test_skip_required":
handle_immediate_test_skip(finding)
return
if finding.type == "master_ci_failure":
handle_master_ci_failure(finding)
return
def handle_immediate_test_skip(finding):
"""
EMERGENCY HANDLER: Skip failing test immediately and create tickets
This is the most critical action - master CI must be fixed ASAP
"""
test_name = finding.test_name
test_file = finding.test_file
framework = finding.get("framework", "unknown")
echo f"🚨 EMERGENCY: Skipping test '{test_name}' in {test_file} to unblock CI"
# Create TWO issues:
# 1. Skip task (high priority, immediate)
# 2. Fix task (tracks the actual bug)
# Issue 1: Skip the test (MUST HAVE/Critical)
skip_issue_body = f"""## EMERGENCY: Skip Flaky Test to Unblock CI
**Test**: `{test_name}`
**File**: `{test_file}`
**Framework**: {framework}
**Commit**: {finding.commit}
This test is failing on master branch, blocking ALL future PRs. It must be skipped immediately.
### Skip Instructions
{get_skip_instructions(framework, test_name, test_file)}
### Definition of Done
- [ ] Test is skipped using appropriate tag/marker
- [ ] PR created and merged to master
- [ ] CI is green on master
- [ ] All other PRs can proceed
**CRITICAL**: This issue should be completed within 1 hour.
---
**Automated by CleverAgents Bot**
Supervisor: System Watchdog | Emergency Response
"""
skip_issue = invoke_subagent(
"new-issue-creator",
f"Create CRITICAL skip issue for test {test_name}",
{
"issue_type": "emergency_skip",
"test_name": test_name,
"test_file": test_file,
"title": f"EMERGENCY: Skip failing test '{test_name}' to unblock master CI",
"body": skip_issue_body,
"labels": ["MoSCoW/Must Have", "Priority/Critical", "Type/Task", "State/Verified"],
"milestone": "current"
}
)
# Issue 2: Fix the test (Should Have/High)
fix_issue_body = f"""## Fix Failing Test
**Test**: `{test_name}`
**File**: `{test_file}`
**Framework**: {framework}
**Related Skip Issue**: #{skip_issue.number}
This test was skipped in issue #{skip_issue.number} due to failures on master. Once the underlying issue is identified and fixed, this test should be re-enabled.
### Investigation Steps
- [ ] Reproduce the test failure locally
- [ ] Identify root cause of flakiness/failure
- [ ] Fix the underlying issue
- [ ] Verify test passes consistently (10+ runs)
- [ ] Remove skip tag/marker
- [ ] Verify test runs in CI
### Possible Causes
{get_failure_analysis_hints(framework, test_name)}
### Definition of Done
- [ ] Root cause identified
- [ ] Fix implemented
- [ ] Test re-enabled (skip tag removed)
- [ ] Test passes consistently
---
**Automated by CleverAgents Bot**
Supervisor: System Watchdog | Test Recovery
"""
fix_issue = invoke_subagent(
"new-issue-creator",
f"Create fix issue for test {test_name}",
{
"issue_type": "test_fix",
"test_name": test_name,
"test_file": test_file,
"title": f"Fix and re-enable test '{test_name}'",
"body": fix_issue_body,
"labels": ["MoSCoW/Should Have", "Priority/High", "Type/Bug", "State/Verified"],
"milestone": "current"
}
)
echo f"✅ Created skip issue #{skip_issue.number} and fix issue #{fix_issue.number}"
def handle_master_ci_failure(finding):
"""
Handle general master CI failures that need immediate attention
"""
check = finding.check
commit = finding.commit
# Create high priority issue for master CI failure
issue_body = f"""## 🚨 CRITICAL: Master CI Failure
**Check**: {check}
**Commit**: {commit}
**Status**: FAILING
The master branch has failing CI which blocks all future PRs. This requires immediate investigation and resolution.
### Immediate Actions Required
1. **Investigate** the specific failure in {check}
2. **Identify** which tests or checks are failing
3. **Skip** any flaky/failing tests if needed
4. **Fix** the underlying issue
5. **Verify** master CI is green
### Investigation Steps
- [ ] Check CI logs for {check} on commit {commit[:8]}
- [ ] Identify specific failing tests/lints/checks
- [ ] Create skip issues for failing tests (if flaky)
- [ ] Create fix issues for underlying problems
- [ ] Verify resolution
**CRITICAL**: Master must be green within 2 hours.
---
**Automated by CleverAgents Bot**
Supervisor: System Watchdog | CI Emergency
"""
issue = invoke_subagent(
"new-issue-creator",
f"Create CRITICAL master CI failure issue",
{
"issue_type": "master_ci_failure",
"check": check,
"commit": commit,
"title": f"CRITICAL: Master CI failure in {check}",
"body": issue_body,
"labels": ["MoSCoW/Must Have", "Priority/Critical", "Type/Bug", "State/Verified"],
"milestone": "current"
}
)
echo f"✅ Created critical master CI issue #{issue.number}"
def get_skip_instructions(framework, test_name, test_file):
"""Generate framework-specific skip instructions"""
if framework == "behave" or "features/" in test_file:
return f"""**For Behave tests:**
1. Find the scenario containing `{test_name}` in `{test_file}`
2. Add `@skip` tag to the scenario:
```gherkin
@skip
Scenario: {test_name}
# existing test content
```
3. Run `nox -e unit_tests` to verify test is skipped
4. Create PR with title "skip: Disable flaky test {test_name}"
"""
elif framework == "robot" or "robot/" in test_file:
return f"""**For Robot Framework tests:**
1. Find the test case `{test_name}` in `{test_file}`
2. Add `[Tags] skip` to the test:
```robot
{test_name}
[Tags] skip
# existing test content
```
3. Run `nox -e integration_tests` to verify test is skipped
4. Create PR with title "skip: Disable flaky test {test_name}"
"""
else:
return f"""**Generic skip instructions:**
1. Locate test `{test_name}` in `{test_file}`
2. Add appropriate skip marker for the test framework
3. Verify test is skipped when running the test suite
4. Create PR with title "skip: Disable flaky test {test_name}"
"""
def get_failure_analysis_hints(framework, test_name):
"""Provide hints for investigating test failures"""
return """**Common flaky test causes:**
- **Timing issues**: Uses `time.sleep()` or `datetime.now()`
- **Random data**: Uses unseeded `random` or `uuid.uuid4()`
- **External dependencies**: Real API calls without mocking
- **Shared resources**: Tests interfere with each other
- **File system race conditions**: Multiple tests access same files
- **Environment dependent**: Different behavior on different systems
**Investigation approach:**
1. Run the test locally 10+ times: `for i in {1..10}; do nox -e <test_suite> || echo "FAIL $i"; done`
2. Check git history: `git log --oneline -10 <test_file>`
3. Look for non-deterministic patterns in the test code
4. Review any recent changes to related modules
"""
def invoke_subagent(agent_type, description, params):
"""Invoke a subagent via curl to the OpenCode server"""
import json
payload = {
"description": description,
"prompt": f"Execute {agent_type} with parameters: {json.dumps(params)}",
"subagent_type": agent_type
}
response = curl(
f"POST {SERVER}/session/{SESSION_ID}/task",
headers={"Content-Type": "application/json"},
data=json.dumps(payload)
)
# Parse response to extract issue number if created
# This is a simplified version - actual implementation would parse the response
import re
issue_match = re.search(r'issue[#\s]+(\d+)', response)
if issue_match:
return {"number": int(issue_match.group(1))}
return {"number": "unknown"}
```
if finding.severity == "CRITICAL":
# Dispatch one-off fix agent immediately via curl/prompt_async
if finding.type in ("no_branch_protection", "status_check_disabled",