diff --git a/.opencode/agents/behave-tester.md b/.opencode/agents/behave-tester.md index e927a85f5..9acf24c93 100644 --- a/.opencode/agents/behave-tester.md +++ b/.opencode/agents/behave-tester.md @@ -82,6 +82,175 @@ Scenario: Bug #123 - Component should handle empty input You write Behave (Cucumber/Gherkin) unit tests for the CleverAgents project following strict BDD standards. +## CRITICAL: Test Stability and Determinism Requirements + +**⚠️ FLAKY TESTS BLOCK ALL CI ⚠️** + +Your tests MUST be 100% deterministic and stable. A single flaky test that gets into master will block ALL future PRs. You are responsible for ensuring every test you write is rock-solid. + +### Non-Deterministic Patterns You MUST AVOID + +**❌ NEVER DO THESE:** + +1. **Time-based dependencies:** + ```python + # BAD - dependent on system time + now = datetime.now() + # BAD - timing-dependent assertions + time.sleep(0.1) + assert response_received # Race condition! + ``` + +2. **Random values without seeding:** + ```python + # BAD - unpredictable values + random_id = random.randint(1, 1000) + # BAD - UUID generation + test_id = str(uuid.uuid4()) + ``` + +3. **File system race conditions:** + ```python + # BAD - parallel test contamination + with open('test_file.txt', 'w') as f: + f.write('test data') + # Another test might modify this file! + ``` + +4. **Network dependencies:** + ```python + # BAD - external service calls + response = requests.get('https://api.example.com') + # BAD - real database connections + db.connect('real://database/url') + ``` + +5. **Order-dependent tests:** + ```python + # BAD - depends on test execution order + global_state = [] # Modified by multiple tests + ``` + +6. **Uncontrolled async operations:** + ```python + # BAD - uncontrolled threading + threading.Thread(target=background_task).start() + # Test continues without synchronization + ``` + +### Required Deterministic Patterns ✅ + +**✅ ALWAYS DO THESE:** + +1. **Use fixed test data:** + ```python + # GOOD - predictable test data + @given('a test user') + def step_test_user(context): + context.user = User( + id=12345, # Fixed ID + name="test_user", + created_at=datetime(2024, 1, 1, 12, 0, 0) # Fixed timestamp + ) + ``` + +2. **Mock external dependencies:** + ```python + # GOOD - mocked external calls + @pytest.fixture(autouse=True) + def mock_api_calls(monkeypatch): + monkeypatch.setattr('requests.get', lambda url: MockResponse()) + ``` + +3. **Use temporary isolated file systems:** + ```python + # GOOD - isolated temp directory + @given('a clean test environment') + def step_clean_environment(context): + context.temp_dir = tempfile.mkdtemp() + os.chdir(context.temp_dir) + ``` + +4. **Seed random generators:** + ```python + # GOOD - seeded randomness + random.seed(42) # Always produces same sequence + test_id = random.randint(1, 1000) # Now predictable + ``` + +5. **Proper test isolation:** + ```python + # GOOD - clean state per test + @before_scenario + def before_scenario(context): + context.test_data = {} # Fresh state + context.mocks = [] + ``` + +6. **Synchronous operations only (unless testing async):** + ```python + # GOOD - synchronous, predictable + result = process_data(input_data) + assert result == expected_output + ``` + +### Test Isolation Requirements + +1. **Each test scenario MUST:** + - Start with clean state (no leftover data from previous tests) + - Not depend on other tests having run first + - Clean up after itself + - Not modify global state that affects other tests + +2. **Use `@before_scenario` and `@after_scenario` hooks:** + ```python + @before_scenario + def setup_test_isolation(context): + # Reset all state + clear_global_caches() + reset_singletons() + context.temp_files = [] + + @after_scenario + def cleanup_test_isolation(context): + # Clean up resources + for temp_file in context.temp_files: + os.remove(temp_file) + ``` + +### Flaky Test Detection During Development + +**YOU MUST verify test stability by:** + +1. **Running tests multiple times:** + ```bash + for i in {1..10}; do + echo "Test run $i" + nox -e unit_tests + if [ $? -ne 0 ]; then + echo "FLAKY TEST DETECTED on run $i" + exit 1 + fi + done + ``` + +2. **Testing with different orders:** + ```bash + # Behave runs tests in file order, but verify scenarios are independent + nox -e unit_tests --random-order # If available in config + ``` + +3. **Monitor for intermittent failures:** + - If a test passes sometimes and fails other times WITH THE SAME CODE + - If failures mention timing, "connection refused", "file not found" randomly + - If error messages contain random elements (UUIDs, timestamps, etc.) + +### E2E Test Exception + +**ONLY End-to-End (E2E) tests** may have some non-determinism due to real system integration. Unit and integration tests MUST be 100% deterministic. + +If you're writing unit tests (which you are), there are **NO EXCEPTIONS** to the determinism requirement. + ## Setup You will be given: diff --git a/.opencode/agents/epic-planner.md b/.opencode/agents/epic-planner.md index 806277c03..2d773424e 100644 --- a/.opencode/agents/epic-planner.md +++ b/.opencode/agents/epic-planner.md @@ -49,6 +49,20 @@ If these are not provided in your context, invoke `ref-reader` IMMEDIATELY to ob - **Milestone Assignment**: Every issue must belong to a milestone - **Dependencies**: Properly link blocking/blocked by relationships +#### ⚠️ CRITICAL: Label Creation PROHIBITED ⚠️ + +**YOU MUST NEVER CREATE NEW LABELS.** All required labels already exist in the system. + +**Label Usage Rules:** +- **ONLY use existing labels** - Check current labels before applying any +- **State/ labels**: State/Unverified, State/Verified, State/In Progress, State/In Review, State/Completed, State/Wont Do, State/Paused +- **Type/ labels**: Type/Epic, Type/Legendary, Type/Bug, Type/Feature, Type/Task, Type/Documentation, Type/Testing, Type/Refactor, Type/Automation +- **Priority/ labels**: Priority/Critical, Priority/High, Priority/Medium, Priority/Low, Priority/Backlog +- **MoSCoW/ labels**: MoSCoW/Must Have, MoSCoW/Should Have, MoSCoW/Could Have +- **If you need a label that doesn't exist**: Create an issue requesting it, DO NOT create it yourself + +**Consequence**: Creating new labels causes confusion and duplicates. Use ONLY the labels listed above. + #### Creating Issues (CONTRIBUTING.md Section: Creating Issues) - **Title Format**: Clear, imperative statements - **Metadata Section**: Must include all required fields diff --git a/.opencode/agents/implementation-worker.md b/.opencode/agents/implementation-worker.md index 3955bf6eb..2c6b93932 100644 --- a/.opencode/agents/implementation-worker.md +++ b/.opencode/agents/implementation-worker.md @@ -44,6 +44,13 @@ permission: 4. **If CI is failing, fix it** - Never assume checks are passing 5. **The all_checks_passing() function MUST query actual CI status** - Never skip this +**🧪 CRITICAL TEST STABILITY RULES 🧪** +1. **ALL tests must be DETERMINISTIC** - No random behavior, timing dependencies, or external calls +2. **Watch for flaky test patterns** in CI failures - they indicate non-deterministic tests +3. **Monitor cross-PR test failures** - same tests failing in multiple PRs = master branch issue +4. **Report flaky tests immediately** via issue comments if detected +5. **Never ignore intermittent test failures** - they grow into CI-blocking problems + You are a dual-mode implementation worker that handles BOTH: 1. **PR Fixing** (pr-fix mode): Fix existing PRs with failing CI, review feedback, or merge conflicts 2. **Issue Implementation** (issue-impl mode): Implement new issues from scratch through PR merge diff --git a/.opencode/agents/new-issue-creator.md b/.opencode/agents/new-issue-creator.md index b896ab2f5..6a5f7b1ed 100644 --- a/.opencode/agents/new-issue-creator.md +++ b/.opencode/agents/new-issue-creator.md @@ -38,6 +38,21 @@ You will be given: If you need project rules for issue formatting, invoke `ref-reader`. +## ⚠️ CRITICAL: Label Creation PROHIBITED ⚠️ + +**YOU MUST NEVER CREATE NEW LABELS.** All required labels already exist in the system. + +**Label Usage Rules:** +- **ONLY use existing labels** when creating issues +- **State/ labels**: State/Unverified (default for new issues), State/Verified, State/In Progress, State/In Review, State/Completed, State/Wont Do, State/Paused +- **Type/ labels**: Type/Epic, Type/Legendary, Type/Bug, Type/Feature, Type/Task, Type/Documentation, Type/Testing, Type/Refactor, Type/Automation +- **Priority/ labels**: Priority/Critical, Priority/High, Priority/Medium, Priority/Low, Priority/Backlog +- **MoSCoW/ labels**: MoSCoW/Must Have, MoSCoW/Should Have, MoSCoW/Could Have +- **Use `forgejo_add_issue_labels`** ONLY with existing labels from the lists above +- **If you need a label that doesn't exist**: Create an issue requesting it, DO NOT create it yourself + +**Consequence**: Creating duplicate labels causes system confusion and conflicts. + ## Required Reading All work must strictly adhere to **`CONTRIBUTING.md`**'s full "Creating diff --git a/.opencode/agents/pr-self-reviewer.md b/.opencode/agents/pr-self-reviewer.md index 573254b41..c8ac2e013 100644 --- a/.opencode/agents/pr-self-reviewer.md +++ b/.opencode/agents/pr-self-reviewer.md @@ -197,11 +197,12 @@ If not provided in your prompt, invoke `ref-reader` to load specification conten - No forbidden patterns (type: ignore, etc.)? - File size limits respected? -**Test Quality** +**Test Quality and Stability** - Do tests verify meaningful behavior (not just coverage padding)? - Are edge cases and error paths tested? - Are test names descriptive and scenarios well-structured? - Is coverage adequate and meaningful? +- **CRITICAL**: Are tests deterministic and stable (see Flaky Test Detection below)? **TDD Tag Compliance (for Bug Fix PRs)** - If PR closes a bug issue #N, verify ALL `@tdd_issue_N` tests have `@tdd_expected_fail` removed @@ -251,6 +252,147 @@ Based on the focus areas assigned for this review, perform deeper analysis: (Add similar deep-dive sections for other focus areas) +### 4.5. CRITICAL: Flaky Test Detection + +**⚠️ FLAKY TESTS BLOCK ALL CI - TOP PRIORITY ⚠️** + +Flaky tests that reach master will block ALL future PRs. You MUST actively detect and flag potential flaky tests: + +#### During Code Review - Examine Tests for Non-Deterministic Patterns + +**❌ RED FLAGS in Unit Tests (Behave):** +```python +# BAD - time dependencies +time.sleep(0.1) +now = datetime.now() + +# BAD - unseeded randomness +random_id = random.randint(1, 1000) +test_uuid = str(uuid.uuid4()) + +# BAD - external dependencies +requests.get("https://api.example.com") + +# BAD - race conditions +threading.Thread(target=worker).start() +# Test continues immediately + +# BAD - file system contamination +with open('shared_file.txt', 'w') as f: + f.write('test data') +``` + +**❌ RED FLAGS in Integration Tests (Robot):** +```robot +# BAD - timing assumptions +Start Service +Sleep 0.1s +Service Should Be Ready + +# BAD - shared resources +Create File /tmp/shared_file.txt test data + +# BAD - random test data +${port}= Generate Random Port +``` + +**✅ GOOD PATTERNS to verify are used:** +```python +# GOOD - fixed test data +test_user = User(id=12345, name="test_user", created_at=FIXED_DATE) + +# GOOD - proper mocking +@mock.patch('requests.get') +def test_api_call(mock_get): + mock_get.return_value = MockResponse() + +# GOOD - seeded randomness +random.seed(42) +test_value = random.randint(1, 100) # Now deterministic + +# GOOD - test isolation +@before_scenario +def setup_clean_state(context): + context.temp_dir = tempfile.mkdtemp() +``` + +#### CI Pattern Analysis - Look for Flaky Test Symptoms + +**When fetching CI logs, watch for these patterns:** + +1. **Intermittent Failures**: + - Same test passing and failing across different runs + - Failures mentioning "timeout", "connection refused", "file not found" + - Error messages with random elements (timestamps, UUIDs) + +2. **Timing-Related Issues**: + - Tests that fail with "expected X but got Y" where values suggest timing + - "Process not ready" or "Service unavailable" errors + - Different results based on system load + +3. **Cross-Test Contamination**: + - Test failures that only occur when run with other specific tests + - Different results when tests run in different orders + - "File already exists" or "Port already in use" errors + +#### Multi-PR Analysis - Check for Master Branch Issues + +**CRITICAL**: Look for patterns across multiple PRs that suggest master has failing tests: + +1. **Same Tests Failing Across Multiple PRs**: + - If the same test/scenario is failing in multiple open PRs + - Especially if the PRs don't touch related code + - This indicates a master branch test that got flaky after merging + +2. **Recent Master Failures** (when possible to check): + - If you can check master CI status, look for any failures + - Master should NEVER have failing tests + - Any master failures require immediate action + +#### Required Actions for Flaky Test Detection + +**If you detect NON-DETERMINISTIC test patterns:** + +1. **REQUEST CHANGES** immediately: + ``` + ## ⚠️ CRITICAL: Flaky Test Patterns Detected + + Found non-deterministic patterns in tests that will cause CI instability: + + **Location**: `features/test_user_service.py:45-52` + **Issue**: Test uses `datetime.now()` creating time-dependent behavior + **Required**: Use fixed timestamp: `datetime(2024, 1, 1, 12, 0, 0)` + **Risk**: This will cause random CI failures blocking all future PRs + + **Location**: `robot/integration_tests.robot:23` + **Issue**: Uses `Sleep 0.1s` instead of proper synchronization + **Required**: Replace with `Wait Until Keyword Succeeds` pattern + + FLAKY TESTS BLOCK ALL CI - THESE MUST BE FIXED BEFORE MERGE + ``` + +2. **If you suspect master branch failures:** + - Add note in review about cross-PR test failure patterns + - Recommend checking master CI status + - Flag for system monitoring + +#### Stable Test Verification Requirements + +**For any PR touching tests, verify the author:** + +1. **Ran tests multiple times** to ensure stability: + ```bash + for i in {1..10}; do nox -e unit_tests || exit 1; done + ``` + +2. **Used deterministic patterns** for all test data + +3. **Properly isolated** test environment (temp dirs, unique ports, etc.) + +4. **Avoided timing dependencies** in favor of condition-based waits + +**Remember**: Unit and Integration tests must be 100% deterministic. Only E2E tests may have some non-determinism. + ### 5. Make a Decision Based on your review, you will either APPROVE or REQUEST CHANGES. diff --git a/.opencode/agents/robot-tester.md b/.opencode/agents/robot-tester.md index fb29eedc3..39ea28e71 100644 --- a/.opencode/agents/robot-tester.md +++ b/.opencode/agents/robot-tester.md @@ -81,6 +81,178 @@ Bug 123 - Component Should Handle Empty Input You write Robot Framework integration tests for the CleverAgents project. +## CRITICAL: Integration Test Stability Requirements + +**⚠️ FLAKY TESTS BLOCK ALL CI ⚠️** + +Integration tests MUST be deterministic and stable. Unlike E2E tests, integration tests should have minimal non-determinism even when testing real components. + +### Non-Deterministic Patterns You MUST AVOID + +**❌ NEVER DO THESE:** + +1. **Uncontrolled timing dependencies:** + ```robot + # BAD - race conditions + Start Background Process + Sleep 0.1s # Hope it's ready! + Check Process Result + ``` + +2. **Unseeded randomness:** + ```robot + # BAD - unpredictable test data + ${random_port}= Generate Random Port + Start Service On Port ${random_port} + ``` + +3. **External service dependencies:** + ```robot + # BAD - real external APIs (unless that's what you're testing) + Send HTTP Request https://api.github.com/users/test + Should Be Equal ${response.status} 200 + ``` + +4. **Shared test resources without isolation:** + ```robot + # BAD - multiple tests using same file/port/database + Create File /tmp/shared_test_file.txt test data + # Another test might modify this! + ``` + +5. **Current time dependencies:** + ```robot + # BAD - dependent on system clock + ${current_time}= Get Current Date + Process Data With Timestamp ${current_time} + ``` + +### Required Stable Patterns ✅ + +**✅ ALWAYS DO THESE:** + +1. **Proper synchronization:** + ```robot + # GOOD - wait for actual conditions + Start Background Process + Wait Until Keyword Succeeds 10s 1s + ... Process Should Be Ready + Check Process Result + ``` + +2. **Isolated test environments:** + ```robot + # GOOD - unique test workspace per test + ${test_dir}= Create Temporary Directory + Set Test Variable ${TEST_WORKSPACE} ${test_dir} + Create File ${TEST_WORKSPACE}/test_file.txt test data + ``` + +3. **Fixed test data:** + ```robot + # GOOD - predictable, deterministic data + ${test_user}= Create Dictionary + ... id=12345 + ... name=test_user + ... created_at=2024-01-01T12:00:00Z + ``` + +4. **Controlled service configurations:** + ```robot + # GOOD - use test-specific ports/configs + ${test_port}= Set Variable ${BASE_TEST_PORT + ${TEST_NUMBER}} + Start Test Service port=${test_port} config=${test_config} + ``` + +5. **Test isolation hooks:** + ```robot + *** Keywords *** + Test Setup + [Documentation] Clean isolated environment for each test + Create Test Database + Start Clean Service Instance + + Test Teardown + [Documentation] Clean up test resources + Stop Service Instance + Delete Test Database + ``` + +### Integration Test Determinism Rules + +1. **Use test-specific instances:** + - Each test should have its own database/service instance + - Use different ports/paths/namespaces per test + - Never share stateful resources between tests + +2. **Proper service lifecycle:** + ```robot + *** Test Cases *** + My Integration Test + [Setup] Setup Clean Test Environment + [Teardown] Cleanup Test Environment + + # Test implementation here + ``` + +3. **Wait for actual conditions, not time:** + ```robot + # GOOD - condition-based waiting + Wait Until Keyword Succeeds 30s 1s + ... Service Health Check Should Pass + + # BAD - time-based assumptions + Sleep 5s # Hope service is ready + ``` + +4. **Idempotent test operations:** + - Tests should work regardless of previous test state + - Clean up completely in teardown + - Don't assume clean starting state without setup + +### Cross-Test Contamination Prevention + +**CRITICAL**: Integration tests often share services. Prevent contamination: + +1. **Database isolation:** + ```robot + *** Keywords *** + Create Test Database + ${db_name}= Set Variable test_db_${TEST_NAME} + Execute SQL CREATE DATABASE ${db_name} + Set Test Variable ${TEST_DATABASE} ${db_name} + ``` + +2. **Service isolation:** + ```robot + Start Isolated Service + ${service_port}= Get Unique Test Port + ${config_file}= Create Test Config port=${service_port} + Start Service config=${config_file} + ``` + +3. **Filesystem isolation:** + ```robot + Setup Test Workspace + ${workspace}= Create Temporary Directory + Set Test Variable ${WORKSPACE} ${workspace} + ``` + +### When Non-Determinism is Acceptable + +**ONLY in these specific cases:** + +1. **Testing actual timing behavior** (rare): + - When the timing IS the feature being tested + - Must include multiple attempts with statistical validation + +2. **Testing real external integrations** (rare): + - When you're actually testing external API integration + - Must include proper error handling for external failures + - Should be clearly marked as potentially flaky + +**For 99% of integration tests, full determinism is required.** + ## Setup You will be given: diff --git a/.opencode/agents/system-watchdog.md b/.opencode/agents/system-watchdog.md index 628e067a9..f908d422f 100644 --- a/.opencode/agents/system-watchdog.md +++ b/.opencode/agents/system-watchdog.md @@ -137,6 +137,11 @@ LOOP FOREVER: cycle += 1 findings = [] + # ── Audit 0: CRITICAL - Master CI Health Monitoring ────────── + # ⚠️ HIGHEST PRIORITY: Master should NEVER have failing tests + # If ANY test fails on master, immediately skip it and create tickets + findings += audit_master_ci_health() + # ── Audit 1: Quality Gate Compliance ───────────────────────── # This is the MOST CRITICAL audit. CONTRIBUTING.md requires ALL # CI checks to pass before merge. Violations mean broken code @@ -236,6 +241,192 @@ LOOP FOREVER: ## Audit Implementations +### Audit 0: CRITICAL - Master CI Health Monitoring + +**Purpose:** Detect and immediately fix ANY test failures on master branch. +Master should NEVER have failing CI. Any failure blocks all future PRs. + +``` +function audit_master_ci_health(): + findings = [] + + # ⚠️ CRITICAL: Check latest master commit CI status + master_commits = GET /repos/{owner}/{repo}/commits?sha=master&limit=3 + + for commit in master_commits[:1]: # Check only latest commit + commit_sha = commit.sha + statuses = GET /repos/{owner}/{repo}/statuses/{commit_sha} + + # Look for CI status checks + ci_statuses = [s for s in statuses if s.context in [ + "status-check", "ci", "CI", "tests", + "unit_tests", "integration_tests", "lint", + "typecheck", "security", "coverage" + ]] + + failing_checks = [s for s in ci_statuses if s.state == "failure"] + + if failing_checks: + # IMMEDIATE ACTION REQUIRED + for check in failing_checks: + findings.append({ + severity: "CRITICAL", + type: "master_ci_failure", + detail: f"MASTER CI FAILING: {check.context} failed on commit {commit_sha[:8]}. This blocks ALL future PRs!", + commit: commit_sha, + check: check.context, + action: "immediate_test_skip_and_tickets", + priority: "EMERGENCY" + }) + + # Investigate which specific tests are failing + findings += investigate_and_skip_failing_tests(commit_sha, failing_checks) + + return findings + +def investigate_and_skip_failing_tests(commit_sha, failing_checks): + findings = [] + + # For each failing CI check, try to get detailed logs + for check in failing_checks: + try: + # Use web authentication to get CI logs + forgejo_web_login() + + # Get the workflow run details for this commit + actions_page = curl_with_cookies( + "https://git.cleverthis.com/cleveragents/cleveragents-core/actions" + ) + + # Parse for workflow run ID matching this commit + run_id = extract_workflow_run_id(actions_page, commit_sha) + + if run_id: + # Get job logs + job_logs = get_workflow_job_logs(run_id, check.context) + + # Parse logs to identify specific failing tests + failing_tests = parse_failing_tests_from_logs(job_logs, check.context) + + if failing_tests: + # Create immediate skip actions for each failing test + for test in failing_tests: + findings.append({ + severity: "CRITICAL", + type: "immediate_test_skip_required", + detail: f"Test '{test.name}' failing on master - MUST skip immediately", + test_name: test.name, + test_file: test.file, + check_type: check.context, + commit: commit_sha, + action: "skip_test_and_create_tickets", + priority: "EMERGENCY" + }) + else: + # Generic CI failure - may need manual investigation + findings.append({ + severity: "CRITICAL", + type: "master_ci_failure_needs_investigation", + detail: f"Master CI check '{check.context}' failing but specific tests not identified", + check: check.context, + commit: commit_sha, + action: "manual_investigation_required" + }) + except Exception as e: + findings.append({ + severity: "HIGH", + type: "ci_investigation_failed", + detail: f"Could not investigate master CI failure for {check.context}: {str(e)}", + check: check.context, + commit: commit_sha + }) + + return findings + +def parse_failing_tests_from_logs(logs, check_type): + """ + Parse CI logs to identify specific failing tests based on test framework + """ + failing_tests = [] + + if not logs: + return failing_tests + + log_text = logs.lower() + + # Parse Behave (unit test) failures + if check_type in ["unit_tests", "tests"]: + # Look for Behave failure patterns + import re + behave_failures = re.findall( + r'FAILED.*?(features/[^\s]+\.feature).*?line (\d+)', + logs + ) + for file, line in behave_failures: + failing_tests.append({ + "name": f"Scenario at line {line}", + "file": file, + "type": "behave", + "framework": "unit" + }) + + # Parse Robot Framework (integration test) failures + elif check_type in ["integration_tests"]: + robot_failures = re.findall( + r'FAIL.*?(robot/[^\s]+\.robot).*?([^\n]+)', + logs + ) + for file, test_name in robot_failures: + failing_tests.append({ + "name": test_name.strip(), + "file": file, + "type": "robot", + "framework": "integration" + }) + + # Parse general test failures (pytest, etc.) + else: + # Generic failure pattern matching + test_failures = re.findall( + r'FAILED (test_[^\s]+|.*test.*\.py::[^\s]+)', + logs + ) + for test in test_failures: + failing_tests.append({ + "name": test, + "file": "unknown", + "type": "generic", + "framework": "unknown" + }) + + return failing_tests + +def get_workflow_job_logs(run_id, job_name): + """ + Get logs for a specific workflow job using web authentication + """ + try: + # Construct the job logs URL + logs_url = f"https://git.cleverthis.com/cleveragents/cleveragents-core/actions/runs/{run_id}/jobs" + + # Get the job list page + jobs_page = curl_with_cookies(logs_url) + + # Find the specific job ID for the failing check + job_id = extract_job_id_for_check(jobs_page, job_name) + + if job_id: + # Get the actual log content + log_url = f"https://git.cleverthis.com/cleveragents/cleveragents-core/actions/runs/{run_id}/jobs/{job_id}/logs" + return curl_with_cookies(log_url) + + except Exception as e: + echo f"Failed to get job logs: {str(e)}" + return None + + return None +``` + ### Audit 1: Quality Gate Compliance **Purpose:** Ensure NO code reaches master without passing ALL CI checks. @@ -1409,6 +1600,255 @@ function audit_system_health_monitoring(): ``` function take_action(finding): + + # ⚠️ CRITICAL: Immediate test skipping for master CI failures + if finding.type == "immediate_test_skip_required": + handle_immediate_test_skip(finding) + return + + if finding.type == "master_ci_failure": + handle_master_ci_failure(finding) + return + +def handle_immediate_test_skip(finding): + """ + EMERGENCY HANDLER: Skip failing test immediately and create tickets + This is the most critical action - master CI must be fixed ASAP + """ + test_name = finding.test_name + test_file = finding.test_file + framework = finding.get("framework", "unknown") + + echo f"🚨 EMERGENCY: Skipping test '{test_name}' in {test_file} to unblock CI" + + # Create TWO issues: + # 1. Skip task (high priority, immediate) + # 2. Fix task (tracks the actual bug) + + # Issue 1: Skip the test (MUST HAVE/Critical) + skip_issue_body = f"""## EMERGENCY: Skip Flaky Test to Unblock CI + +**Test**: `{test_name}` +**File**: `{test_file}` +**Framework**: {framework} +**Commit**: {finding.commit} + +This test is failing on master branch, blocking ALL future PRs. It must be skipped immediately. + +### Skip Instructions + +{get_skip_instructions(framework, test_name, test_file)} + +### Definition of Done + +- [ ] Test is skipped using appropriate tag/marker +- [ ] PR created and merged to master +- [ ] CI is green on master +- [ ] All other PRs can proceed + +**CRITICAL**: This issue should be completed within 1 hour. + +--- +**Automated by CleverAgents Bot** +Supervisor: System Watchdog | Emergency Response +""" + + skip_issue = invoke_subagent( + "new-issue-creator", + f"Create CRITICAL skip issue for test {test_name}", + { + "issue_type": "emergency_skip", + "test_name": test_name, + "test_file": test_file, + "title": f"EMERGENCY: Skip failing test '{test_name}' to unblock master CI", + "body": skip_issue_body, + "labels": ["MoSCoW/Must Have", "Priority/Critical", "Type/Task", "State/Verified"], + "milestone": "current" + } + ) + + # Issue 2: Fix the test (Should Have/High) + fix_issue_body = f"""## Fix Failing Test + +**Test**: `{test_name}` +**File**: `{test_file}` +**Framework**: {framework} +**Related Skip Issue**: #{skip_issue.number} + +This test was skipped in issue #{skip_issue.number} due to failures on master. Once the underlying issue is identified and fixed, this test should be re-enabled. + +### Investigation Steps + +- [ ] Reproduce the test failure locally +- [ ] Identify root cause of flakiness/failure +- [ ] Fix the underlying issue +- [ ] Verify test passes consistently (10+ runs) +- [ ] Remove skip tag/marker +- [ ] Verify test runs in CI + +### Possible Causes + +{get_failure_analysis_hints(framework, test_name)} + +### Definition of Done + +- [ ] Root cause identified +- [ ] Fix implemented +- [ ] Test re-enabled (skip tag removed) +- [ ] Test passes consistently + +--- +**Automated by CleverAgents Bot** +Supervisor: System Watchdog | Test Recovery +""" + + fix_issue = invoke_subagent( + "new-issue-creator", + f"Create fix issue for test {test_name}", + { + "issue_type": "test_fix", + "test_name": test_name, + "test_file": test_file, + "title": f"Fix and re-enable test '{test_name}'", + "body": fix_issue_body, + "labels": ["MoSCoW/Should Have", "Priority/High", "Type/Bug", "State/Verified"], + "milestone": "current" + } + ) + + echo f"✅ Created skip issue #{skip_issue.number} and fix issue #{fix_issue.number}" + +def handle_master_ci_failure(finding): + """ + Handle general master CI failures that need immediate attention + """ + check = finding.check + commit = finding.commit + + # Create high priority issue for master CI failure + issue_body = f"""## 🚨 CRITICAL: Master CI Failure + +**Check**: {check} +**Commit**: {commit} +**Status**: FAILING + +The master branch has failing CI which blocks all future PRs. This requires immediate investigation and resolution. + +### Immediate Actions Required + +1. **Investigate** the specific failure in {check} +2. **Identify** which tests or checks are failing +3. **Skip** any flaky/failing tests if needed +4. **Fix** the underlying issue +5. **Verify** master CI is green + +### Investigation Steps + +- [ ] Check CI logs for {check} on commit {commit[:8]} +- [ ] Identify specific failing tests/lints/checks +- [ ] Create skip issues for failing tests (if flaky) +- [ ] Create fix issues for underlying problems +- [ ] Verify resolution + +**CRITICAL**: Master must be green within 2 hours. + +--- +**Automated by CleverAgents Bot** +Supervisor: System Watchdog | CI Emergency +""" + + issue = invoke_subagent( + "new-issue-creator", + f"Create CRITICAL master CI failure issue", + { + "issue_type": "master_ci_failure", + "check": check, + "commit": commit, + "title": f"CRITICAL: Master CI failure in {check}", + "body": issue_body, + "labels": ["MoSCoW/Must Have", "Priority/Critical", "Type/Bug", "State/Verified"], + "milestone": "current" + } + ) + + echo f"✅ Created critical master CI issue #{issue.number}" + +def get_skip_instructions(framework, test_name, test_file): + """Generate framework-specific skip instructions""" + if framework == "behave" or "features/" in test_file: + return f"""**For Behave tests:** +1. Find the scenario containing `{test_name}` in `{test_file}` +2. Add `@skip` tag to the scenario: + ```gherkin + @skip + Scenario: {test_name} + # existing test content + ``` +3. Run `nox -e unit_tests` to verify test is skipped +4. Create PR with title "skip: Disable flaky test {test_name}" +""" + elif framework == "robot" or "robot/" in test_file: + return f"""**For Robot Framework tests:** +1. Find the test case `{test_name}` in `{test_file}` +2. Add `[Tags] skip` to the test: + ```robot + {test_name} + [Tags] skip + # existing test content + ``` +3. Run `nox -e integration_tests` to verify test is skipped +4. Create PR with title "skip: Disable flaky test {test_name}" +""" + else: + return f"""**Generic skip instructions:** +1. Locate test `{test_name}` in `{test_file}` +2. Add appropriate skip marker for the test framework +3. Verify test is skipped when running the test suite +4. Create PR with title "skip: Disable flaky test {test_name}" +""" + +def get_failure_analysis_hints(framework, test_name): + """Provide hints for investigating test failures""" + return """**Common flaky test causes:** +- **Timing issues**: Uses `time.sleep()` or `datetime.now()` +- **Random data**: Uses unseeded `random` or `uuid.uuid4()` +- **External dependencies**: Real API calls without mocking +- **Shared resources**: Tests interfere with each other +- **File system race conditions**: Multiple tests access same files +- **Environment dependent**: Different behavior on different systems + +**Investigation approach:** +1. Run the test locally 10+ times: `for i in {1..10}; do nox -e || echo "FAIL $i"; done` +2. Check git history: `git log --oneline -10 ` +3. Look for non-deterministic patterns in the test code +4. Review any recent changes to related modules +""" + +def invoke_subagent(agent_type, description, params): + """Invoke a subagent via curl to the OpenCode server""" + import json + + payload = { + "description": description, + "prompt": f"Execute {agent_type} with parameters: {json.dumps(params)}", + "subagent_type": agent_type + } + + response = curl( + f"POST {SERVER}/session/{SESSION_ID}/task", + headers={"Content-Type": "application/json"}, + data=json.dumps(payload) + ) + + # Parse response to extract issue number if created + # This is a simplified version - actual implementation would parse the response + import re + issue_match = re.search(r'issue[#\s]+(\d+)', response) + if issue_match: + return {"number": int(issue_match.group(1))} + + return {"number": "unknown"} +``` if finding.severity == "CRITICAL": # Dispatch one-off fix agent immediately via curl/prompt_async if finding.type in ("no_branch_protection", "status_check_disabled",