cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity

docs(ci): add CI incident runbook and update quality gate documentation #2809

Closed
freemo wants to merge 2 commits from docs/ci-incident-runbook-2597 into master
pull from: docs/ci-incident-runbook-2597
merge into: cleveragents:master
Conversation 12 Commits 2 Files Changed 5
+479 -38
5 changed files with 479 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+35
View File
@@ -88,6 +88,41 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
### Fixed
- **Docs — CI incident runbook and quality gate documentation**: Added
`docs/development/ci-incident-runbook.md` documenting how to diagnose,
triage, and recover from master branch CI quality gate failures. Updated
`docs/development/ci-cd.md` and `docs/development/quality-automation.md` to
accurately reflect the full 11-job `status-check` consolidation gate
(including the previously undocumented `e2e_tests` and `helm` jobs), the
job dependency graph, and the prohibition on direct pushes to `master`. (#2597)
- **CLI — `agents plan list --namespace/-n` filter option**: `agents plan list`
now accepts `--namespace`/`-n` to filter plans by namespace, mirroring the
existing `--namespace` option on `agents action list`. The service layer
already supported namespace filtering; only the CLI layer was missing the
option. The TUI Filters panel displays the active namespace filter when
provided. (#2165)
- **MCP — Correct error extraction from MCP 1.4.0 protocol responses**:
`MCPToolAdapter.invoke()` now extracts error messages from
`content[0].text` per the MCP 1.4.0 protocol, replacing the previous
`result.get('error', 'unknown error')` lookup that silently swallowed all
real error messages from MCP 1.4.0-compliant servers. (#2158)
- **CI — Nox output captured as Forgejo CI artifacts**: All 8 nox-running CI
jobs now capture stdout+stderr to `build/nox-<job>-output.log` via `tee`
and upload the log as a named artifact (`ci-logs-<job>`, 30-day retention).
Agent definitions updated with a CI Log Artifacts section explaining which
artifact corresponds to each domain. (#2750)
- **Agents — Correct IssueMeta schema for Forgejo dependency API**: All agent
definitions that call the Forgejo blocks/dependencies REST API now use the
correct `IssueMeta` schema (`{"owner": "...", "repo": "...", "index": N}`)
instead of the undocumented `{"dependency_id": N}` format that returned
404 errors. Affects `ca-new-issue-creator`, `ca-pr-api-creator`,
`ca-state-reconciler`, `ca-project-owner`, `ca-backlog-groomer`, and
`ca-epic-planner`. (#2750)
- Fixed session leak in all `AutomationProfileRepository` public methods
(`get_by_name()`, `list_all()`, `upsert()`, and `delete()`): added
`finally: if self._auto_commit: session.close()` blocks matching the
docs/development/ci-cd.md
+51 -27
View File
@@ -15,24 +15,29 @@ pull requests with the following protections enforced.
#### Required Status Checks
Every pull request targeting `master` must pass **all** of the following CI jobs
before merging is allowed:
before merging is allowed. These are enforced by the `status-check` consolidation
gate — if any single job fails, the entire gate fails and the PR cannot merge.
| CI Job | What It Checks | Failure Means |
|--------|---------------|---------------|
| `lint` | Ruff format and lint (nox) | Code style violations or lint errors |
| `typecheck` | Pyright strict type checking (nox) | Type errors in `src/` |
| `security` | Security scan + dead code (nox) | Security vulnerabilities or dead code |
| `quality` | Radon complexity (nox) | Extremely complex methods (31+ cyclomatic complexity) |
| `unit_tests` | BDD unit tests via `nox -s unit_tests` | Failing unit test scenarios |
| `integration_tests` | Robot integration tests via `nox -s integration_tests` | Failing integration tests |
| `coverage` | Test coverage measurement (nox) | Coverage dropped below 97% |
| `build` | Wheel build | Build failure |
| # | CI Job | Nox Session(s) | What It Checks | Failure Means |
|---|--------|----------------|----------------|---------------|
| 1 | `lint` | `nox -s lint` + `nox -s format -- --check` | Ruff lint rules and code formatting | Code style violations or lint errors |
| 2 | `typecheck` | `nox -s typecheck` | Pyright strict type checking | Type errors in `src/` |
| 3 | `security` | `nox -s security_scan` + `nox -s dead_code` | Bandit HIGH gate, Semgrep custom rules, Vulture dead-code | Security vulnerabilities or dead code |
| 4 | `quality` | `nox -s complexity` | Radon cyclomatic complexity | Extremely complex methods (31+ cyclomatic complexity) |
| 5 | `unit_tests` | `nox -s unit_tests` | All Behave BDD scenarios under `features/` | Failing unit test scenarios |
| 6 | `integration_tests` | `nox -s integration_tests` | Robot Framework tests under `robot/` (excl. slow/E2E) | Failing integration tests |
| 7 | `e2e_tests` | `nox -s e2e_tests` | End-to-end Robot tests under `robot/e2e/` with real LLM keys | Failing end-to-end tests |
| 8 | `coverage` | `nox -s coverage_report` | Slipcover test coverage ≥ 97% | Coverage dropped below 97% |
| 9 | `build` | `nox -s build` | Python wheel build | Build failure |
| 10 | `docker` | Docker CLI | Docker image build + smoke test (`--version`) | Image build or startup failure |
| 11 | `helm` | Helm CLI + kubeconform | Helm chart lint, template render, and Kubernetes manifest validation | Chart or manifest invalid |
**Deployment-gating jobs** (run after core checks pass):
**Job dependencies** (some jobs only run after others pass):
| CI Job | Depends On | What It Checks |
|--------|-----------|----------------|
| `docker` | `lint`, `typecheck`, `unit_tests`, `security` | Docker image builds and runs |
| CI Job | Depends On |
|--------|-----------|
| `coverage` | `lint`, `typecheck`, `security`, `quality` |
| `docker` | `lint`, `typecheck`, `security`, `quality`, `unit_tests` |
#### Required Reviews
@@ -44,7 +49,9 @@ before merging is allowed:
#### Additional Protections
- **No direct pushes** to `master`. All changes must come via pull request.
The `no-commit-to-branch` pre-commit hook enforces this locally.
The `no-commit-to-branch` pre-commit hook enforces this locally. Direct
pushes bypass CI entirely and are the primary cause of master CI breakage
(see [CI Incident Runbook](ci-incident-runbook.md)).
- **No force pushes** to `master`. History must be preserved.
- **No deletions** of the `master` branch.
- **Require branches to be up-to-date** before merging (prevents merge skew).
@@ -155,15 +162,24 @@ PR template at `.forgejo/pull_request_template.md` in the repository root):
```
lint ──────────────────┐
typecheck ─────────────┤
├── coverage (needs lint + typecheck)
├── coverage (needs lint + typecheck + security + quality)
security ──────────────┤
├── docker (needs lint + typecheck + unit_tests + security)
unit_tests ────────────┘
quality ───────────────┘
└── docker (needs lint + typecheck + security + quality + unit_tests)
unit_tests ─────────────── (independent; also feeds docker)
integration_tests ──────── (independent)
quality ────────────────── (independent)
e2e_tests ──────────────── (independent; requires LLM API keys)
build ──────────────────── (independent)
helm ───────────────────── (independent)
┌── status-check (needs ALL 11 jobs above)
```
The `status-check` consolidation job is the single gate that branch protection
checks. It fails if **any** of the 11 dependent jobs fail, skipped, or are
cancelled. A broken `master` blocks all open PRs. See the
[CI Incident Runbook](ci-incident-runbook.md) for diagnosis and recovery
procedures.
### Nox-Based CI
All CI jobs now run their checks through nox sessions rather than invoking
@@ -173,19 +189,27 @@ appropriate nox session.
### Quality Gates Summary
All gates must pass for a PR to be mergeable:
All gates must pass for a PR to be mergeable. **No gate may be suppressed,
bypassed, or weakened** — fixes must address the actual code, not the
enforcement configuration. See [CI Incident Runbook](ci-incident-runbook.md)
for the complete list of prohibited suppression techniques.
| Gate | Threshold | Enforced By |
|------|-----------|-------------|
| Formatting | Zero violations | `lint` job (ruff format) |
| Linting | Zero violations | `lint` job (ruff check) |
| Formatting | Zero violations | `lint` job (`nox -s format -- --check`) |
| Linting | Zero violations | `lint` job (`nox -s lint`) |
| Type Safety | Zero errors | `typecheck` job (pyright strict) |
| Security | Zero HIGH severity | `security` job (bandit) |
| Dead Code | Zero findings | `security` job (vulture) |
| Complexity | No grade-F methods | `quality` job (radon) |
| Unit Tests | All pass (3.11-3.13) | `behave` job |
| Coverage | >= 97% | `coverage` job (nox) |
| Dead Code | Zero findings ≥80% confidence | `security` job (vulture) |
| Custom Rules | Zero violations | `security` job (semgrep) |
| Complexity | No grade-F methods (31+) | `quality` job (radon) |
| Unit Tests | All Behave scenarios pass | `unit_tests` job |
| Integration Tests | All Robot suites pass | `integration_tests` job |
| E2E Tests | All E2E Robot suites pass | `e2e_tests` job |
| Coverage | ≥ 97% | `coverage` job (`nox -s coverage_report`) |
| Build | Wheel builds | `build` job |
| Docker | Images build and smoke-test | `docker` job |
| Helm | Chart lints, renders, and validates | `helm` job |
### Nightly Quality Monitoring
docs/development/ci-incident-runbook.md
+374
View File
@@ -0,0 +1,374 @@
# CI Incident Runbook: Master Branch Quality Gate Failures
This runbook documents how to diagnose, triage, and resolve CI quality gate
failures on the `master` branch. It was written in response to issue #2597,
which identified a critical pattern of direct pushes to `master` bypassing
the PR process and introducing regressions.
## Why a Broken Master Is a Critical Incident
When `master` CI is broken, **all development work stops**:
1. **All open PRs are blocked.** The `status-check` consolidation gate requires
all 11 CI jobs to succeed. Any failure prevents any PR from merging.
2. **New branches inherit failures.** Any branch created from a broken `master`
starts with broken code.
3. **Branch protection enforces up-to-date branches.** PRs must be rebased on
`master` before they can merge — a broken `master` guarantees they will also
fail CI.
4. **The TDD workflow is broken.** TDD issue-capture tests require merging to
`master` before bug fixes can begin.
5. **No releases can be cut.** Release tags are pushed from `master`; a broken
`master` means no releases.
**Treat a broken `master` as the single highest-priority incident. All other
work is blocked until `master` is green.**
---
## The 11-Job Status-Check Gate
The `status-check` consolidation job (defined in `.forgejo/workflows/ci.yml`)
requires **all** of the following jobs to report `success` before any PR can
merge:
| # | CI Job | Nox Session(s) | What It Checks |
|---|--------|----------------|----------------|
| 1 | `lint` | `nox -s lint` + `nox -s format -- --check` | Ruff lint rules and code formatting |
| 2 | `typecheck` | `nox -s typecheck` | Pyright strict type checking |
| 3 | `security` | `nox -s security_scan` + `nox -s dead_code` | Bandit HIGH gate, Semgrep custom rules, Vulture dead-code (≥80% confidence) |
| 4 | `quality` | `nox -s complexity` | Radon cyclomatic complexity (no grade-F methods) |
| 5 | `unit_tests` | `nox -s unit_tests` | All Behave BDD scenarios under `features/` |
| 6 | `integration_tests` | `nox -s integration_tests` | All Robot Framework tests under `robot/` (excluding slow, discovery, code_blocks, wip, E2E, tdd_fixture) |
| 7 | `e2e_tests` | `nox -s e2e_tests` | End-to-end Robot tests with real LLM API keys under `robot/e2e/` |
| 8 | `coverage` | `nox -s coverage_report` | Slipcover test coverage ≥ **97%** fail-under threshold |
| 9 | `build` | `nox -s build` | Python wheel build |
| 10 | `docker` | Docker CLI | Docker image build (`Dockerfile` + `Dockerfile.server`) and smoke test |
| 11 | `helm` | Helm CLI + kubeconform | Helm chart lint, template render, and Kubernetes manifest validation |
The `coverage` job additionally depends on `lint`, `typecheck`, `security`, and
`quality` passing first. The `docker` job depends on `lint`, `typecheck`,
`security`, `quality`, and `unit_tests`.
---
## How to Diagnose a Broken Master
### Step 1: Identify the Failing Jobs
Navigate to the Forgejo Actions UI for the latest `master` commit and check
which jobs are failing. The `status-check` job output lists each job's result:
```
lint: failure
typecheck: success
security: success
quality: success
unit_tests: failure
integration_tests: success
e2e_tests: failure
coverage: skipped
build: success
docker: skipped
helm: success
```
Download the log artifact for each failing job (e.g., `ci-logs-lint`,
`ci-logs-unit-tests`, `ci-logs-e2e-tests`) from the Actions UI under
**Artifacts**.
### Step 2: Identify the Root Cause Commit
Check the recent commit history on `master` for any direct pushes (commits not
from a PR merge):
```bash
git log --oneline -10 origin/master
```
Direct pushes bypass CI entirely. Look for commits that were not preceded by a
PR merge commit. These are the most likely source of regressions.
### Step 3: Reproduce Locally
Clone the `master` branch and run the failing nox sessions locally:
```bash
git clone https://git.cleverthis.com/cleveragents/cleveragents-core.git
cd cleveragents-core
git checkout master
# Run the specific failing sessions
nox -s lint
nox -s format -- --check
nox -s unit_tests
nox -s e2e_tests
nox -s coverage_report
```
---
## Triage by Failure Type
### Lint Failures (`lint` job)
**Symptoms:** Ruff reports lint violations or format differences.
**Reproduce:**
```bash
nox -s lint
nox -s format -- --check
```
**Fix:** Run `nox -s format` to auto-fix formatting, then fix any remaining
lint violations reported by `nox -s lint`. Commit the result.
**Prohibited fixes:**
- Adding `# noqa` comments
- Modifying `pyproject.toml [tool.ruff]` to disable rules or add ignores
### Unit Test Failures (`unit_tests` job)
**Symptoms:** One or more Behave BDD scenarios fail.
**Reproduce:**
```bash
nox -s unit_tests
# Or run a specific feature file:
nox -s unit_tests -- features/failing_feature.feature
```
**Fix:** Identify the failing scenario and fix the **source code** to match the
expected behavior. Do **not** modify test expectations or skip scenarios.
**Prohibited fixes:**
- Adding `@skip`, `@xfail`, or `@unittest.skip` tags to scenarios
- Deleting or removing test scenarios or assertions
- Modifying test expectations to match broken behavior
### Integration Test Failures (`integration_tests` job)
**Symptoms:** One or more Robot Framework tests fail.
**Reproduce:**
```bash
nox -s integration_tests
# Or run a specific suite:
nox -s integration_tests -- --suite robot/failing_suite.robot
```
**Note:** Integration tests require real LLM API keys (`ANTHROPIC_API_KEY`,
`OPENAI_API_KEY`). Set these in your environment before running locally.
**Fix:** Identify the failing test and fix the underlying source code. Check
`robot/helpers_common.py` for the shared `reset_global_state()` function if
tests are failing due to state leakage between test runs.
### E2E Test Failures (`e2e_tests` job)
**Symptoms:** End-to-end Robot Framework tests under `robot/e2e/` fail.
**Reproduce:**
```bash
nox -s e2e_tests
```
**Note:** E2E tests require real LLM API keys (`ANTHROPIC_API_KEY`,
`OPENAI_API_KEY`, `GOOGLE_API_KEY`). These tests exercise full end-to-end
workflows with live LLM providers and have a 45-minute timeout in CI.
**Fix:** Fix the underlying source code. E2E tests must not be skipped or
mocked.
### Coverage Failures (`coverage` job)
**Symptoms:** `nox -s coverage_report` reports coverage below 97%.
**Reproduce:**
```bash
nox -s coverage_report
# Open the HTML report to identify uncovered lines:
open build/htmlcov/index.html
```
**Fix:** Write new Behave scenarios targeting the uncovered code paths. Coverage
must be ≥97% as measured by `nox -s coverage_report`.
**Prohibited fixes:**
- Lowering the `--fail-under` threshold in `noxfile.py`
- Adding `# pragma: no cover` comments
- Expanding the `omit` list in `pyproject.toml [tool.coverage.run]`
### Type Check Failures (`typecheck` job)
**Symptoms:** Pyright reports type errors.
**Reproduce:**
```bash
nox -s typecheck
```
**Fix:** Fix the type errors in source code. Add explicit type annotations where
missing.
**Prohibited fixes:**
- Adding `# type: ignore` comments
- Modifying `pyrightconfig.json` to relax strictness or add exclusions
- Adding paths to `pyproject.toml [tool.pyright]` exclusions
### Security Failures (`security` job)
**Symptoms:** Bandit reports HIGH-severity findings, Semgrep detects custom rule
violations, or Vulture detects dead code.
**Reproduce:**
```bash
nox -s security_scan
nox -s dead_code
```
**Fix:**
- **Bandit:** Fix the security issue in source code. If it is a genuine false
positive, consider a safer alternative first. Only as a last resort, add to
`[tool.bandit]` `skips` in `pyproject.toml`.
- **Semgrep:** Fix the code pattern flagged by the custom rule (see
`.semgrep.yml` for rule descriptions).
- **Vulture:** If the symbol is intentionally unused (e.g., required by a
protocol), add it to `vulture_whitelist.py` with a comment explaining why.
### Build Failures (`build` job)
**Symptoms:** `nox -s build` fails to produce a wheel.
**Reproduce:**
```bash
nox -s build
```
**Fix:** Fix the build error. Common causes include missing `__init__.py` files,
broken imports, or `pyproject.toml` misconfiguration.
### Docker Failures (`docker` job)
**Symptoms:** Docker image build fails or the smoke test (`--version`) fails.
**Reproduce:**
```bash
docker build -t cleveragents:test .
docker run --rm cleveragents:test --version
docker build -f Dockerfile.server -t cleveragents-server:test .
```
**Fix:** Fix the Dockerfile or the application code causing the build or smoke
test failure.
### Helm Failures (`helm` job)
**Symptoms:** Helm lint, template render, or kubeconform validation fails.
**Reproduce:**
```bash
helm dependency build ./k8s
helm lint ./k8s --set database.url="postgresql+asyncpg://user:pass@db-host:5432/cleveragents"
helm template cleveragents ./k8s \
--set database.url="postgresql+asyncpg://user:pass@db-host:5432/cleveragents" > /tmp/rendered.yaml
kubeconform -strict -ignore-missing-schemas -kubernetes-version 1.29.0 -summary /tmp/rendered.yaml
```
**Fix:** Fix the Helm chart templates or values in `k8s/`.
---
## The Fix Branch Workflow
When `master` is broken, the fix **must** go through the PR process — never
push directly to `master` again:
1. **Create a fix branch** from the current `master` HEAD:
```bash
git checkout -b fix/master-ci-quality-gates origin/master
```
2. **Fix the actual code** (see triage sections above). Never suppress or bypass
quality gates.
3. **Verify locally** that all nox sessions pass:
```bash
nox # runs all default sessions
```
4. **Commit** with the conventional commit format:
```
fix(ci): restore all CI quality gates to passing on master
- Fixed lint violations in src/...
- Fixed failing Behave scenario in features/...
- Fixed E2E test failure in robot/e2e/...
- Coverage restored to 97.X%
ISSUES CLOSED: #2597
```
5. **Push** the fix branch and **open a PR** to `master`.
6. **Wait for CI** to pass on the fix branch before merging.
7. **After merge**, verify the CI pipeline on the resulting `master` commit
passes all 11 jobs.
---
## Prohibited Actions
The following changes are **strictly prohibited** when fixing CI failures.
They suppress or bypass quality enforcement rather than fixing the underlying
code:
| Prohibited Change | Why |
|---|---|
| Adding `# type: ignore` or Pyright suppression directives | Hides type errors instead of fixing them |
| Adding `# noqa` or Ruff suppression directives | Hides lint violations instead of fixing them |
| Adding `@skip`, `@xfail`, `@unittest.skip` to tests | Hides test failures instead of fixing the code |
| Modifying `pyrightconfig.json` to relax strictness | Weakens the type checking gate |
| Modifying `pyproject.toml [tool.ruff]` to disable rules | Weakens the lint gate |
| Modifying Bandit config to suppress findings | Weakens the security gate |
| Modifying `.semgrep.yml` to exclude patterns | Weakens the security gate |
| Modifying `vulture_whitelist.py` to suppress legitimate findings | Weakens the dead-code gate |
| Reducing the coverage threshold below 97% | Weakens the coverage gate |
| Modifying `.forgejo/workflows/ci.yml` to skip or make optional any required job | Weakens the CI gate |
| Deleting or removing test files, scenarios, or assertions | Hides test failures |
| Adding `success_codes` workarounds to nox sessions | Hides failures |
| Pushing directly to `master` without a PR | Bypasses CI entirely |
---
## Prevention: No Direct Pushes to Master
The root cause of issue #2597 was direct pushes to `master` that bypassed CI.
The `no-commit-to-branch` pre-commit hook prevents this locally, but it can be
bypassed with `--no-verify`.
**Never push directly to `master`.** All changes must go through a PR with CI
passing. This is enforced by branch protection rules in Forgejo (see
[CI/CD Pipeline and Branch Protection](ci-cd.md)).
If you find yourself needing to push directly to `master` for an emergency fix,
stop and ask: can this wait for a PR? In almost all cases, the answer is yes.
A broken `master` caused by a direct push is far more damaging than the delay
of a PR review.
---
## Related Documentation
- [CI/CD Pipeline and Branch Protection](ci-cd.md) — Full CI pipeline reference
and branch protection rules
- [Quality Automation Guide](quality-automation.md) — Pre-commit hooks, nox
sessions, and quality gate details
- [Testing Guide](testing.md) — Unit tests (Behave), integration tests (Robot
Framework), and coverage requirements
## Related Issues
- **#2597** — fix(ci): restore all CI quality gates to passing on master
- **#2463** — Earlier automated report of CI failures on master (superseded by #2597)
docs/development/quality-automation.md
+18 -11
View File
@@ -71,17 +71,24 @@ The CI pipeline runs on **Forgejo Actions** (`.forgejo/workflows/ci.yml`).
### CI Jobs
| Job | Trigger | Purpose | Failure Impact |
| ----------- | ------- | ----------------------------------- | ---------------------- |
| `lint` | Push/PR | Ruff format + lint check | Blocks merge |
| `typecheck` | Push/PR | Pyright type checking | Blocks merge |
| `security` | Push/PR | Bandit + Semgrep + Vulture | Blocks merge |
| `quality` | Push/PR | Radon complexity check | Blocks merge (grade F) |
| `unit_tests`| Push/PR | Behave BDD tests (Python 3.13) | Blocks merge |
| `coverage` | Push/PR | Coverage measurement | Blocks merge (<97%) |
| `build` | Push/PR | Wheel build | Blocks release |
| `docker` | Push/PR | Docker image build + test | Blocks deployment |
| `helm` | Push/PR | Helm chart lint + template | Blocks deployment |
All 11 jobs below must pass for the `status-check` consolidation gate to
succeed. A failure in any single job blocks all PR merges. See
[CI Incident Runbook](ci-incident-runbook.md) for diagnosis and recovery.
| Job | Trigger | Purpose | Failure Impact |
| ------------------- | ------- | ---------------------------------------------- | --------------------------- |
| `lint` | Push/PR | Ruff format + lint check | Blocks merge |
| `typecheck` | Push/PR | Pyright strict type checking | Blocks merge |
| `security` | Push/PR | Bandit + Semgrep + Vulture | Blocks merge |
| `quality` | Push/PR | Radon complexity check | Blocks merge (grade F) |
| `unit_tests` | Push/PR | Behave BDD tests (Python 3.13) | Blocks merge |
| `integration_tests` | Push/PR | Robot Framework integration tests | Blocks merge |
| `e2e_tests` | Push/PR | Robot Framework E2E tests (real LLM API keys) | Blocks merge |
| `coverage` | Push/PR | Coverage measurement (≥97% threshold) | Blocks merge (<97%) |
| `build` | Push/PR | Wheel build | Blocks release |
| `docker` | Push/PR | Docker image build + smoke test | Blocks deployment |
| `helm` | Push/PR | Helm chart lint, template render, kubeconform | Blocks deployment |
| `status-check` | Push/PR | Consolidation gate (requires all 11 above) | Blocks merge (branch guard) |
### Nightly Quality
mkdocs.yml
+1
View File
@@ -24,6 +24,7 @@ nav:
- TUI: api/tui.md
- Development:
- CI/CD Pipeline: development/ci-cd.md
- CI Incident Runbook: development/ci-incident-runbook.md
- Quality Automation: development/quality-automation.md
- Testing Guide: development/testing.md
- Review Playbook: development/review_playbook.md