cleveragents-core/docs/development/ops-runbook.md

# Ops Runbook: TLS Certificate Management

This runbook documents procedures for diagnosing, renewing, and monitoring
TLS certificates for CleverAgents infrastructure hostnames.

## Affected Hostnames

| Hostname | Purpose |
|---|---|
| `git.cleverthis.com` | Primary Forgejo git server (production) |
| `git.dev.cleveragents.com` | Development Forgejo git server |
| `git.cleveragents.com` | Alias / secondary git endpoint |

## Diagnosing TLS Issues

### Symptom: `gnutls_handshake() failed: The server name sent was not recognized`

This error indicates one of:

1. **Missing SAN** — The TLS certificate does not list the target hostname as a
   Subject Alternative Name (SAN).
2. **SNI misconfiguration** — The web server (nginx/caddy/traefik) is not routing
   the SNI hostname to the correct virtual host.
3. **Expired certificate** — The certificate has passed its `notAfter` date.

### Step 1: Inspect the Certificate

Use `openssl s_client` to retrieve and inspect the certificate:

```bash
# Check SANs and expiry for git.dev.cleveragents.com
openssl s_client \
  -connect git.dev.cleveragents.com:443 \
  -servername git.dev.cleveragents.com \
  2>/dev/null | openssl x509 -noout -text | grep -A 10 "Subject Alternative Name"

# Check expiry date
openssl s_client \
  -connect git.dev.cleveragents.com:443 \
  -servername git.dev.cleveragents.com \
  2>/dev/null | openssl x509 -noout -dates
```

Or use the project's TLS check script (no external tools required):

```bash
python scripts/check-tls-cert.py git.dev.cleveragents.com git.cleveragents.com
```

### Step 2: Identify the Root Cause

| Symptom | Root Cause | Fix |
|---|---|---|
| `CERTIFICATE_VERIFY_FAILED` with `hostname mismatch` | Hostname not in SANs | Reissue certificate with correct SANs |
| `SSL_ERROR_RX_RECORD_TOO_LONG` | HTTP served on HTTPS port | Fix web server config |
| `gnutls_handshake() failed: The server name sent was not recognized` | SNI virtual host not configured | Add SNI vhost binding |
| Certificate expired | `notAfter` in the past | Renew certificate immediately |
| `GIT_SSL_NO_VERIFY` bypasses but still fails | Server-side SNI rejection | Fix SNI routing (not a client cert issue) |

### Step 3: Verify SNI Routing

If the certificate has the correct SANs but the error persists, the web server
may not be routing the SNI hostname to the correct virtual host:

```bash
# nginx: check for server_name directive
grep -r "server_name.*git.dev.cleveragents.com" /etc/nginx/

# caddy: check Caddyfile
grep -r "git.dev.cleveragents.com" /etc/caddy/

# traefik: check router rules
grep -r "git.dev.cleveragents.com" /etc/traefik/
```

---

## Certificate Renewal Procedure

### Let's Encrypt (Certbot) — Recommended

Most CleverAgents infrastructure uses Let's Encrypt certificates managed by
certbot. The renewal process is:

#### 1. Verify certbot is installed and configured

```bash
certbot --version
certbot certificates
```

#### 2. Check which certificate covers the affected hostname

```bash
certbot certificates | grep -A 5 "git.dev.cleveragents.com"
```

#### 3. Renew the certificate

```bash
# Dry run first to verify renewal will succeed
certbot renew --dry-run --cert-name <cert-name>

# Actual renewal
certbot renew --cert-name <cert-name>
```

#### 4. If the hostname is missing from the certificate's SANs

The certificate must be reissued with the additional hostname:

```bash
# Add git.dev.cleveragents.com to an existing certificate
certbot certonly \
  --expand \
  --cert-name <cert-name> \
  -d git.cleverthis.com \
  -d git.cleveragents.com \
  -d git.dev.cleveragents.com
```

> **Note:** Replace `<cert-name>` with the actual certificate name shown by
> `certbot certificates`. The `-d` flags must list ALL hostnames the certificate
> should cover, including existing ones.

#### 5. Reload the web server

After renewal, reload the web server to pick up the new certificate:

```bash
# nginx
systemctl reload nginx

# caddy
systemctl reload caddy

# traefik (usually auto-reloads; check logs)
journalctl -u traefik -n 50
```

#### 6. Verify the fix

```bash
python scripts/check-tls-cert.py git.dev.cleveragents.com git.cleveragents.com

# Or with openssl
openssl s_client \
  -connect git.dev.cleveragents.com:443 \
  -servername git.dev.cleveragents.com \
  2>/dev/null | openssl x509 -noout -text | grep -A 10 "Subject Alternative Name"
```

#### 7. Verify git clone works

```bash
git clone https://git.dev.cleveragents.com/cleveragents/cleveragents-core.git /tmp/test-clone
rm -rf /tmp/test-clone
```

---

### Manual Certificate (non-Let's Encrypt)

If the infrastructure uses manually managed certificates:

1. Generate a new CSR including all required SANs:

   ```bash
   # Create OpenSSL config with SANs
   cat > /tmp/san.cnf << EOF
   [req]
   distinguished_name = req_distinguished_name
   req_extensions = v3_req
   prompt = no

   [req_distinguished_name]
   CN = git.cleverthis.com

   [v3_req]
   subjectAltName = @alt_names

   [alt_names]
   DNS.1 = git.cleverthis.com
   DNS.2 = git.cleveragents.com
   DNS.3 = git.dev.cleveragents.com
   EOF

   # Generate private key and CSR
   openssl req -new -newkey rsa:4096 -nodes \
     -keyout /etc/ssl/private/git-cleverthis.key \
     -out /tmp/git-cleverthis.csr \
     -config /tmp/san.cnf
   ```

2. Submit the CSR to your CA and obtain the signed certificate.

3. Install the certificate and reload the web server.

---

## Certificate Expiry Monitoring

### Automated Monitoring with the TLS Check Script

Run the TLS check script regularly to detect expiring certificates before they
cause outages:

```bash
# Check all infrastructure hostnames with 30-day warning threshold
python scripts/check-tls-cert.py \
  git.cleverthis.com \
  git.cleveragents.com \
  git.dev.cleveragents.com \
  --warn-days 30
```

### Recommended Alert Thresholds

| Threshold | Action |
|---|---|
| 30 days | Warning — schedule renewal |
| 14 days | Urgent — renew immediately |
| 7 days | Critical — escalate to on-call |
| 0 days | Outage — emergency renewal |

### Cron Job Setup

Add a daily cron job to check certificate health:

```bash
# /etc/cron.d/tls-cert-check
0 8 * * * root cd /path/to/cleveragents-core && \
  python scripts/check-tls-cert.py \
    git.cleverthis.com \
    git.cleveragents.com \
    git.dev.cleveragents.com \
    --warn-days 30 \
  | mail -s "TLS Cert Check" ops@cleverthis.com
```

### Let's Encrypt Auto-Renewal

Let's Encrypt certificates are valid for 90 days. Certbot installs a systemd
timer or cron job that attempts renewal when the certificate has fewer than
30 days remaining:

```bash
# Check certbot timer status
systemctl status certbot.timer

# Manually trigger renewal check
certbot renew --dry-run
```

---

## Related Issues

- **#1532** — BUG-HUNT: TLS Configuration Error on `git.cleveragents.com`
- **#1541** — TEST-INFRA: Unable to push to repository
- **#1543** — fix(infra): resolve TLS handshake failure on `git.dev.cleveragents.com`

---

## Escalation Path

1. **First responder:** Check certificate with `scripts/check-tls-cert.py`
2. **If renewal needed:** Follow [Certificate Renewal Procedure](#certificate-renewal-procedure)
3. **If SNI misconfiguration:** Contact server admin with web server config access
4. **If CA/PKI issue:** Escalate to infrastructure team lead

---

*Last updated: 2026-04-02 — Added as part of issue #1543 resolution.*