cleveragents-core/docs/development/ops-runbook.md

Ops Runbook: TLS Certificate Management

This runbook documents procedures for diagnosing, renewing, and monitoring TLS certificates for CleverAgents infrastructure hostnames.

Affected Hostnames

Hostname	Purpose
git.cleverthis.com	Primary Forgejo git server (production)
git.dev.cleveragents.com	Development Forgejo git server
git.cleveragents.com	Alias / secondary git endpoint

Diagnosing TLS Issues

Symptom: gnutls_handshake() failed: The server name sent was not recognized

This error indicates one of:

1. Missing SAN — The TLS certificate does not list the target hostname as a Subject Alternative Name (SAN).
2. SNI misconfiguration — The web server (nginx/caddy/traefik) is not routing the SNI hostname to the correct virtual host.
3. Expired certificate — The certificate has passed its notAfter date.

Step 1: Inspect the Certificate

Use openssl s_client to retrieve and inspect the certificate:

# Check SANs and expiry for git.dev.cleveragents.com
openssl s_client \
  -connect git.dev.cleveragents.com:443 \
  -servername git.dev.cleveragents.com \
  2>/dev/null | openssl x509 -noout -text | grep -A 10 "Subject Alternative Name"

# Check expiry date
openssl s_client \
  -connect git.dev.cleveragents.com:443 \
  -servername git.dev.cleveragents.com \
  2>/dev/null | openssl x509 -noout -dates

Or use the project's TLS check script (no external tools required):

python scripts/check-tls-cert.py git.dev.cleveragents.com git.cleveragents.com

Step 2: Identify the Root Cause

Symptom	Root Cause	Fix
CERTIFICATE_VERIFY_FAILED with hostname mismatch	Hostname not in SANs	Reissue certificate with correct SANs
SSL_ERROR_RX_RECORD_TOO_LONG	HTTP served on HTTPS port	Fix web server config
gnutls_handshake() failed: The server name sent was not recognized	SNI virtual host not configured	Add SNI vhost binding
Certificate expired	notAfter in the past	Renew certificate immediately
GIT_SSL_NO_VERIFY bypasses but still fails	Server-side SNI rejection	Fix SNI routing (not a client cert issue)

Step 3: Verify SNI Routing

If the certificate has the correct SANs but the error persists, the web server may not be routing the SNI hostname to the correct virtual host:

# nginx: check for server_name directive
grep -r "server_name.*git.dev.cleveragents.com" /etc/nginx/

# caddy: check Caddyfile
grep -r "git.dev.cleveragents.com" /etc/caddy/

# traefik: check router rules
grep -r "git.dev.cleveragents.com" /etc/traefik/

---

Certificate Renewal Procedure

Let's Encrypt (Certbot) — Recommended

Most CleverAgents infrastructure uses Let's Encrypt certificates managed by certbot. The renewal process is:

1. Verify certbot is installed and configured

certbot --version
certbot certificates

2. Check which certificate covers the affected hostname

certbot certificates | grep -A 5 "git.dev.cleveragents.com"

3. Renew the certificate

# Dry run first to verify renewal will succeed
certbot renew --dry-run --cert-name <cert-name>

# Actual renewal
certbot renew --cert-name <cert-name>

4. If the hostname is missing from the certificate's SANs

The certificate must be reissued with the additional hostname:

# Add git.dev.cleveragents.com to an existing certificate
certbot certonly \
  --expand \
  --cert-name <cert-name> \
  -d git.cleverthis.com \
  -d git.cleveragents.com \
  -d git.dev.cleveragents.com

Note: Replace <cert-name> with the actual certificate name shown by certbot certificates. The -d flags must list ALL hostnames the certificate should cover, including existing ones.

5. Reload the web server

After renewal, reload the web server to pick up the new certificate:

# nginx
systemctl reload nginx

# caddy
systemctl reload caddy

# traefik (usually auto-reloads; check logs)
journalctl -u traefik -n 50

6. Verify the fix

python scripts/check-tls-cert.py git.dev.cleveragents.com git.cleveragents.com

# Or with openssl
openssl s_client \
  -connect git.dev.cleveragents.com:443 \
  -servername git.dev.cleveragents.com \
  2>/dev/null | openssl x509 -noout -text | grep -A 10 "Subject Alternative Name"

7. Verify git clone works

git clone https://git.dev.cleveragents.com/cleveragents/cleveragents-core.git /tmp/test-clone
rm -rf /tmp/test-clone

---

Manual Certificate (non-Let's Encrypt)

If the infrastructure uses manually managed certificates:

1. Generate a new CSR including all required SANs:

   # Create OpenSSL config with SANs
   cat > /tmp/san.cnf << EOF
   [req]
   distinguished_name = req_distinguished_name
   req_extensions = v3_req
   prompt = no
   
   [req_distinguished_name]
   CN = git.cleverthis.com
   
   [v3_req]
   subjectAltName = @alt_names
   
   [alt_names]
   DNS.1 = git.cleverthis.com
   DNS.2 = git.cleveragents.com
   DNS.3 = git.dev.cleveragents.com
   EOF
   
   # Generate private key and CSR
   openssl req -new -newkey rsa:4096 -nodes \
     -keyout /etc/ssl/private/git-cleverthis.key \
     -out /tmp/git-cleverthis.csr \
     -config /tmp/san.cnf
   

2. Submit the CSR to your CA and obtain the signed certificate.

3. Install the certificate and reload the web server.

---

Certificate Expiry Monitoring

Automated Monitoring with the TLS Check Script

Run the TLS check script regularly to detect expiring certificates before they cause outages:

# Check all infrastructure hostnames with 30-day warning threshold
python scripts/check-tls-cert.py \
  git.cleverthis.com \
  git.cleveragents.com \
  git.dev.cleveragents.com \
  --warn-days 30

Recommended Alert Thresholds

Threshold	Action
30 days	Warning — schedule renewal
14 days	Urgent — renew immediately
7 days	Critical — escalate to on-call
0 days	Outage — emergency renewal

Cron Job Setup

Add a daily cron job to check certificate health:

# /etc/cron.d/tls-cert-check
0 8 * * * root cd /path/to/cleveragents-core && \
  python scripts/check-tls-cert.py \
    git.cleverthis.com \
    git.cleveragents.com \
    git.dev.cleveragents.com \
    --warn-days 30 \
  | mail -s "TLS Cert Check" ops@cleverthis.com

Let's Encrypt Auto-Renewal

Let's Encrypt certificates are valid for 90 days. Certbot installs a systemd timer or cron job that attempts renewal when the certificate has fewer than 30 days remaining:

# Check certbot timer status
systemctl status certbot.timer

# Manually trigger renewal check
certbot renew --dry-run

---

Related Issues

• #1532 — BUG-HUNT: TLS Configuration Error on git.cleveragents.com
• #1541 — TEST-INFRA: Unable to push to repository
• #1543 — fix(infra): resolve TLS handshake failure on git.dev.cleveragents.com

---

Escalation Path

1. First responder: Check certificate with scripts/check-tls-cert.py
2. If renewal needed: Follow Certificate Renewal Procedure
3. If SNI misconfiguration: Contact server admin with web server config access
4. If CA/PKI issue: Escalate to infrastructure team lead

---

Last updated: 2026-04-02 — Added as part of issue #1543 resolution.