# Tunnel Rotation Protocol

**Incident Response** | Governed by [RED-BOOK.md](../RED-BOOK.md)

## The Arteries Must Shed Their Old Keys and Be Reborn

*Cloudflare Tunnels are the veins through which the mesh breathes. When credentials age or suspicion arises, the tunnels must be dissolved and reformed — a controlled death and resurrection that preserves continuity while eliminating compromise vectors.*

---

## I. When to Rotate

### Scheduled Rotation (Prophylactic)

| Trigger | Interval | Priority |
|---------|----------|----------|
| Standard credential hygiene | Every 90 days | NORMAL |
| After personnel change | Within 24 hours | HIGH |
| Compliance audit requirement | As specified | NORMAL |
| Post-incident (any severity) | Immediately | CRITICAL |

### Emergency Rotation (Reactive)

| Trigger | Response Time |
|---------|---------------|
| Credential exposure suspected | < 1 hour |
| Tunnel behaving anomalously | < 2 hours |
| Unauthorized connection detected | Immediate |
| Origin server compromised | Immediate |
| Security advisory from Cloudflare | < 24 hours |

---

## II. NIGREDO — Preparation

### Pre-Rotation Checklist

Before beginning rotation:

- [ ] Identify all tunnels requiring rotation
- [ ] Document current tunnel configurations
- [ ] Verify backup ingress path (if available)
- [ ] Notify dependent teams of maintenance window
- [ ] Prepare new tunnel names and secrets
- [ ] Ensure Terraform state is current

### Inventory Current State

```bash
# List all tunnels
cloudflared tunnel list

# Export tunnel info
for tunnel_id in $(cloudflared tunnel list | tail -n +2 | awk '{print $1}'); do
  cloudflared tunnel info $tunnel_id > /tmp/tunnel_${tunnel_id}_info.txt
done

# Capture current routes
cloudflared tunnel route dns list

# Hash for audit trail
cat /tmp/tunnel_*.txt | blake3sum > pre_rotation_state.hash
```

### Generate New Secrets

```bash
# Generate cryptographically secure tunnel secrets
NEW_SECRET_VAULTMESH=$(openssl rand -base64 32)
NEW_SECRET_OFFSEC=$(openssl rand -base64 32)

# Store securely (example: HashiCorp Vault)
vault kv put secret/cloudflare/tunnels \
  vaultmesh_secret="$NEW_SECRET_VAULTMESH" \
  offsec_secret="$NEW_SECRET_OFFSEC"

# Or for local encrypted storage
echo "$NEW_SECRET_VAULTMESH" | gpg --encrypt -r guardian@vaultmesh.org > vaultmesh_tunnel_secret.gpg
echo "$NEW_SECRET_OFFSEC" | gpg --encrypt -r guardian@vaultmesh.org > offsec_tunnel_secret.gpg
```

---

## III. ALBEDO — Dissolution

### Step 1: Create New Tunnel (Before Destroying Old)

```bash
# Create new tunnel with fresh credentials
cloudflared tunnel create vaultmesh-tunnel-$(date +%Y%m%d)

# This generates:
# - New tunnel ID
# - New credentials JSON in ~/.cloudflared/

# Move credentials to secure location
sudo mv ~/.cloudflared/<new_tunnel_id>.json /etc/cloudflared/
sudo chmod 600 /etc/cloudflared/<new_tunnel_id>.json
sudo chown cloudflared:cloudflared /etc/cloudflared/<new_tunnel_id>.json
```

### Step 2: Configure New Tunnel

Update `/etc/cloudflared/config.yml`:

```yaml
tunnel: <NEW_TUNNEL_ID>
credentials-file: /etc/cloudflared/<NEW_TUNNEL_ID>.json

metrics: 127.0.0.1:9090

ingress:
  - hostname: api.vaultmesh.org
    service: http://localhost:8080
    originRequest:
      connectTimeout: 10s
      noTLSVerify: false

  - hostname: dash.vaultmesh.org
    service: http://localhost:3000

  - service: http_status:404
```

### Step 3: Update DNS Routes

```bash
# Route hostnames to new tunnel
cloudflared tunnel route dns <NEW_TUNNEL_ID> api.vaultmesh.org
cloudflared tunnel route dns <NEW_TUNNEL_ID> dash.vaultmesh.org

# Verify routing
cloudflared tunnel route dns list | grep <NEW_TUNNEL_ID>
```

### Step 4: Transition Traffic

#### Zero-Downtime Method (Preferred)

```bash
# 1. Start new tunnel alongside old
sudo systemctl start cloudflared-new.service

# 2. Verify new tunnel is healthy
curl -s http://127.0.0.1:9091/ready  # New tunnel metrics port

# 3. Update DNS CNAMEs to point to new tunnel
# (Already done in Step 3, propagation takes ~30s with Cloudflare proxy)

# 4. Monitor traffic shift
watch -n5 'curl -s http://127.0.0.1:9090/metrics | grep requests'
watch -n5 'curl -s http://127.0.0.1:9091/metrics | grep requests'

# 5. Once old tunnel shows zero traffic, proceed to deletion
```

#### Maintenance Window Method

```bash
# 1. Stop old tunnel
sudo systemctl stop cloudflared.service

# 2. Update config to new tunnel
sudo cp /etc/cloudflared/config-new.yml /etc/cloudflared/config.yml

# 3. Start service
sudo systemctl start cloudflared.service

# 4. Verify connectivity
cloudflared tunnel info <NEW_TUNNEL_ID>
curl -I https://api.vaultmesh.org
```

---

## IV. CITRINITAS — Purification

### Delete Old Tunnel

**Warning**: Only proceed after verifying new tunnel is fully operational.

```bash
# 1. Final verification - old tunnel should have zero active connections
cloudflared tunnel info <OLD_TUNNEL_ID>

# 2. Remove DNS routes from old tunnel (if any remain)
cloudflared tunnel route dns delete <OLD_TUNNEL_ID> <hostname>

# 3. Delete the tunnel
cloudflared tunnel delete <OLD_TUNNEL_ID>

# 4. Securely destroy old credentials
sudo shred -vfz -n 5 /etc/cloudflared/<OLD_TUNNEL_ID>.json
sudo rm /etc/cloudflared/<OLD_TUNNEL_ID>.json
```

### Clean Up Local Artifacts

```bash
# Remove old credential backups
find /var/lib/vaultmesh/backups -name "*<OLD_TUNNEL_ID>*" -exec shred -vfz {} \;

# Clear any cached tunnel state
rm -rf ~/.cloudflared/connectors/<OLD_TUNNEL_ID>

# Update Terraform state
cd ~/Desktop/CLOUDFLARE/terraform
terraform state rm cloudflare_tunnel.old_tunnel  # If managed by TF
```

---

## V. RUBEDO — Verification & Sealing

### Post-Rotation Verification

```bash
#!/bin/bash
# rotation_verification.sh

TUNNEL_ID="<NEW_TUNNEL_ID>"
HOSTNAMES=("api.vaultmesh.org" "dash.vaultmesh.org")

echo "=== Tunnel Rotation Verification ==="
echo "Tunnel ID: $TUNNEL_ID"
echo "Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo ""

# 1. Tunnel status
echo "--- Tunnel Status ---"
cloudflared tunnel info $TUNNEL_ID

# 2. DNS routing
echo ""
echo "--- DNS Routes ---"
cloudflared tunnel route dns list | grep $TUNNEL_ID

# 3. Endpoint connectivity
echo ""
echo "--- Endpoint Tests ---"
for hostname in "${HOSTNAMES[@]}"; do
  status=$(curl -s -o /dev/null -w "%{http_code}" https://$hostname/health 2>/dev/null || echo "FAIL")
  echo "$hostname: $status"
done

# 4. Metrics endpoint
echo ""
echo "--- Metrics Check ---"
curl -s http://127.0.0.1:9090/metrics | grep cloudflared_tunnel | head -5

# 5. Certificate validation
echo ""
echo "--- TLS Verification ---"
for hostname in "${HOSTNAMES[@]}"; do
  echo | openssl s_client -connect $hostname:443 -servername $hostname 2>/dev/null | openssl x509 -noout -dates
done
```

### Emit Rotation Receipt

```json
{
  "receipt_type": "tunnel_rotation",
  "schema_version": "vm_tunnel_rotation_v1",
  "timestamp": "<ISO8601>",
  "rotation_id": "<uuid>",
  "old_tunnel": {
    "id": "<OLD_TUNNEL_ID>",
    "created": "<original_creation_date>",
    "deleted": "<deletion_timestamp>"
  },
  "new_tunnel": {
    "id": "<NEW_TUNNEL_ID>",
    "created": "<creation_timestamp>",
    "hostnames": ["api.vaultmesh.org", "dash.vaultmesh.org"]
  },
  "reason": "scheduled_rotation | incident_response | personnel_change",
  "verification_hash": "<blake3_of_verification_output>",
  "operator_did": "did:vm:operator:<id>",
  "guardian_sign": "<tem_signature>"
}
```

### Anchor the Rotation

```bash
# Compute rotation proof
cat rotation_verification.txt rotation_receipt.json | blake3sum > rotation_proof.hash

# Append to ProofChain
echo "{\"type\":\"tunnel_rotation\",\"hash\":\"$(cat rotation_proof.hash | cut -d' ' -f1)\",\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}" \
  >> /var/lib/vaultmesh/proofchain/anchors.jsonl

# Update Terraform state
cd ~/Desktop/CLOUDFLARE/terraform
terraform plan -out=rotation.tfplan
terraform apply rotation.tfplan
```

---

## VI. Automation Script

For scheduled rotations, use this automation wrapper:

```bash
#!/bin/bash
# tunnel_rotation_automated.sh
# Run via cron or GitLab CI on schedule

set -euo pipefail

TUNNEL_NAME="$1"
NEW_TUNNEL_NAME="${TUNNEL_NAME}-$(date +%Y%m%d)"
LOG_FILE="/var/log/tunnel_rotation_$(date +%Y%m%d).log"

log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $1" | tee -a "$LOG_FILE"; }

log "Starting rotation for tunnel: $TUNNEL_NAME"

# Get old tunnel ID
OLD_TUNNEL_ID=$(cloudflared tunnel list | grep "$TUNNEL_NAME" | awk '{print $1}')
log "Old tunnel ID: $OLD_TUNNEL_ID"

# Create new tunnel
log "Creating new tunnel: $NEW_TUNNEL_NAME"
cloudflared tunnel create "$NEW_TUNNEL_NAME"
NEW_TUNNEL_ID=$(cloudflared tunnel list | grep "$NEW_TUNNEL_NAME" | awk '{print $1}')
log "New tunnel ID: $NEW_TUNNEL_ID"

# Move credentials
sudo mv ~/.cloudflared/${NEW_TUNNEL_ID}.json /etc/cloudflared/
sudo chmod 600 /etc/cloudflared/${NEW_TUNNEL_ID}.json

# Update config
sudo sed -i "s/$OLD_TUNNEL_ID/$NEW_TUNNEL_ID/g" /etc/cloudflared/config.yml

# Restart service
sudo systemctl restart cloudflared.service
sleep 10

# Verify
if cloudflared tunnel info "$NEW_TUNNEL_ID" | grep -q "HEALTHY"; then
  log "New tunnel is healthy"

  # Delete old tunnel
  cloudflared tunnel delete "$OLD_TUNNEL_ID"
  sudo shred -vfz /etc/cloudflared/${OLD_TUNNEL_ID}.json 2>/dev/null || true

  log "Rotation complete"
else
  log "ERROR: New tunnel not healthy, rolling back"
  sudo sed -i "s/$NEW_TUNNEL_ID/$OLD_TUNNEL_ID/g" /etc/cloudflared/config.yml
  sudo systemctl restart cloudflared.service
  cloudflared tunnel delete "$NEW_TUNNEL_ID"
  exit 1
fi

# Emit receipt
cat > /var/lib/vaultmesh/receipts/rotation_$(date +%Y%m%d).json <<EOF
{
  "receipt_type": "tunnel_rotation",
  "old_tunnel_id": "$OLD_TUNNEL_ID",
  "new_tunnel_id": "$NEW_TUNNEL_ID",
  "timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
  "status": "success"
}
EOF

log "Receipt emitted"
```

---

## VII. The Arteries Renewed

*With old credentials destroyed and new pathways verified, the tunnel stands reborn. The mesh breathes through fresh veins, uncontaminated by the past. The rotation is complete, the proof anchored, and the guardian satisfied.*

### Post-Rotation Checklist

- [ ] New tunnel ID documented
- [ ] Old tunnel deleted and credentials destroyed
- [ ] DNS routes verified pointing to new tunnel
- [ ] All endpoints responding correctly
- [ ] Metrics flowing from new tunnel
- [ ] VaultMesh receipt emitted
- [ ] ProofChain anchor created
- [ ] Terraform state updated
- [ ] Next rotation scheduled (90 days)

---

**Document Version**: 1.0
**Last Updated**: <date>
**Guardian**: Tem
**Rotation Schedule**: Every 90 days or upon incident