vm-cloudflare/playbooks/TUNNEL-ROTATION-PROTOCOL.md

Tunnel Rotation Protocol

Incident Response | Governed by RED-BOOK.md

The Arteries Must Shed Their Old Keys and Be Reborn

Cloudflare Tunnels are the veins through which the mesh breathes. When credentials age or suspicion arises, the tunnels must be dissolved and reformed — a controlled death and resurrection that preserves continuity while eliminating compromise vectors.

---

I. When to Rotate

Scheduled Rotation (Prophylactic)

Trigger	Interval	Priority
Standard credential hygiene	Every 90 days	NORMAL
After personnel change	Within 24 hours	HIGH
Compliance audit requirement	As specified	NORMAL
Post-incident (any severity)	Immediately	CRITICAL

Emergency Rotation (Reactive)

Trigger	Response Time
Credential exposure suspected	< 1 hour
Tunnel behaving anomalously	< 2 hours
Unauthorized connection detected	Immediate
Origin server compromised	Immediate
Security advisory from Cloudflare	< 24 hours

---

II. NIGREDO — Preparation

Pre-Rotation Checklist

Before beginning rotation:

• Identify all tunnels requiring rotation
• Document current tunnel configurations
• Verify backup ingress path (if available)
• Notify dependent teams of maintenance window
• Prepare new tunnel names and secrets
• Ensure Terraform state is current

Inventory Current State

# List all tunnels
cloudflared tunnel list

# Export tunnel info
for tunnel_id in $(cloudflared tunnel list | tail -n +2 | awk '{print $1}'); do
  cloudflared tunnel info $tunnel_id > /tmp/tunnel_${tunnel_id}_info.txt
done

# Capture current routes
cloudflared tunnel route dns list

# Hash for audit trail
cat /tmp/tunnel_*.txt | blake3sum > pre_rotation_state.hash

Generate New Secrets

# Generate cryptographically secure tunnel secrets
NEW_SECRET_VAULTMESH=$(openssl rand -base64 32)
NEW_SECRET_OFFSEC=$(openssl rand -base64 32)

# Store securely (example: HashiCorp Vault)
vault kv put secret/cloudflare/tunnels \
  vaultmesh_secret="$NEW_SECRET_VAULTMESH" \
  offsec_secret="$NEW_SECRET_OFFSEC"

# Or for local encrypted storage
echo "$NEW_SECRET_VAULTMESH" | gpg --encrypt -r guardian@vaultmesh.org > vaultmesh_tunnel_secret.gpg
echo "$NEW_SECRET_OFFSEC" | gpg --encrypt -r guardian@vaultmesh.org > offsec_tunnel_secret.gpg

---

III. ALBEDO — Dissolution

Step 1: Create New Tunnel (Before Destroying Old)

# Create new tunnel with fresh credentials
cloudflared tunnel create vaultmesh-tunnel-$(date +%Y%m%d)

# This generates:
# - New tunnel ID
# - New credentials JSON in ~/.cloudflared/

# Move credentials to secure location
sudo mv ~/.cloudflared/<new_tunnel_id>.json /etc/cloudflared/
sudo chmod 600 /etc/cloudflared/<new_tunnel_id>.json
sudo chown cloudflared:cloudflared /etc/cloudflared/<new_tunnel_id>.json

Step 2: Configure New Tunnel

Update /etc/cloudflared/config.yml:

tunnel: <NEW_TUNNEL_ID>
credentials-file: /etc/cloudflared/<NEW_TUNNEL_ID>.json

metrics: 127.0.0.1:9090

ingress:
  - hostname: api.vaultmesh.org
    service: http://localhost:8080
    originRequest:
      connectTimeout: 10s
      noTLSVerify: false

  - hostname: dash.vaultmesh.org
    service: http://localhost:3000

  - service: http_status:404

Step 3: Update DNS Routes

# Route hostnames to new tunnel
cloudflared tunnel route dns <NEW_TUNNEL_ID> api.vaultmesh.org
cloudflared tunnel route dns <NEW_TUNNEL_ID> dash.vaultmesh.org

# Verify routing
cloudflared tunnel route dns list | grep <NEW_TUNNEL_ID>

Step 4: Transition Traffic

Zero-Downtime Method (Preferred)

# 1. Start new tunnel alongside old
sudo systemctl start cloudflared-new.service

# 2. Verify new tunnel is healthy
curl -s http://127.0.0.1:9091/ready  # New tunnel metrics port

# 3. Update DNS CNAMEs to point to new tunnel
# (Already done in Step 3, propagation takes ~30s with Cloudflare proxy)

# 4. Monitor traffic shift
watch -n5 'curl -s http://127.0.0.1:9090/metrics | grep requests'
watch -n5 'curl -s http://127.0.0.1:9091/metrics | grep requests'

# 5. Once old tunnel shows zero traffic, proceed to deletion

Maintenance Window Method

# 1. Stop old tunnel
sudo systemctl stop cloudflared.service

# 2. Update config to new tunnel
sudo cp /etc/cloudflared/config-new.yml /etc/cloudflared/config.yml

# 3. Start service
sudo systemctl start cloudflared.service

# 4. Verify connectivity
cloudflared tunnel info <NEW_TUNNEL_ID>
curl -I https://api.vaultmesh.org

---

IV. CITRINITAS — Purification

Delete Old Tunnel

Warning: Only proceed after verifying new tunnel is fully operational.

# 1. Final verification - old tunnel should have zero active connections
cloudflared tunnel info <OLD_TUNNEL_ID>

# 2. Remove DNS routes from old tunnel (if any remain)
cloudflared tunnel route dns delete <OLD_TUNNEL_ID> <hostname>

# 3. Delete the tunnel
cloudflared tunnel delete <OLD_TUNNEL_ID>

# 4. Securely destroy old credentials
sudo shred -vfz -n 5 /etc/cloudflared/<OLD_TUNNEL_ID>.json
sudo rm /etc/cloudflared/<OLD_TUNNEL_ID>.json

Clean Up Local Artifacts

# Remove old credential backups
find /var/lib/vaultmesh/backups -name "*<OLD_TUNNEL_ID>*" -exec shred -vfz {} \;

# Clear any cached tunnel state
rm -rf ~/.cloudflared/connectors/<OLD_TUNNEL_ID>

# Update Terraform state
cd ~/Desktop/CLOUDFLARE/terraform
terraform state rm cloudflare_tunnel.old_tunnel  # If managed by TF

---

V. RUBEDO — Verification & Sealing

Post-Rotation Verification

#!/bin/bash
# rotation_verification.sh

TUNNEL_ID="<NEW_TUNNEL_ID>"
HOSTNAMES=("api.vaultmesh.org" "dash.vaultmesh.org")

echo "=== Tunnel Rotation Verification ==="
echo "Tunnel ID: $TUNNEL_ID"
echo "Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo ""

# 1. Tunnel status
echo "--- Tunnel Status ---"
cloudflared tunnel info $TUNNEL_ID

# 2. DNS routing
echo ""
echo "--- DNS Routes ---"
cloudflared tunnel route dns list | grep $TUNNEL_ID

# 3. Endpoint connectivity
echo ""
echo "--- Endpoint Tests ---"
for hostname in "${HOSTNAMES[@]}"; do
  status=$(curl -s -o /dev/null -w "%{http_code}" https://$hostname/health 2>/dev/null || echo "FAIL")
  echo "$hostname: $status"
done

# 4. Metrics endpoint
echo ""
echo "--- Metrics Check ---"
curl -s http://127.0.0.1:9090/metrics | grep cloudflared_tunnel | head -5

# 5. Certificate validation
echo ""
echo "--- TLS Verification ---"
for hostname in "${HOSTNAMES[@]}"; do
  echo | openssl s_client -connect $hostname:443 -servername $hostname 2>/dev/null | openssl x509 -noout -dates
done

Emit Rotation Receipt

{
  "receipt_type": "tunnel_rotation",
  "schema_version": "vm_tunnel_rotation_v1",
  "timestamp": "<ISO8601>",
  "rotation_id": "<uuid>",
  "old_tunnel": {
    "id": "<OLD_TUNNEL_ID>",
    "created": "<original_creation_date>",
    "deleted": "<deletion_timestamp>"
  },
  "new_tunnel": {
    "id": "<NEW_TUNNEL_ID>",
    "created": "<creation_timestamp>",
    "hostnames": ["api.vaultmesh.org", "dash.vaultmesh.org"]
  },
  "reason": "scheduled_rotation | incident_response | personnel_change",
  "verification_hash": "<blake3_of_verification_output>",
  "operator_did": "did:vm:operator:<id>",
  "guardian_sign": "<tem_signature>"
}

Anchor the Rotation

# Compute rotation proof
cat rotation_verification.txt rotation_receipt.json | blake3sum > rotation_proof.hash

# Append to ProofChain
echo "{\"type\":\"tunnel_rotation\",\"hash\":\"$(cat rotation_proof.hash | cut -d' ' -f1)\",\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}" \
  >> /var/lib/vaultmesh/proofchain/anchors.jsonl

# Update Terraform state
cd ~/Desktop/CLOUDFLARE/terraform
terraform plan -out=rotation.tfplan
terraform apply rotation.tfplan

---

VI. Automation Script

For scheduled rotations, use this automation wrapper:

#!/bin/bash
# tunnel_rotation_automated.sh
# Run via cron or GitLab CI on schedule

set -euo pipefail

TUNNEL_NAME="$1"
NEW_TUNNEL_NAME="${TUNNEL_NAME}-$(date +%Y%m%d)"
LOG_FILE="/var/log/tunnel_rotation_$(date +%Y%m%d).log"

log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $1" | tee -a "$LOG_FILE"; }

log "Starting rotation for tunnel: $TUNNEL_NAME"

# Get old tunnel ID
OLD_TUNNEL_ID=$(cloudflared tunnel list | grep "$TUNNEL_NAME" | awk '{print $1}')
log "Old tunnel ID: $OLD_TUNNEL_ID"

# Create new tunnel
log "Creating new tunnel: $NEW_TUNNEL_NAME"
cloudflared tunnel create "$NEW_TUNNEL_NAME"
NEW_TUNNEL_ID=$(cloudflared tunnel list | grep "$NEW_TUNNEL_NAME" | awk '{print $1}')
log "New tunnel ID: $NEW_TUNNEL_ID"

# Move credentials
sudo mv ~/.cloudflared/${NEW_TUNNEL_ID}.json /etc/cloudflared/
sudo chmod 600 /etc/cloudflared/${NEW_TUNNEL_ID}.json

# Update config
sudo sed -i "s/$OLD_TUNNEL_ID/$NEW_TUNNEL_ID/g" /etc/cloudflared/config.yml

# Restart service
sudo systemctl restart cloudflared.service
sleep 10

# Verify
if cloudflared tunnel info "$NEW_TUNNEL_ID" | grep -q "HEALTHY"; then
  log "New tunnel is healthy"

  # Delete old tunnel
  cloudflared tunnel delete "$OLD_TUNNEL_ID"
  sudo shred -vfz /etc/cloudflared/${OLD_TUNNEL_ID}.json 2>/dev/null || true

  log "Rotation complete"
else
  log "ERROR: New tunnel not healthy, rolling back"
  sudo sed -i "s/$NEW_TUNNEL_ID/$OLD_TUNNEL_ID/g" /etc/cloudflared/config.yml
  sudo systemctl restart cloudflared.service
  cloudflared tunnel delete "$NEW_TUNNEL_ID"
  exit 1
fi

# Emit receipt
cat > /var/lib/vaultmesh/receipts/rotation_$(date +%Y%m%d).json <<EOF
{
  "receipt_type": "tunnel_rotation",
  "old_tunnel_id": "$OLD_TUNNEL_ID",
  "new_tunnel_id": "$NEW_TUNNEL_ID",
  "timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
  "status": "success"
}
EOF

log "Receipt emitted"

---

VII. The Arteries Renewed

With old credentials destroyed and new pathways verified, the tunnel stands reborn. The mesh breathes through fresh veins, uncontaminated by the past. The rotation is complete, the proof anchored, and the guardian satisfied.

Post-Rotation Checklist

• New tunnel ID documented
• Old tunnel deleted and credentials destroyed
• DNS routes verified pointing to new tunnel
• All endpoints responding correctly
• Metrics flowing from new tunnel
• VaultMesh receipt emitted
• ProofChain anchor created
• Terraform state updated
• Next rotation scheduled (90 days)

---

Document Version: 1.0 Last Updated: Guardian: Tem Rotation Schedule: Every 90 days or upon incident