Initial commit: Cloudflare infrastructure with WAF Intelligence
- Complete Cloudflare Terraform configuration (DNS, WAF, tunnels, access) - WAF Intelligence MCP server with threat analysis and ML classification - GitOps automation with PR workflows and drift detection - Observatory monitoring stack with Prometheus/Grafana - IDE operator rules for governed development - Security playbooks and compliance frameworks - Autonomous remediation and state reconciliation
This commit is contained in:
Vault Sovereign
396
playbooks/TUNNEL-ROTATION-PROTOCOL.md
Normal file
396
playbooks/TUNNEL-ROTATION-PROTOCOL.md
Normal file
@@ -0,0 +1,396 @@
|
||||
# Tunnel Rotation Protocol
|
||||
|
||||
**Incident Response** | Governed by [RED-BOOK.md](../RED-BOOK.md)
|
||||
|
||||
## The Arteries Must Shed Their Old Keys and Be Reborn
|
||||
|
||||
*Cloudflare Tunnels are the veins through which the mesh breathes. When credentials age or suspicion arises, the tunnels must be dissolved and reformed — a controlled death and resurrection that preserves continuity while eliminating compromise vectors.*
|
||||
|
||||
---
|
||||
|
||||
## I. When to Rotate
|
||||
|
||||
### Scheduled Rotation (Prophylactic)
|
||||
|
||||
| Trigger | Interval | Priority |
|
||||
|---------|----------|----------|
|
||||
| Standard credential hygiene | Every 90 days | NORMAL |
|
||||
| After personnel change | Within 24 hours | HIGH |
|
||||
| Compliance audit requirement | As specified | NORMAL |
|
||||
| Post-incident (any severity) | Immediately | CRITICAL |
|
||||
|
||||
### Emergency Rotation (Reactive)
|
||||
|
||||
| Trigger | Response Time |
|
||||
|---------|---------------|
|
||||
| Credential exposure suspected | < 1 hour |
|
||||
| Tunnel behaving anomalously | < 2 hours |
|
||||
| Unauthorized connection detected | Immediate |
|
||||
| Origin server compromised | Immediate |
|
||||
| Security advisory from Cloudflare | < 24 hours |
|
||||
|
||||
---
|
||||
|
||||
## II. NIGREDO — Preparation
|
||||
|
||||
### Pre-Rotation Checklist
|
||||
|
||||
Before beginning rotation:
|
||||
|
||||
- [ ] Identify all tunnels requiring rotation
|
||||
- [ ] Document current tunnel configurations
|
||||
- [ ] Verify backup ingress path (if available)
|
||||
- [ ] Notify dependent teams of maintenance window
|
||||
- [ ] Prepare new tunnel names and secrets
|
||||
- [ ] Ensure Terraform state is current
|
||||
|
||||
### Inventory Current State
|
||||
|
||||
```bash
|
||||
# List all tunnels
|
||||
cloudflared tunnel list
|
||||
|
||||
# Export tunnel info
|
||||
for tunnel_id in $(cloudflared tunnel list | tail -n +2 | awk '{print $1}'); do
|
||||
cloudflared tunnel info $tunnel_id > /tmp/tunnel_${tunnel_id}_info.txt
|
||||
done
|
||||
|
||||
# Capture current routes
|
||||
cloudflared tunnel route dns list
|
||||
|
||||
# Hash for audit trail
|
||||
cat /tmp/tunnel_*.txt | blake3sum > pre_rotation_state.hash
|
||||
```
|
||||
|
||||
### Generate New Secrets
|
||||
|
||||
```bash
|
||||
# Generate cryptographically secure tunnel secrets
|
||||
NEW_SECRET_VAULTMESH=$(openssl rand -base64 32)
|
||||
NEW_SECRET_OFFSEC=$(openssl rand -base64 32)
|
||||
|
||||
# Store securely (example: HashiCorp Vault)
|
||||
vault kv put secret/cloudflare/tunnels \
|
||||
vaultmesh_secret="$NEW_SECRET_VAULTMESH" \
|
||||
offsec_secret="$NEW_SECRET_OFFSEC"
|
||||
|
||||
# Or for local encrypted storage
|
||||
echo "$NEW_SECRET_VAULTMESH" | gpg --encrypt -r guardian@vaultmesh.org > vaultmesh_tunnel_secret.gpg
|
||||
echo "$NEW_SECRET_OFFSEC" | gpg --encrypt -r guardian@vaultmesh.org > offsec_tunnel_secret.gpg
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## III. ALBEDO — Dissolution
|
||||
|
||||
### Step 1: Create New Tunnel (Before Destroying Old)
|
||||
|
||||
```bash
|
||||
# Create new tunnel with fresh credentials
|
||||
cloudflared tunnel create vaultmesh-tunnel-$(date +%Y%m%d)
|
||||
|
||||
# This generates:
|
||||
# - New tunnel ID
|
||||
# - New credentials JSON in ~/.cloudflared/
|
||||
|
||||
# Move credentials to secure location
|
||||
sudo mv ~/.cloudflared/<new_tunnel_id>.json /etc/cloudflared/
|
||||
sudo chmod 600 /etc/cloudflared/<new_tunnel_id>.json
|
||||
sudo chown cloudflared:cloudflared /etc/cloudflared/<new_tunnel_id>.json
|
||||
```
|
||||
|
||||
### Step 2: Configure New Tunnel
|
||||
|
||||
Update `/etc/cloudflared/config.yml`:
|
||||
|
||||
```yaml
|
||||
tunnel: <NEW_TUNNEL_ID>
|
||||
credentials-file: /etc/cloudflared/<NEW_TUNNEL_ID>.json
|
||||
|
||||
metrics: 127.0.0.1:9090
|
||||
|
||||
ingress:
|
||||
- hostname: api.vaultmesh.org
|
||||
service: http://localhost:8080
|
||||
originRequest:
|
||||
connectTimeout: 10s
|
||||
noTLSVerify: false
|
||||
|
||||
- hostname: dash.vaultmesh.org
|
||||
service: http://localhost:3000
|
||||
|
||||
- service: http_status:404
|
||||
```
|
||||
|
||||
### Step 3: Update DNS Routes
|
||||
|
||||
```bash
|
||||
# Route hostnames to new tunnel
|
||||
cloudflared tunnel route dns <NEW_TUNNEL_ID> api.vaultmesh.org
|
||||
cloudflared tunnel route dns <NEW_TUNNEL_ID> dash.vaultmesh.org
|
||||
|
||||
# Verify routing
|
||||
cloudflared tunnel route dns list | grep <NEW_TUNNEL_ID>
|
||||
```
|
||||
|
||||
### Step 4: Transition Traffic
|
||||
|
||||
#### Zero-Downtime Method (Preferred)
|
||||
|
||||
```bash
|
||||
# 1. Start new tunnel alongside old
|
||||
sudo systemctl start cloudflared-new.service
|
||||
|
||||
# 2. Verify new tunnel is healthy
|
||||
curl -s http://127.0.0.1:9091/ready # New tunnel metrics port
|
||||
|
||||
# 3. Update DNS CNAMEs to point to new tunnel
|
||||
# (Already done in Step 3, propagation takes ~30s with Cloudflare proxy)
|
||||
|
||||
# 4. Monitor traffic shift
|
||||
watch -n5 'curl -s http://127.0.0.1:9090/metrics | grep requests'
|
||||
watch -n5 'curl -s http://127.0.0.1:9091/metrics | grep requests'
|
||||
|
||||
# 5. Once old tunnel shows zero traffic, proceed to deletion
|
||||
```
|
||||
|
||||
#### Maintenance Window Method
|
||||
|
||||
```bash
|
||||
# 1. Stop old tunnel
|
||||
sudo systemctl stop cloudflared.service
|
||||
|
||||
# 2. Update config to new tunnel
|
||||
sudo cp /etc/cloudflared/config-new.yml /etc/cloudflared/config.yml
|
||||
|
||||
# 3. Start service
|
||||
sudo systemctl start cloudflared.service
|
||||
|
||||
# 4. Verify connectivity
|
||||
cloudflared tunnel info <NEW_TUNNEL_ID>
|
||||
curl -I https://api.vaultmesh.org
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## IV. CITRINITAS — Purification
|
||||
|
||||
### Delete Old Tunnel
|
||||
|
||||
**Warning**: Only proceed after verifying new tunnel is fully operational.
|
||||
|
||||
```bash
|
||||
# 1. Final verification - old tunnel should have zero active connections
|
||||
cloudflared tunnel info <OLD_TUNNEL_ID>
|
||||
|
||||
# 2. Remove DNS routes from old tunnel (if any remain)
|
||||
cloudflared tunnel route dns delete <OLD_TUNNEL_ID> <hostname>
|
||||
|
||||
# 3. Delete the tunnel
|
||||
cloudflared tunnel delete <OLD_TUNNEL_ID>
|
||||
|
||||
# 4. Securely destroy old credentials
|
||||
sudo shred -vfz -n 5 /etc/cloudflared/<OLD_TUNNEL_ID>.json
|
||||
sudo rm /etc/cloudflared/<OLD_TUNNEL_ID>.json
|
||||
```
|
||||
|
||||
### Clean Up Local Artifacts
|
||||
|
||||
```bash
|
||||
# Remove old credential backups
|
||||
find /var/lib/vaultmesh/backups -name "*<OLD_TUNNEL_ID>*" -exec shred -vfz {} \;
|
||||
|
||||
# Clear any cached tunnel state
|
||||
rm -rf ~/.cloudflared/connectors/<OLD_TUNNEL_ID>
|
||||
|
||||
# Update Terraform state
|
||||
cd ~/Desktop/CLOUDFLARE/terraform
|
||||
terraform state rm cloudflare_tunnel.old_tunnel # If managed by TF
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## V. RUBEDO — Verification & Sealing
|
||||
|
||||
### Post-Rotation Verification
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# rotation_verification.sh
|
||||
|
||||
TUNNEL_ID="<NEW_TUNNEL_ID>"
|
||||
HOSTNAMES=("api.vaultmesh.org" "dash.vaultmesh.org")
|
||||
|
||||
echo "=== Tunnel Rotation Verification ==="
|
||||
echo "Tunnel ID: $TUNNEL_ID"
|
||||
echo "Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
echo ""
|
||||
|
||||
# 1. Tunnel status
|
||||
echo "--- Tunnel Status ---"
|
||||
cloudflared tunnel info $TUNNEL_ID
|
||||
|
||||
# 2. DNS routing
|
||||
echo ""
|
||||
echo "--- DNS Routes ---"
|
||||
cloudflared tunnel route dns list | grep $TUNNEL_ID
|
||||
|
||||
# 3. Endpoint connectivity
|
||||
echo ""
|
||||
echo "--- Endpoint Tests ---"
|
||||
for hostname in "${HOSTNAMES[@]}"; do
|
||||
status=$(curl -s -o /dev/null -w "%{http_code}" https://$hostname/health 2>/dev/null || echo "FAIL")
|
||||
echo "$hostname: $status"
|
||||
done
|
||||
|
||||
# 4. Metrics endpoint
|
||||
echo ""
|
||||
echo "--- Metrics Check ---"
|
||||
curl -s http://127.0.0.1:9090/metrics | grep cloudflared_tunnel | head -5
|
||||
|
||||
# 5. Certificate validation
|
||||
echo ""
|
||||
echo "--- TLS Verification ---"
|
||||
for hostname in "${HOSTNAMES[@]}"; do
|
||||
echo | openssl s_client -connect $hostname:443 -servername $hostname 2>/dev/null | openssl x509 -noout -dates
|
||||
done
|
||||
```
|
||||
|
||||
### Emit Rotation Receipt
|
||||
|
||||
```json
|
||||
{
|
||||
"receipt_type": "tunnel_rotation",
|
||||
"schema_version": "vm_tunnel_rotation_v1",
|
||||
"timestamp": "<ISO8601>",
|
||||
"rotation_id": "<uuid>",
|
||||
"old_tunnel": {
|
||||
"id": "<OLD_TUNNEL_ID>",
|
||||
"created": "<original_creation_date>",
|
||||
"deleted": "<deletion_timestamp>"
|
||||
},
|
||||
"new_tunnel": {
|
||||
"id": "<NEW_TUNNEL_ID>",
|
||||
"created": "<creation_timestamp>",
|
||||
"hostnames": ["api.vaultmesh.org", "dash.vaultmesh.org"]
|
||||
},
|
||||
"reason": "scheduled_rotation | incident_response | personnel_change",
|
||||
"verification_hash": "<blake3_of_verification_output>",
|
||||
"operator_did": "did:vm:operator:<id>",
|
||||
"guardian_sign": "<tem_signature>"
|
||||
}
|
||||
```
|
||||
|
||||
### Anchor the Rotation
|
||||
|
||||
```bash
|
||||
# Compute rotation proof
|
||||
cat rotation_verification.txt rotation_receipt.json | blake3sum > rotation_proof.hash
|
||||
|
||||
# Append to ProofChain
|
||||
echo "{\"type\":\"tunnel_rotation\",\"hash\":\"$(cat rotation_proof.hash | cut -d' ' -f1)\",\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}" \
|
||||
>> /var/lib/vaultmesh/proofchain/anchors.jsonl
|
||||
|
||||
# Update Terraform state
|
||||
cd ~/Desktop/CLOUDFLARE/terraform
|
||||
terraform plan -out=rotation.tfplan
|
||||
terraform apply rotation.tfplan
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## VI. Automation Script
|
||||
|
||||
For scheduled rotations, use this automation wrapper:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# tunnel_rotation_automated.sh
|
||||
# Run via cron or GitLab CI on schedule
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
TUNNEL_NAME="$1"
|
||||
NEW_TUNNEL_NAME="${TUNNEL_NAME}-$(date +%Y%m%d)"
|
||||
LOG_FILE="/var/log/tunnel_rotation_$(date +%Y%m%d).log"
|
||||
|
||||
log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $1" | tee -a "$LOG_FILE"; }
|
||||
|
||||
log "Starting rotation for tunnel: $TUNNEL_NAME"
|
||||
|
||||
# Get old tunnel ID
|
||||
OLD_TUNNEL_ID=$(cloudflared tunnel list | grep "$TUNNEL_NAME" | awk '{print $1}')
|
||||
log "Old tunnel ID: $OLD_TUNNEL_ID"
|
||||
|
||||
# Create new tunnel
|
||||
log "Creating new tunnel: $NEW_TUNNEL_NAME"
|
||||
cloudflared tunnel create "$NEW_TUNNEL_NAME"
|
||||
NEW_TUNNEL_ID=$(cloudflared tunnel list | grep "$NEW_TUNNEL_NAME" | awk '{print $1}')
|
||||
log "New tunnel ID: $NEW_TUNNEL_ID"
|
||||
|
||||
# Move credentials
|
||||
sudo mv ~/.cloudflared/${NEW_TUNNEL_ID}.json /etc/cloudflared/
|
||||
sudo chmod 600 /etc/cloudflared/${NEW_TUNNEL_ID}.json
|
||||
|
||||
# Update config
|
||||
sudo sed -i "s/$OLD_TUNNEL_ID/$NEW_TUNNEL_ID/g" /etc/cloudflared/config.yml
|
||||
|
||||
# Restart service
|
||||
sudo systemctl restart cloudflared.service
|
||||
sleep 10
|
||||
|
||||
# Verify
|
||||
if cloudflared tunnel info "$NEW_TUNNEL_ID" | grep -q "HEALTHY"; then
|
||||
log "New tunnel is healthy"
|
||||
|
||||
# Delete old tunnel
|
||||
cloudflared tunnel delete "$OLD_TUNNEL_ID"
|
||||
sudo shred -vfz /etc/cloudflared/${OLD_TUNNEL_ID}.json 2>/dev/null || true
|
||||
|
||||
log "Rotation complete"
|
||||
else
|
||||
log "ERROR: New tunnel not healthy, rolling back"
|
||||
sudo sed -i "s/$NEW_TUNNEL_ID/$OLD_TUNNEL_ID/g" /etc/cloudflared/config.yml
|
||||
sudo systemctl restart cloudflared.service
|
||||
cloudflared tunnel delete "$NEW_TUNNEL_ID"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Emit receipt
|
||||
cat > /var/lib/vaultmesh/receipts/rotation_$(date +%Y%m%d).json <<EOF
|
||||
{
|
||||
"receipt_type": "tunnel_rotation",
|
||||
"old_tunnel_id": "$OLD_TUNNEL_ID",
|
||||
"new_tunnel_id": "$NEW_TUNNEL_ID",
|
||||
"timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
|
||||
"status": "success"
|
||||
}
|
||||
EOF
|
||||
|
||||
log "Receipt emitted"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## VII. The Arteries Renewed
|
||||
|
||||
*With old credentials destroyed and new pathways verified, the tunnel stands reborn. The mesh breathes through fresh veins, uncontaminated by the past. The rotation is complete, the proof anchored, and the guardian satisfied.*
|
||||
|
||||
### Post-Rotation Checklist
|
||||
|
||||
- [ ] New tunnel ID documented
|
||||
- [ ] Old tunnel deleted and credentials destroyed
|
||||
- [ ] DNS routes verified pointing to new tunnel
|
||||
- [ ] All endpoints responding correctly
|
||||
- [ ] Metrics flowing from new tunnel
|
||||
- [ ] VaultMesh receipt emitted
|
||||
- [ ] ProofChain anchor created
|
||||
- [ ] Terraform state updated
|
||||
- [ ] Next rotation scheduled (90 days)
|
||||
|
||||
---
|
||||
|
||||
**Document Version**: 1.0
|
||||
**Last Updated**: <date>
|
||||
**Guardian**: Tem
|
||||
**Rotation Schedule**: Every 90 days or upon incident
|
||||
Reference in New Issue
Block a user