vm-cloudflare/TUNNEL-HARDENING.md

# Cloudflare Tunnel Hardening Guide

## Purpose
Security hardening guide for `cloudflared` deployments across VaultMesh and OffSec infrastructure. Ensures tunnels are isolated, credentials are protected, and monitoring is in place.

---

## 1. Secure Installation

### Binary Verification
```bash
# Download official binary
curl -L https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 -o cloudflared

# Verify checksum (get from GitHub releases)
sha256sum cloudflared

# Make executable and move to secure location
chmod +x cloudflared
sudo mv cloudflared /usr/local/bin/
```

### Package Installation (Preferred)
```bash
# Debian/Ubuntu
curl -fsSL https://pkg.cloudflare.com/cloudflare-main.gpg | sudo tee /usr/share/keyrings/cloudflare-main.gpg >/dev/null
echo 'deb [signed-by=/usr/share/keyrings/cloudflare-main.gpg] https://pkg.cloudflare.com/cloudflared jammy main' | sudo tee /etc/apt/sources.list.d/cloudflared.list
sudo apt update && sudo apt install cloudflared
```

---

## 2. Credential Governance

### Credential Storage
```bash
# Create secure directory
sudo mkdir -p /etc/cloudflared
sudo chmod 700 /etc/cloudflared

# Store credentials with root-only access
sudo mv cert.pem /etc/cloudflared/
sudo mv <tunnel-id>.json /etc/cloudflared/
sudo chmod 600 /etc/cloudflared/*
sudo chown root:root /etc/cloudflared/*
```

### Credential Rotation
- **Rotate tunnel credentials every 90 days**
- Delete old tunnel, create new one
- Update systemd service with new credential path
- Emit VaultMesh receipt for rotation event

### Never Do
- [ ] Store credentials in world-readable locations
- [ ] Embed credentials in container images
- [ ] Commit credentials to git
- [ ] Use long-lived tokens without rotation policy

---

## 3. Systemd Service Isolation

### Hardened Service File
```ini
# /etc/systemd/system/cloudflared.service
[Unit]
Description=Cloudflare Tunnel
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/local/bin/cloudflared tunnel --config /etc/cloudflared/config.yml run
Restart=on-failure
RestartSec=5

# Security Hardening
User=cloudflared
Group=cloudflared
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadOnlyPaths=/
ReadWritePaths=/var/log/cloudflared
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictSUIDSGID=yes
RestrictNamespaces=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
RestrictRealtime=yes
SystemCallFilter=@system-service
SystemCallArchitectures=native

[Install]
WantedBy=multi-user.target
```

### Create Service User
```bash
sudo useradd -r -s /usr/sbin/nologin cloudflared
sudo chown -R cloudflared:cloudflared /etc/cloudflared
sudo mkdir -p /var/log/cloudflared
sudo chown cloudflared:cloudflared /var/log/cloudflared
```

---

## 4. Configuration Hardening

### Minimal Config (`/etc/cloudflared/config.yml`)
```yaml
tunnel: <TUNNEL_ID>
credentials-file: /etc/cloudflared/<TUNNEL_ID>.json

# Metrics for monitoring
metrics: 127.0.0.1:9090

# Ingress rules - explicit deny-by-default
ingress:
  - hostname: app.vaultmesh.org
    service: http://127.0.0.1:8080
    originRequest:
      noTLSVerify: false
      connectTimeout: 10s

  - hostname: api.vaultmesh.org
    service: http://127.0.0.1:8081
    originRequest:
      httpHostHeader: api.internal

  # Catch-all: deny everything else
  - service: http_status:404
```

### Security Settings
- **Always set catch-all to 404** - no accidental exposure
- **Use localhost bindings** - origins never exposed publicly
- **Enable TLS verification** - don't disable unless absolutely necessary
- **Set connection timeouts** - prevent resource exhaustion

---

## 5. Origin Server Lockdown

### Firewall Rules
```bash
# Allow only localhost connections to origin services
sudo iptables -A INPUT -p tcp --dport 8080 -s 127.0.0.1 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 8080 -j DROP

# Or with UFW
sudo ufw allow from 127.0.0.1 to any port 8080
sudo ufw deny 8080
```

### No Public Ports
- Origin servers should have **zero public ports**
- All traffic flows through Cloudflare Tunnel
- If SSH needed, use Cloudflare Access for SSH

---

## 6. Monitoring & Alerting

### Metrics Endpoint
```bash
# Enable in config.yml
metrics: 127.0.0.1:9090

# Scrape with Prometheus
curl http://127.0.0.1:9090/metrics
```

### Key Metrics to Monitor
- `cloudflared_tunnel_total_requests` - request volume
- `cloudflared_tunnel_request_errors` - error rate
- `cloudflared_tunnel_concurrent_requests_per_tunnel` - load
- `cloudflared_tunnel_response_by_code` - HTTP status distribution
- `cloudflared_tunnel_server_locations` - edge connectivity

### Alert Conditions
- [ ] Tunnel disconnected > 1 minute
- [ ] Error rate > 5%
- [ ] Connection to 0 edge servers
- [ ] Credential expiry approaching (30 days)

### Log Forwarding
```bash
# Send logs to syslog/SIEM
cloudflared tunnel --loglevel info --logfile /var/log/cloudflared/tunnel.log run
```

---

## 7. VaultMesh Integration

### Receipt Hooks
Every tunnel operation should emit a VaultMesh receipt:

```yaml
# Tunnel events to capture
- tunnel_created
- tunnel_deleted
- credential_rotated
- config_updated
- service_added
- service_removed
```

### Snapshot Anchoring
```bash
# Weekly config snapshot
cloudflared tunnel info <TUNNEL_ID> > /var/lib/vaultmesh/snapshots/tunnel-$(date +%Y%m%d).json

# Hash and anchor
blake3sum /var/lib/vaultmesh/snapshots/tunnel-*.json >> /var/lib/vaultmesh/anchors/tunnel-hashes.log
```

### Audit Trail
- All tunnel changes logged with timestamp + actor
- Changes require dual approval for production tunnels
- Emergency access via break-glass procedure (logged separately)

---

## 8. Multi-Tunnel Architecture

### Per-Service Tunnels
For OffSec cluster, use dedicated tunnels:
- `tunnel-vaultmesh-core` → Core API
- `tunnel-vaultmesh-guardian` → Guardian services
- `tunnel-offsec-web` → Public OffSec sites
- `tunnel-offsec-internal` → Internal tools

### Benefits
- Blast radius containment
- Independent credential rotation
- Granular Access policies per tunnel

---

## 9. Security Checklist

### Installation
- [ ] Binary verified via checksum
- [ ] Installed from official package repo
- [ ] Running as non-root user

### Credentials
- [ ] Stored in /etc/cloudflared with 600 permissions
- [ ] Owned by root or service user only
- [ ] Rotation schedule documented (90 days)
- [ ] No credentials in git/images

### Service
- [ ] Systemd hardening directives applied
- [ ] NoNewPrivileges=yes
- [ ] PrivateTmp=yes
- [ ] ProtectSystem=strict

### Configuration
- [ ] Catch-all ingress returns 404
- [ ] All services bound to localhost
- [ ] TLS verification enabled
- [ ] Metrics endpoint enabled

### Monitoring
- [ ] Prometheus scraping metrics
- [ ] Alerts for disconnection/errors
- [ ] Logs forwarded to SIEM
- [ ] VaultMesh receipts emitted

### Network
- [ ] Origin has no public ports
- [ ] Firewall blocks non-localhost to origin ports
- [ ] Only Cloudflare Tunnel provides ingress

---

## 10. Emergency Procedures

### Tunnel Compromise Response
1. Immediately delete compromised tunnel in CF dashboard
2. Revoke associated credentials
3. Create new tunnel with fresh credentials
4. Update config and restart service
5. Emit incident receipt in VaultMesh
6. Review Access logs for unauthorized access

### Credential Leak Response
1. Rotate credentials immediately
2. Review Cloudflare audit logs
3. Check for unauthorized tunnel connections
4. Update all systems with new credentials
5. Document in incident report