Initial commit: Cloudflare infrastructure with WAF Intelligence
- Complete Cloudflare Terraform configuration (DNS, WAF, tunnels, access) - WAF Intelligence MCP server with threat analysis and ML classification - GitOps automation with PR workflows and drift detection - Observatory monitoring stack with Prometheus/Grafana - IDE operator rules for governed development - Security playbooks and compliance frameworks - Autonomous remediation and state reconciliation
This commit is contained in:
Vault Sovereign
343
gitops/README.md
Normal file
343
gitops/README.md
Normal file
@@ -0,0 +1,343 @@
|
||||
# Phase 6 - GitOps PR Workflows
|
||||
|
||||
Cloudflare Mesh Observatory - Automated Drift Remediation & Plan Comments
|
||||
|
||||
## Overview
|
||||
|
||||
Phase 6 completes the observability feedback loop by converting alerts and drift
|
||||
detection into actionable Merge Requests.
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
||||
│ Observatory │────▶│ Alerts │────▶│ GitOps │
|
||||
│ (Phase 5A) │ │ (Phase 5B) │ │ (Phase 6) │
|
||||
└─────────────┘ └─────────────┘ └─────────────┘
|
||||
│ │ │
|
||||
│ │ ▼
|
||||
│ │ ┌─────────────┐
|
||||
│ │ │ Drift PR │
|
||||
│ │ │ Created │
|
||||
│ │ └─────────────┘
|
||||
│ │ │
|
||||
│ │ ▼
|
||||
│ │ ┌─────────────┐
|
||||
│ └───────────▶│ Review & │
|
||||
│ │ Merge │
|
||||
│ └─────────────┘
|
||||
│ │
|
||||
└───────────────────────────────────────┘
|
||||
Terraform Apply
|
||||
```
|
||||
|
||||
## Components
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `config.yml` | GitOps configuration, risk classification, compliance mapping |
|
||||
| `plan_summarizer.py` | Parses terraform plan JSON, scores risk, generates markdown |
|
||||
| `drift_pr_bot.py` | Creates drift remediation MRs in GitLab/GitHub |
|
||||
| `ci_plan_comment.py` | Posts plan summaries as MR comments |
|
||||
| `webhook_receiver.py` | Receives Alertmanager webhooks, triggers pipelines |
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Configure Environment
|
||||
|
||||
```bash
|
||||
# Copy and edit config
|
||||
cd ~/Desktop/CLOUDFLARE/gitops
|
||||
cp config.yml config.local.yml # optional local override
|
||||
|
||||
# Set environment variables
|
||||
export GITLAB_TOKEN="glpat-xxxx"
|
||||
export GITLAB_PROJECT_ID="12345678"
|
||||
export SLACK_WEBHOOK_URL="https://hooks.slack.com/..."
|
||||
```
|
||||
|
||||
### 2. Test Plan Summarizer
|
||||
|
||||
```bash
|
||||
# Generate a terraform plan first
|
||||
cd ../terraform
|
||||
terraform init
|
||||
terraform plan -out=plan.tfplan
|
||||
|
||||
# Run summarizer
|
||||
cd ../gitops
|
||||
python3 plan_summarizer.py --format markdown
|
||||
python3 plan_summarizer.py --format json
|
||||
```
|
||||
|
||||
### 3. Test Drift PR Bot (Dry Run)
|
||||
|
||||
```bash
|
||||
python3 drift_pr_bot.py --dry-run
|
||||
```
|
||||
|
||||
### 4. Start Webhook Receiver (Optional)
|
||||
|
||||
```bash
|
||||
python3 webhook_receiver.py --port 8080
|
||||
# POST to http://localhost:8080/webhook/alert
|
||||
```
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
### Risk Classification
|
||||
|
||||
The `config.yml` maps Cloudflare resources to risk levels:
|
||||
|
||||
```yaml
|
||||
risk:
|
||||
dns:
|
||||
resource_types:
|
||||
- "cloudflare_record"
|
||||
- "cloudflare_zone"
|
||||
base_risk: "high"
|
||||
|
||||
waf:
|
||||
resource_types:
|
||||
- "cloudflare_waf_rule"
|
||||
- "cloudflare_firewall_rule"
|
||||
base_risk: "high"
|
||||
|
||||
actions:
|
||||
create:
|
||||
modifier: 0 # Neutral
|
||||
update:
|
||||
modifier: 1 # +1 level
|
||||
delete:
|
||||
modifier: 2 # +2 levels (always dangerous)
|
||||
```
|
||||
|
||||
### Compliance Frameworks
|
||||
|
||||
Map resources/actions to compliance frameworks:
|
||||
|
||||
```yaml
|
||||
compliance:
|
||||
frameworks:
|
||||
- name: "SOC2"
|
||||
triggers:
|
||||
- resource_types: ["cloudflare_zone_settings_override"]
|
||||
fields: ["ssl", "always_use_https"]
|
||||
- resource_types: ["cloudflare_waf_rule"]
|
||||
actions: ["delete"]
|
||||
|
||||
- name: "PCI-DSS"
|
||||
triggers:
|
||||
- resource_types: ["cloudflare_zone_settings_override"]
|
||||
fields: ["min_tls_version"]
|
||||
```
|
||||
|
||||
### Drift PR Settings
|
||||
|
||||
```yaml
|
||||
drift_pr:
|
||||
branch_prefix: "drift/remediation-"
|
||||
title_prefix: "Drift Remediation"
|
||||
labels:
|
||||
- "drift"
|
||||
- "terraform"
|
||||
|
||||
# Auto-assign reviewers by category
|
||||
reviewer_mapping:
|
||||
dns: ["dns-team"]
|
||||
waf: ["security-team"]
|
||||
tunnels: ["infra-team"]
|
||||
```
|
||||
|
||||
## GitLab CI Integration
|
||||
|
||||
Three jobs are added to `.gitlab-ci.yml`:
|
||||
|
||||
### 1. Plan Comment on MRs
|
||||
|
||||
```yaml
|
||||
gitops:plan_comment:
|
||||
stage: gitops
|
||||
script:
|
||||
- python3 gitops/ci_plan_comment.py
|
||||
rules:
|
||||
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
|
||||
```
|
||||
|
||||
Posts a rich markdown comment showing:
|
||||
- Overall risk level
|
||||
- Action breakdown (create/update/delete)
|
||||
- Affected zones
|
||||
- Compliance flags
|
||||
- Resource change table
|
||||
|
||||
### 2. Drift Remediation
|
||||
|
||||
```yaml
|
||||
gitops:drift_remediation:
|
||||
stage: gitops
|
||||
script:
|
||||
- python3 gitops/drift_pr_bot.py
|
||||
rules:
|
||||
- if: $CI_PIPELINE_SOURCE == "schedule" && $GITOPS_DRIFT_CHECK == "true"
|
||||
- if: $CI_PIPELINE_SOURCE == "trigger" && $GITOPS_TRIGGER_SOURCE == "alert"
|
||||
```
|
||||
|
||||
Triggered by:
|
||||
- Scheduled pipelines (daily drift check)
|
||||
- Alertmanager webhooks (alert-triggered)
|
||||
|
||||
### 3. Risk Gate
|
||||
|
||||
```yaml
|
||||
gitops:risk_gate:
|
||||
stage: gitops
|
||||
script:
|
||||
- |
|
||||
RISK=$(python3 plan_summarizer.py --format json | ...)
|
||||
if [ "$RISK" = "CRITICAL" ]; then
|
||||
exit 1
|
||||
fi
|
||||
allow_failure: true
|
||||
```
|
||||
|
||||
Blocks auto-merge for CRITICAL risk changes.
|
||||
|
||||
## Alertmanager Integration
|
||||
|
||||
### Add Webhook Receiver
|
||||
|
||||
Add to `observatory/alertmanager/alertmanager.yml`:
|
||||
|
||||
```yaml
|
||||
receivers:
|
||||
- name: 'gitops-webhook'
|
||||
webhook_configs:
|
||||
- url: 'http://gitops-webhook:8080/webhook/alert'
|
||||
send_resolved: false
|
||||
```
|
||||
|
||||
### Route Drift Alerts
|
||||
|
||||
```yaml
|
||||
route:
|
||||
routes:
|
||||
- match:
|
||||
alertname: DNSDriftDetected
|
||||
receiver: 'gitops-webhook'
|
||||
continue: true
|
||||
|
||||
- match:
|
||||
alertname: WAFRuleMissing
|
||||
receiver: 'gitops-webhook'
|
||||
continue: true
|
||||
```
|
||||
|
||||
## Output Examples
|
||||
|
||||
### MR Comment
|
||||
|
||||
```markdown
|
||||
## 🟠 Terraform Plan Summary
|
||||
|
||||
**Overall Risk:** 🟠 **HIGH**
|
||||
**Total Changes:** `5`
|
||||
|
||||
**Actions:** create=2, update=2, delete=1
|
||||
|
||||
**By Category:**
|
||||
- dns: 3
|
||||
- waf: 2
|
||||
|
||||
**Affected Zones:** `example.com`, `staging.example.com`
|
||||
|
||||
**Compliance Impact:**
|
||||
- ⚠️ SOC2
|
||||
- ⚠️ PCI-DSS
|
||||
|
||||
### Resource Changes
|
||||
|
||||
| Resource | Actions | Risk | Compliance |
|
||||
|----------|---------|------|------------|
|
||||
| `cloudflare_record.api` | `delete` | **CRITICAL** | SOC2 |
|
||||
| `cloudflare_waf_rule.sqli` | `update` | **HIGH** | PCI-DSS |
|
||||
...
|
||||
```
|
||||
|
||||
### JSON Output
|
||||
|
||||
```json
|
||||
{
|
||||
"total_changes": 5,
|
||||
"overall_risk": "HIGH",
|
||||
"by_action": {"create": 2, "update": 2, "delete": 1},
|
||||
"by_risk": {"LOW": 1, "MEDIUM": 1, "HIGH": 2, "CRITICAL": 1},
|
||||
"by_category": {"dns": 3, "waf": 2},
|
||||
"affected_zones": ["example.com", "staging.example.com"],
|
||||
"compliance_violations": ["SOC2", "PCI-DSS"],
|
||||
"changes": [...]
|
||||
}
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Description |
|
||||
|----------|----------|-------------|
|
||||
| `GITLAB_TOKEN` | Yes | GitLab API token with `api` scope |
|
||||
| `GITLAB_PROJECT_ID` | Yes | Target project ID |
|
||||
| `GITLAB_BASE_URL` | No | GitLab instance URL (default: gitlab.com) |
|
||||
| `GITLAB_TRIGGER_TOKEN` | No | For pipeline triggers from webhooks |
|
||||
| `SLACK_WEBHOOK_URL` | No | Slack notifications |
|
||||
| `GITOPS_DRY_RUN` | No | Set `true` to skip actual PR creation |
|
||||
| `WEBHOOK_SECRET` | No | HMAC secret for webhook verification |
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **Token Scope**: Use minimal GitLab token scope (`api` for MR creation)
|
||||
2. **Webhook Security**: Set `WEBHOOK_SECRET` for signature verification
|
||||
3. **Review Before Merge**: Always review auto-generated PRs
|
||||
4. **Compliance Blocking**: Consider `block_on_violation: true` for strict mode
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Plan Summarizer Fails
|
||||
|
||||
```bash
|
||||
# Check terraform plan exists
|
||||
ls -la terraform/plan.tfplan
|
||||
|
||||
# Run terraform show manually
|
||||
cd terraform
|
||||
terraform show -json plan.tfplan | head -100
|
||||
```
|
||||
|
||||
### MR Comment Not Posted
|
||||
|
||||
```bash
|
||||
# Check CI variables are set
|
||||
echo $GITLAB_TOKEN
|
||||
echo $CI_MERGE_REQUEST_IID
|
||||
|
||||
# Run comment script manually
|
||||
python3 ci_plan_comment.py --dry-run
|
||||
```
|
||||
|
||||
### Webhook Not Triggering
|
||||
|
||||
```bash
|
||||
# Check webhook receiver logs
|
||||
curl -X POST http://localhost:8080/webhook/alert \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"alerts":[{"labels":{"alertname":"DNSDriftDetected"}}]}'
|
||||
|
||||
# Check Alertmanager config
|
||||
amtool config show
|
||||
```
|
||||
|
||||
## Next Phases
|
||||
|
||||
- **Phase 7 (WAF Intelligence)**: ML-lite analysis of attack patterns
|
||||
- **Phase 8 (Zero Trust Auditor)**: Identity policy compliance
|
||||
- **Phase 9 (VaultMesh Integration)**: ProofChain anchoring
|
||||
|
||||
---
|
||||
|
||||
*Phase 6 GitOps - Cloudflare Mesh Observatory*
|
||||
Reference in New Issue
Block a user