353 lines
7.7 KiB
Markdown
353 lines
7.7 KiB
Markdown
# CONTROLLED FAILURE DRILL RUNBOOK
|
|
|
|
**Classification:** OPERATIONAL / DRILL
|
|
**Version:** 1.0
|
|
**Date:** December 18, 2025
|
|
|
|
---
|
|
|
|
## Purpose
|
|
|
|
Convert "governance exists" into **governance survives contact with chaos**.
|
|
|
|
## Principles
|
|
|
|
- **No real damage**: dry-run actions, sandbox targets only
|
|
- **Single escalation axis**: one chain at a time
|
|
- **Receipts or it didn't happen**: every transition traceable
|
|
- **Auto-return**: TTL de-escalation must fire and be receipted
|
|
|
|
---
|
|
|
|
## Drill 0: Pre-flight Safety Gates
|
|
|
|
**Duration:** 2 minutes
|
|
**Goal:** Confirm training mode active
|
|
|
|
### Checklist
|
|
|
|
```
|
|
[ ] DRILL_MODE environment variable set
|
|
[ ] Phoenix destructive ops: disabled/dry-run
|
|
[ ] Treasury: training budget isolated (ceiling: 1000 units)
|
|
[ ] Guardian anchoring: tagged DRILL/*
|
|
[ ] OFFSEC: simulated mode
|
|
```
|
|
|
|
### Verification Command
|
|
|
|
```python
|
|
from vaultmesh_mcp.tools import cognitive_context
|
|
|
|
ctx = cognitive_context(include=["health", "treasury"])
|
|
assert ctx["health"]["status"] == "operational"
|
|
# Verify training budget exists
|
|
```
|
|
|
|
### Pass Condition
|
|
|
|
Receipt/log proves `DRILL_MODE = ON`
|
|
|
|
---
|
|
|
|
## Drill 1: False-Positive Threat → Tem → De-escalate
|
|
|
|
**Duration:** 5 minutes
|
|
**Marker:** `DRILL/FP-THREAT/{date}`
|
|
|
|
### Trigger
|
|
|
|
```python
|
|
from vaultmesh_mcp.tools import (
|
|
escalate_on_threat,
|
|
cognitive_decide,
|
|
cognitive_invoke_tem,
|
|
deescalate,
|
|
get_escalation_history,
|
|
EscalationType,
|
|
DeescalationType,
|
|
)
|
|
|
|
# Step 1: Inject synthetic threat
|
|
DRILL_MARKER = "DRILL/FP-THREAT/2025-12-18"
|
|
|
|
result = escalate_on_threat(
|
|
current_profile="operator",
|
|
threat_id=f"thr_{DRILL_MARKER}",
|
|
threat_type="synthetic_drill",
|
|
confidence=0.92
|
|
)
|
|
escalation_id = result["escalation_id"]
|
|
```
|
|
|
|
### Expected Chain
|
|
|
|
| Step | Profile | Action | Receipt Type |
|
|
|------|---------|--------|--------------|
|
|
| 1 | 👁 OBSERVER | Read context | None (read-only) |
|
|
| 2 | ⚙ OPERATOR | Escalation request | `profile_escalation` |
|
|
| 3 | 🛡 GUARDIAN | Decision made | `cognitive_decision` |
|
|
| 4 | 🛡 GUARDIAN | Tem invoked | `tem_invocation` |
|
|
| 5 | ⚙ OPERATOR | TTL de-escalation | `profile_deescalation` |
|
|
| 6 | 👁 OBSERVER | Return to baseline | `profile_deescalation` |
|
|
|
|
### Verification
|
|
|
|
```python
|
|
# Check escalation receipts
|
|
history = get_escalation_history()
|
|
assert any(DRILL_MARKER in str(h) for h in history["history"])
|
|
|
|
# Verify Tem context hash exists
|
|
assert result.get("tem_context_hash") is not None
|
|
|
|
# Verify reversibility
|
|
assert result["reversible"] == True
|
|
```
|
|
|
|
### Pass Conditions
|
|
|
|
- [ ] Every profile transition emits receipt
|
|
- [ ] Tem context hash captured in escalation receipt
|
|
- [ ] Reversibility flag set correctly
|
|
- [ ] De-escalation occurs at TTL
|
|
- [ ] Final state: baseline 👁
|
|
|
|
### Fail Conditions
|
|
|
|
- [ ] Any transition without receipt
|
|
- [ ] Tem invoked without Guardian authority
|
|
- [ ] TTL does not de-escalate
|
|
- [ ] De-escalation not receipted
|
|
|
|
---
|
|
|
|
## Drill 2: Budget Pressure Test
|
|
|
|
**Duration:** 3 minutes
|
|
**Goal:** Prevent unauthorized mutation
|
|
|
|
### Setup
|
|
|
|
```python
|
|
from vaultmesh_mcp.tools import treasury_create_budget, treasury_debit
|
|
|
|
# Create minimal training budget
|
|
treasury_create_budget(
|
|
budget_id="drill-budget-001",
|
|
name="Drill Training Budget",
|
|
allocated=100, # Very low
|
|
currency="DRILL_UNITS"
|
|
)
|
|
```
|
|
|
|
### Trigger
|
|
|
|
```python
|
|
# Attempt to exceed budget
|
|
result = treasury_debit(
|
|
budget_id="drill-budget-001",
|
|
amount=500, # Exceeds allocation
|
|
description="DRILL: Intentional over-budget attempt"
|
|
)
|
|
```
|
|
|
|
### Expected Outcome
|
|
|
|
```python
|
|
assert result.get("error") is not None
|
|
assert "insufficient" in result["error"].lower()
|
|
```
|
|
|
|
### Pass Conditions
|
|
|
|
- [ ] Block occurs before write
|
|
- [ ] Receipt shows: attempted action, requested cost, available balance, denial reason
|
|
- [ ] System state unchanged
|
|
|
|
---
|
|
|
|
## Drill 3: Escalation Abuse Attempt
|
|
|
|
**Duration:** 3 minutes
|
|
**Goal:** Constitution enforcement
|
|
|
|
### Trigger: Skip Levels
|
|
|
|
```python
|
|
from vaultmesh_mcp.tools import escalate, EscalationType
|
|
|
|
# Attempt OPERATOR → PHOENIX (skipping GUARDIAN)
|
|
result = escalate(
|
|
from_profile="operator",
|
|
to_profile="phoenix",
|
|
escalation_type=EscalationType.THREAT_DETECTED,
|
|
)
|
|
```
|
|
|
|
### Expected Outcome
|
|
|
|
```python
|
|
assert result["success"] == False
|
|
assert "No escalation path" in result.get("error", "")
|
|
```
|
|
|
|
### Trigger: Missing Approval
|
|
|
|
```python
|
|
# Attempt GUARDIAN → PHOENIX without approval
|
|
result = escalate(
|
|
from_profile="guardian",
|
|
to_profile="phoenix",
|
|
escalation_type=EscalationType.CRISIS_DECLARED,
|
|
# approved_by intentionally missing
|
|
)
|
|
```
|
|
|
|
### Expected Outcome
|
|
|
|
```python
|
|
assert result["success"] == False
|
|
assert "requires approval" in result.get("error", "")
|
|
```
|
|
|
|
### Pass Conditions
|
|
|
|
- [ ] No profile change occurs
|
|
- [ ] Denial includes which requirement failed
|
|
- [ ] Denial is receipted (if implemented)
|
|
|
|
---
|
|
|
|
## Drill 4: Phoenix Readiness (Non-Destructive)
|
|
|
|
**Duration:** 5 minutes
|
|
**Goal:** Enter Phoenix, validate controls, return
|
|
|
|
### Trigger
|
|
|
|
```python
|
|
from vaultmesh_mcp.tools import escalate_to_phoenix, get_active_escalations
|
|
|
|
# Legitimate Phoenix activation
|
|
result = escalate_to_phoenix(
|
|
reason="DRILL: Phoenix readiness test",
|
|
approved_by="did:vm:sovereign:drill-approver"
|
|
)
|
|
```
|
|
|
|
### Verification
|
|
|
|
```python
|
|
# Phoenix is active
|
|
active = get_active_escalations()
|
|
phoenix_active = any(
|
|
e["to_profile"] == "phoenix"
|
|
for e in active["escalations"]
|
|
)
|
|
assert phoenix_active
|
|
|
|
# Verify TTL is set
|
|
assert result.get("expires_at") is not None
|
|
```
|
|
|
|
### De-escalation Test
|
|
|
|
```python
|
|
import time
|
|
# Wait for TTL or manually de-escalate
|
|
from vaultmesh_mcp.tools import deescalate, DeescalationType
|
|
|
|
deescalate(
|
|
escalation_id=result["escalation_id"],
|
|
deescalation_type=DeescalationType.CRISIS_CONCLUDED,
|
|
reason="DRILL: Phoenix test complete"
|
|
)
|
|
|
|
# Verify return to baseline
|
|
active = get_active_escalations()
|
|
assert active["active_count"] == 0
|
|
```
|
|
|
|
### Pass Conditions
|
|
|
|
- [ ] Phoenix activation receipt generated
|
|
- [ ] Destructive ops blocked in drill mode
|
|
- [ ] TTL de-escalation works from Phoenix
|
|
- [ ] Return to 🛡/⚙/👁 with receipts
|
|
|
|
---
|
|
|
|
## Post-Drill: Artifact Pack Generation
|
|
|
|
### Required Artifacts
|
|
|
|
```python
|
|
from vaultmesh_mcp.tools import (
|
|
get_escalation_history,
|
|
cognitive_audit_trail,
|
|
guardian_status,
|
|
)
|
|
|
|
# 1. Escalation timeline
|
|
escalations = get_escalation_history()
|
|
|
|
# 2. Decision audit
|
|
decisions = cognitive_audit_trail()
|
|
|
|
# 3. Scroll state
|
|
scrolls = guardian_status()
|
|
|
|
# Generate artifact
|
|
artifact = {
|
|
"drill_id": "DRILL-2025-12-18-001",
|
|
"escalations": escalations,
|
|
"decisions": decisions,
|
|
"scroll_roots": scrolls,
|
|
"pass_fail": {
|
|
"drill_0": None, # Fill after each drill
|
|
"drill_1": None,
|
|
"drill_2": None,
|
|
"drill_3": None,
|
|
"drill_4": None,
|
|
}
|
|
}
|
|
```
|
|
|
|
### Expected Artifact Contents
|
|
|
|
| Field | Description |
|
|
|-------|-------------|
|
|
| Timeline | (who) (what) (why) (authority) (cost) (proof hashes) (reversibility) |
|
|
| Denials | List of denials + constitutional rule enforced |
|
|
| Baseline Proof | Final state normal, budgets intact, no latent elevation |
|
|
|
|
---
|
|
|
|
## Execution Order
|
|
|
|
**Recommended sequence for maximum signal, minimal risk:**
|
|
|
|
```
|
|
Drill 0 (gates) → Drill 1 (threat) → Drill 3 (abuse) → Drill 2 (budget) → Drill 4 (phoenix)
|
|
```
|
|
|
|
---
|
|
|
|
## Quick Reference: Expected Receipt Types
|
|
|
|
| Event | Receipt Type | Scroll |
|
|
|-------|--------------|--------|
|
|
| Profile escalation | `profile_escalation` | identity |
|
|
| Profile de-escalation | `profile_deescalation` | identity |
|
|
| Cognitive decision | `cognitive_decision` | cognitive |
|
|
| Tem invocation | `tem_invocation` | cognitive |
|
|
| Budget denial | `treasury_denial` | treasury |
|
|
| Auth failure | `auth_failure` | identity |
|
|
|
|
---
|
|
|
|
*Drill complete when all artifacts collected and pass/fail documented.*
|
|
|
|
🜄 **Solve et Coagula**
|