Initialize repository snapshot

2025-12-27 00:10:32 +00:00
commit 110d644e10
281 changed files with 40331 additions and 0 deletions
--- a/docs/VAULTMESH-ETERNAL-PATTERN.md
+++ b/docs/VAULTMESH-ETERNAL-PATTERN.md
@@ -0,0 +1,507 @@
+# VAULTMESH-ETERNAL-PATTERN.md
+
+**Canonical Design Pattern for All VaultMesh Subsystems**
+
+> *Every serious subsystem in VaultMesh should feel different in flavor, but identical in **shape**.*
+
+This document defines that shared shape — the **Eternal Pattern**.
+
+It is the architectural law that binds Drills, Oracle, Guardian, Treasury, Mesh, and any future module into one Civilization Ledger.
+
+---
+
+## 1. Core Idea (One-Line Contract)
+
+All VaultMesh subsystems follow this arc:
+
+> **Real-world intent → Engine → Structured JSON → Receipt → Scroll → Guardian Anchor**
+
+If a new feature does **not** fit this pattern, it's either:
+
+- not finished yet, or
+- not part of the Ledger core.
+
+---
+
+## 2. Three-Layer VaultMesh Stack
+
+At the highest level, VaultMesh is three stacked layers:
+
+```
+┌───────────────────────────────────────────────┐
+│  L1 — Experience Layer                        │
+│  (Humans & Agents)                            │
+│  • CLI / UI / MCP tools / agents              │
+│  • "Ask a question", "start a drill",         │
+│    "anchor now", "run settlement"             │
+└───────────────────────────────────────────────┘
+                 │
+                 ▼
+┌───────────────────────────────────────────────┐
+│  L2 — Engine Layer                            │
+│  (Domain Engines & Contracts)                 │
+│  • Domain logics: Drills, Oracle, Guardian,   │
+│    Treasury, Mesh, OffSec, etc.               │
+│  • Contracts (plans)                          │
+│  • Runners (state machines)                   │
+│  • State JSON (progress)                      │
+└───────────────────────────────────────────────┘
+                 │
+                 ▼
+┌───────────────────────────────────────────────┐
+│  L3 — Ledger Layer                            │
+│  (Receipts, Scrolls, ProofChain, Anchors)     │
+│  • Receipts in append-only JSONL files        │
+│  • Scrolls per domain (Drills, Compliance,    │
+│    Guardian, Treasury, Mesh, etc.)            │
+│  • Merkle roots (ROOT.<scroll>.txt)           │
+│  • Guardian anchor cycles (local/OTS/chain)   │
+└───────────────────────────────────────────────┘
+```
+
+Everything you build plugs into this stack.
+
+---
+
+## 3. Eternal Pattern — Generic Lifecycle
+
+This is the reusable template for any new subsystem "X".
+
+### 3.1 Experience Layer (L1) — Intent In
+
+**Goal**: Take messy human/agent intent and normalize it.
+
+**Typical surfaces**:
+- CLI (`vm-drills`, `vm-oracle`, `guardian`, `vm-treasury`, etc.)
+- MCP tools (e.g. `oracle_answer`)
+- Web / TUI / dashboard
+- Automation hooks (cron, CI, schedulers)
+
+**Typical inputs**:
+- "Run a security drill for IoT ↔ OT"
+- "Are we compliant with Annex IV today?"
+- "Anchor this ProofChain root"
+- "Reconcile treasury balances between nodes"
+- "Apply this mesh topology change"
+
+**L1 should**:
+- Capture the raw intent
+- Attach minimal context (who, when, where)
+- Hand it off to the appropriate Engine in L2
+
+---
+
+### 3.2 Engine Layer (L2) — Plan and Execute
+
+Every Engine follows the same internal shape:
+
+#### Step 1 — Plan → `contract.json`
+
+An Engine takes the intent and creates a **contract**:
+
+`contract.json` (or equivalent JSON struct) contains:
+- `id`: unique contract / drill / run id
+- `title`: short human title
+- `severity` / `priority` (optional, domain-specific)
+- `stages[]` / `steps[]`:
+  - ordered id, skill / module, workflow, role, objective
+- high-level `objectives[]`
+
+**Example** (Drill contract snippet):
+
+```json
+{
+  "id": "drill-1764691390",
+  "title": "IoT device bridging into OT with weak detection",
+  "stages": [
+    {
+      "id": "stage-1-iot-wireless-security",
+      "order": 1,
+      "skill": "iot-wireless-security",
+      "workflow": "IoT Device Recon and Fingerprinting",
+      "role": "primary"
+    },
+    {
+      "id": "stage-2-ot-ics-security",
+      "order": 2,
+      "skill": "ot-ics-security",
+      "workflow": "OT Asset and Network Mapping",
+      "role": "supporting"
+    }
+  ]
+}
+```
+
+For Oracle, the "contract" can be implicit:
+- the `vm_oracle_answer_v1` payload is itself the "answer contract".
+
+For future Engines (Treasury, Mesh), mirror the same concept:
+- plan file describing what will happen.
+
+#### Step 2 — Execute → `state.json` + `outputs/`
+
+A **runner** component walks through the contract and tracks reality.
+
+**Typical commands**:
+- `init contract.json` → `state.json`
+- `next state.json` → show next stage/checklist
+- `complete-stage <id> --outputs ...` → update `state.json`
+
+The state file (e.g. `drill_state.json`) should contain:
+- `drill_id` / `run_id`
+- `status` (`pending` | `in_progress` | `completed` | `aborted`)
+- `stages[]` with status, timestamps, attached outputs
+- `created_at`, `updated_at`
+- optional `tags` / `context`
+
+**Example** (simplified):
+
+```json
+{
+  "drill_id": "drill-1764691390",
+  "status": "completed",
+  "created_at": "2025-12-02T10:03:00Z",
+  "updated_at": "2025-12-02T11:45:00Z",
+  "stages": [
+    {
+      "id": "stage-1-iot-wireless-security",
+      "status": "completed",
+      "outputs": [
+        "inventory.yaml",
+        "topology.png",
+        "findings.md"
+      ]
+    }
+  ]
+}
+```
+
+Runners exist today for Drills; the same pattern will apply to other Engines.
+
+#### Step 3 — Seal → Receipts
+
+A **sealer** takes:
+- `contract.json`
+- `state.json`
+- `outputs/` (optional, usually via manifest or aggregated hash)
+
+And produces a **receipt** in L3.
+
+**Example** (Drills sealer behavior):
+- Copies contract + state into `cases/drills/<drill-id>/`
+- Mirrors `outputs/`
+- Computes blake3 or similar hash over `drill_state.json` (and later outputs manifest)
+- Derives summary metrics:
+  - `status`
+  - `stages_total`
+  - `stages_completed`
+  - unique domains / workflows
+- Appends a receipt entry to `receipts/drills/drill_runs.jsonl`
+- Calls a generic receipts Merkle updater to update `ROOT.drills.txt`
+- Optionally triggers ANCHOR via Guardian
+
+This "seal" step is what promotes local execution into **civilization evidence**.
+
+---
+
+### 3.3 Ledger Layer (L3) — Scrolls, Roots, Anchors
+
+L3 is shared by all subsystems; only field names differ.
+
+#### 3.3.1 Scrolls
+
+A **scroll** is just a logical ledger space for a domain.
+
+**Examples**:
+- `Drills` (security drills and exercises)
+- `Compliance` (Oracle answers)
+- `Guardian` (anchor events, healing proofs)
+- `Treasury` (credit/debit/settlements)
+- `Mesh` (topology & configuration changes)
+- `OffSec` (real incident & red-team case receipts)
+- `Identity` (DIDs, credentials, auth events)
+- `Observability` (metrics, logs, traces, alerts)
+- `Automation` (workflow executions, approvals)
+
+Each scroll has:
+- 1+ JSONL files under `receipts/<scroll>/`
+- 1 Merkle root file `ROOT.<scroll>.txt`
+
+#### 3.3.2 Receipts
+
+Receipts are append-only JSON objects with at least:
+- `type`: operation type (e.g. `security_drill_run`, `oracle_answer`)
+- domain-specific fields
+- one or more hash fields:
+  - `root_hash` / `answer_hash` / etc.
+- optional `tags`:
+  - `tags: [ "drill", "iot", "kubernetes" ]`
+
+**Drill Receipt** (shape):
+
+```json
+{
+  "type": "security_drill_run",
+  "drill_id": "drill-1764691390",
+  "prompt": "IoT device bridging into OT network with weak detection",
+  "timestamp_started": "2025-12-02T10:03:00Z",
+  "timestamp_completed": "2025-12-02T11:45:00Z",
+  "status": "completed",
+  "stages_total": 3,
+  "stages_completed": 3,
+  "domains": ["iot-wireless-security", "ot-ics-security", "detection-defense-ir"],
+  "workflows": [
+    "IoT Device Recon and Fingerprinting",
+    "OT Asset and Network Mapping",
+    "IR Triage and Containment"
+  ],
+  "severity": "unknown",
+  "tags": ["drill", "iot", "ot", "detection"],
+  "root_hash": "<blake3(drill_state.json or bundle)>",
+  "proof_path": "cases/drills/drill-1764691390/PROOF.json",
+  "artifacts_manifest": "cases/drills/drill-1764691390/ARTIFACTS.sha256"
+}
+```
+
+**Oracle Answer Receipt** (shape):
+
+```json
+{
+  "scroll": "Compliance",
+  "issuer": "did:vm:node:oracle-01",
+  "body": {
+    "op_type": "oracle_answer",
+    "question": "Are we compliant with Annex IV?",
+    "model_id": "gpt-4.1",
+    "citations_used": ["VM-AI-TECHDOC-001 §4.2", "..."],
+    "compliance_flags": {
+      "insufficient_context": false,
+      "ambiguous_requirements": true,
+      "out_of_scope_question": false
+    },
+    "answer_hash": "blake3:...",
+    "context_docs": ["VM-AI-TECHDOC-001_Annex_IV_Technical_Documentation.docx"],
+    "frameworks": ["AI_Act"],
+    "extra": {
+      "version": "v0.5.0",
+      "prompt_version": "vm_oracle_answer_v1"
+    }
+  }
+}
+```
+
+#### 3.3.3 ProofChain & Guardian Anchors
+
+- A receipts update tool (or ProofChain engine) computes Merkle roots over each scroll's JSONL.
+- Guardian sees the new root via `ProofChain.current_root_hex()`.
+- Guardian's Anchor module:
+  - Submits `root_hex` → anchor backend (HTTP/CLI/blockchain/OTS)
+  - Keeps an internal `AnchorStatus` (`last_root`, `last_anchor_id`, `count`).
+  - Emits `SecurityEvents` (`AnchorSuccess`, `AnchorFailure`, `AnchorDivergence`).
+
+---
+
+## 4. Existing Subsystems Mapped to the Pattern
+
+### 4.1 Security Drills (Security Lab Suite)
+
+**Experience (L1)**:
+- `security_lab_router.py` (select skill)
+- `security_lab_chain_engine.py` (multi-skill chain)
+- CLI usage:
+  - `security_lab_chain_engine.py --contract "prompt"` → `contract.json`
+  - `security_lab_drill_runner.py init/next/complete-stage`
+
+**Engine (L2)**:
+- `contract.json` (drill plan)
+- `drill_state.json` (progress)
+- Runners hydrate stages from runbooks (actions, expected_outputs).
+
+**Ledger (L3)**:
+- `security_drill_seal_run.py`:
+  - Syncs case directory
+  - Hashes state
+  - Appends drill receipt → `receipts/drills/drill_runs.jsonl`
+  - Updates `ROOT.drills.txt`
+  - Optionally auto-anchors using existing anchor scripts.
+
+---
+
+### 4.2 Oracle Node (Compliance Appliance)
+
+**Experience (L1)**:
+- MCP server exposing `oracle_answer` tool.
+
+**Engine (L2)**:
+- Corpus loader/search: `corpus/loader.py`, `corpus/search.py`
+- Prompt + schema: `prompts/` (`vm_oracle_answer_v1`, `build_oracle_prompt()`)
+- LLM abstraction: `oracle/llm.py`
+- End-to-end:
+  - question → context → prompt → LLM JSON → schema validation.
+
+**Ledger (L3)**:
+- `emit_oracle_answer_receipt()`
+- Hash:
+  - `answer_hash = "blake3:" + blake3(canonical_answer_json).hexdigest()`
+- Receipts POSTed to `VAULTMESH_RECEIPT_ENDPOINT` (e.g. `/api/receipts/oracle`).
+- Scroll: `Compliance`.
+
+---
+
+### 4.3 Guardian (Anchor-Integrated Sentinel)
+
+**Experience (L1)**:
+- `guardian_cli`:
+  - `guardian anchor-status`
+  - `guardian anchor-now` (with capability)
+- Portal HTTP routes:
+  - `GET /guardian/anchor-status`
+  - `POST /guardian/anchor-now`
+
+**Engine (L2)**:
+- Rust crate `guardian`:
+  - Holds `ProofChain`, `AnchorClient`, `AnchorVerifier`, config.
+  - `run_anchor_cycle(&ProofChain)` → `AnchorVerdict`
+  - `spawn_anchor_task()` for periodic anchoring.
+
+**Ledger (L3)**:
+- Anchors + anchor events:
+  - `anchor_success`/`failure`/`divergence` events.
+  - Can be streamed into `receipts/guardian/anchor_events.jsonl` with `ROOT.guardian.txt` and anchored further (if desired).
+
+---
+
+## 5. Adding New Domains (Treasury, Mesh, OffSec, etc.)
+
+When adding a new subsystem "X" (e.g. Treasury, Mesh), follow this checklist.
+
+### 5.1 Scroll Definition
+
+1. **Pick a scroll name**:
+   - Treasury / Mesh / OffSec / Identity / Observability / Automation, etc.
+
+2. **Define**:
+   - JSONL path: `receipts/<scroll>/<file>.jsonl`
+   - Root file: `ROOT.<scroll>.txt`
+
+3. **Define 1–3 receipt types**:
+   - Treasury:
+     - `treasury_credit`, `treasury_debit`, `treasury_settlement`
+   - Mesh:
+     - `mesh_route_change`, `mesh_node_join`, `mesh_node_leave`
+
+### 5.2 Engine API
+
+For each engine:
+- **Define a Plan API**:
+  - `*_plan_*.py` → produce `contract.json`
+- **Define a Runner**:
+  - `*_runner.py` → manage `state.json` + `outputs/`
+- **Define a Sealer**:
+  - `*_seal_*.py` → write receipts, update roots, maybe anchor.
+
+### 5.3 Query CLI
+
+Add a small query layer:
+- Treasury:
+  - `treasury_query_runs.py`:
+    - filters: node, asset, date range, tags.
+- Mesh:
+  - `mesh_query_changes.py`:
+    - filters: node, segment, change type, date.
+
+This makes scrolls self-explaining and agent-friendly.
+
+---
+
+## 6. Design Gate: "Is It Aligned With the Eternal Pattern?"
+
+Use this quick checklist whenever you design a new feature or refactor an old one.
+
+### 6.1 Experience Layer
+
+- [ ] Is there a clear entrypoint (CLI, MCP tool, HTTP route)?
+- [ ] Is the intent clearly represented in a structured form (arguments, payload, contract)?
+
+### 6.2 Engine Layer
+
+- [ ] Does the subsystem produce a contract (even if implicit)?
+- [ ] Is there a state object tracking progress or outcomes?
+- [ ] Are the actions and outputs visible and inspectable (e.g. via JSON + files)?
+
+### 6.3 Ledger Layer
+
+- [ ] Does the subsystem emit a receipt for its important operations?
+- [ ] Are receipts written to an append-only JSONL file?
+- [ ] Is the JSONL covered by a Merkle root in `ROOT.<scroll>.txt`?
+- [ ] Does Guardian have a way to anchor the relevant root(s)?
+- [ ] Is there/will there be a simple query tool for this scroll?
+
+**If any of these is "no", you have a clear next step.**
+
+---
+
+## 7. Future Extensions (Stable Pattern, Evolving Domains)
+
+The Eternal Pattern is deliberately minimal:
+- It doesn't care what chain you anchor to.
+- It doesn't care which LLM model you use.
+- It doesn't care whether the Runner is human-driven or fully autonomous.
+
+As VaultMesh evolves, you can:
+- **Swap LLMs** → Oracle stays the same; receipts remain valid.
+- **Swap anchor backends** (OTS, Ethereum, Bitcoin, custom chain) → roots remain valid.
+- **Add automated agents** (vm-copilot, OffSec agents, Mesh guardians) → they all just become more Experience Layer clients of the same Engine + Ledger.
+
+**The shape does not change.**
+
+---
+
+## 8. Short Human Explanation (for README / Auditors)
+
+VaultMesh treats every serious operation — a security drill, a compliance answer, an anchor event, a treasury transfer — as a small story with a beginning, middle, and end:
+
+1. A **human or agent** expresses intent
+2. An **engine** plans and executes the work, tracking state
+3. The outcome is **sealed** into an append-only ledger, hashed, merklized, and anchored
+
+This pattern — **Intent → Engine → Receipt → Scroll → Anchor** — is the same across all domains.
+
+It's what makes VaultMesh composable, auditable, and explainable to both humans and machines.
+
+---
+
+## 9. Engine Specifications Index
+
+The following engine specifications implement the Eternal Pattern:
+
+| Engine | Scroll | Description | Receipt Types |
+|--------|--------|-------------|---------------|
+| [VAULTMESH-CONSOLE-ENGINE.md](./VAULTMESH-CONSOLE-ENGINE.md) | `Console` | AI agent sessions, code operations, sovereign development | `console_session_start`, `console_session_end`, `console_command`, `console_file_edit`, `console_tool_call`, `console_approval`, `console_git_commit`, `console_agent_spawn` |
+| [VAULTMESH-MESH-ENGINE.md](./VAULTMESH-MESH-ENGINE.md) | `Mesh` | Federation topology, node management, routes, capabilities | `mesh_node_join`, `mesh_node_leave`, `mesh_route_change`, `mesh_capability_grant`, `mesh_capability_revoke`, `mesh_topology_snapshot` |
+| [VAULTMESH-OFFSEC-ENGINE.md](./VAULTMESH-OFFSEC-ENGINE.md) | `OffSec` | Security incidents, red team engagements, vulnerability tracking | `offsec_incident`, `offsec_redteam`, `offsec_vuln_discovery`, `offsec_remediation`, `offsec_threat_intel`, `offsec_forensic_snapshot` |
+| [VAULTMESH-IDENTITY-ENGINE.md](./VAULTMESH-IDENTITY-ENGINE.md) | `Identity` | DIDs, verifiable credentials, authentication, authorization | `identity_did_create`, `identity_did_rotate`, `identity_did_revoke`, `identity_credential_issue`, `identity_credential_revoke`, `identity_auth_event`, `identity_authz_decision` |
+| [VAULTMESH-OBSERVABILITY-ENGINE.md](./VAULTMESH-OBSERVABILITY-ENGINE.md) | `Observability` | Metrics, logs, traces, alerts, SLOs | `obs_metric_snapshot`, `obs_log_batch`, `obs_trace_complete`, `obs_alert_fired`, `obs_alert_resolved`, `obs_slo_report`, `obs_anomaly_detected` |
+| [VAULTMESH-AUTOMATION-ENGINE.md](./VAULTMESH-AUTOMATION-ENGINE.md) | `Automation` | n8n workflows, schedules, triggers, approvals | `auto_workflow_register`, `auto_workflow_execute`, `auto_workflow_complete`, `auto_schedule_create`, `auto_trigger_fire`, `auto_approval_request`, `auto_approval_decision` |
+| [VAULTMESH-PSI-FIELD-ENGINE.md](./VAULTMESH-PSI-FIELD-ENGINE.md) | `PsiField` | Alchemical consciousness, phase transitions, transmutations | `psi_phase_transition`, `psi_emergence_event`, `psi_transmutation`, `psi_resonance`, `psi_integration`, `psi_oracle_insight` |
+| [VAULTMESH-FEDERATION-PROTOCOL.md](./VAULTMESH-FEDERATION-PROTOCOL.md) | `Federation` | Cross-mesh trust, witness verification, cross-anchoring | `fed_trust_proposal`, `fed_trust_established`, `fed_trust_revoked`, `fed_witness_event`, `fed_cross_anchor`, `fed_schema_sync` |
+| [VAULTMESH-CONSTITUTIONAL-GOVERNANCE.md](./VAULTMESH-CONSTITUTIONAL-GOVERNANCE.md) | `Governance` | Constitutional rules, amendments, enforcement, violations | `gov_proposal`, `gov_vote`, `gov_ratification`, `gov_amendment`, `gov_executive_order`, `gov_violation`, `gov_enforcement` |
+
+**Implementation Reference**:
+| Document | Description |
+|----------|-------------|
+| [VAULTMESH-IMPLEMENTATION-SCAFFOLDS.md](./VAULTMESH-IMPLEMENTATION-SCAFFOLDS.md) | Rust structs, Python CLI, directory structure |
+| [VAULTMESH-MCP-SERVERS.md](./VAULTMESH-MCP-SERVERS.md) | MCP server implementations for Claude integration, tool definitions, gateway |
+| [VAULTMESH-DEPLOYMENT-MANIFESTS.md](./VAULTMESH-DEPLOYMENT-MANIFESTS.md) | Kubernetes manifests, Docker Compose, infrastructure-as-code |
+| [VAULTMESH-MONITORING-STACK.md](./VAULTMESH-MONITORING-STACK.md) | Prometheus config, Grafana dashboards, alerting rules, metrics |
+| [VAULTMESH-TESTING-FRAMEWORK.md](./VAULTMESH-TESTING-FRAMEWORK.md) | Property-based tests, integration tests, chaos tests, fixtures |
+| [VAULTMESH-MIGRATION-GUIDE.md](./VAULTMESH-MIGRATION-GUIDE.md) | Version upgrades, migration scripts, rollback procedures |
+
+Each engine specification follows the same structure:
+1. **Scroll Definition** (JSONL path, root file, receipt types)
+2. **Core Concepts** (domain-specific entities)
+3. **Mapping to Eternal Pattern** (L1, L2, L3)
+4. **Query Interface**
+5. **Design Gate Checklist**
+6. **Integration Points**
+7. **Future Extensions**