vm-core/docs/VAULTMESH-ETERNAL-PATTERN.md

# VAULTMESH-ETERNAL-PATTERN.md

**Canonical Design Pattern for All VaultMesh Subsystems**

> *Every serious subsystem in VaultMesh should feel different in flavor, but identical in **shape**.*

This document defines that shared shape — the **Eternal Pattern**.

It is the architectural law that binds Drills, Oracle, Guardian, Treasury, Mesh, and any future module into one Civilization Ledger.

---

## 1. Core Idea (One-Line Contract)

All VaultMesh subsystems follow this arc:

> **Real-world intent → Engine → Structured JSON → Receipt → Scroll → Guardian Anchor**

If a new feature does **not** fit this pattern, it's either:

- not finished yet, or
- not part of the Ledger core.

---

## 2. Three-Layer VaultMesh Stack

At the highest level, VaultMesh is three stacked layers:

```
┌───────────────────────────────────────────────┐
│  L1 — Experience Layer                        │
│  (Humans & Agents)                            │
│  • CLI / UI / MCP tools / agents              │
│  • "Ask a question", "start a drill",         │
│    "anchor now", "run settlement"             │
└───────────────────────────────────────────────┘
                 │
                 ▼
┌───────────────────────────────────────────────┐
│  L2 — Engine Layer                            │
│  (Domain Engines & Contracts)                 │
│  • Domain logics: Drills, Oracle, Guardian,   │
│    Treasury, Mesh, OffSec, etc.               │
│  • Contracts (plans)                          │
│  • Runners (state machines)                   │
│  • State JSON (progress)                      │
└───────────────────────────────────────────────┘
                 │
                 ▼
┌───────────────────────────────────────────────┐
│  L3 — Ledger Layer                            │
│  (Receipts, Scrolls, ProofChain, Anchors)     │
│  • Receipts in append-only JSONL files        │
│  • Scrolls per domain (Drills, Compliance,    │
│    Guardian, Treasury, Mesh, etc.)            │
│  • Merkle roots (ROOT.<scroll>.txt)           │
│  • Guardian anchor cycles (local/OTS/chain)   │
└───────────────────────────────────────────────┘
```

Everything you build plugs into this stack.

---

## 3. Eternal Pattern — Generic Lifecycle

This is the reusable template for any new subsystem "X".

### 3.1 Experience Layer (L1) — Intent In

**Goal**: Take messy human/agent intent and normalize it.

**Typical surfaces**:
- CLI (`vm-drills`, `vm-oracle`, `guardian`, `vm-treasury`, etc.)
- MCP tools (e.g. `oracle_answer`)
- Web / TUI / dashboard
- Automation hooks (cron, CI, schedulers)

**Typical inputs**:
- "Run a security drill for IoT ↔ OT"
- "Are we compliant with Annex IV today?"
- "Anchor this ProofChain root"
- "Reconcile treasury balances between nodes"
- "Apply this mesh topology change"

**L1 should**:
- Capture the raw intent
- Attach minimal context (who, when, where)
- Hand it off to the appropriate Engine in L2

---

### 3.2 Engine Layer (L2) — Plan and Execute

Every Engine follows the same internal shape:

#### Step 1 — Plan → `contract.json`

An Engine takes the intent and creates a **contract**:

`contract.json` (or equivalent JSON struct) contains:
- `id`: unique contract / drill / run id
- `title`: short human title
- `severity` / `priority` (optional, domain-specific)
- `stages[]` / `steps[]`:
  - ordered id, skill / module, workflow, role, objective
- high-level `objectives[]`

**Example** (Drill contract snippet):

```json
{
  "id": "drill-1764691390",
  "title": "IoT device bridging into OT with weak detection",
  "stages": [
    {
      "id": "stage-1-iot-wireless-security",
      "order": 1,
      "skill": "iot-wireless-security",
      "workflow": "IoT Device Recon and Fingerprinting",
      "role": "primary"
    },
    {
      "id": "stage-2-ot-ics-security",
      "order": 2,
      "skill": "ot-ics-security",
      "workflow": "OT Asset and Network Mapping",
      "role": "supporting"
    }
  ]
}
```

For Oracle, the "contract" can be implicit:
- the `vm_oracle_answer_v1` payload is itself the "answer contract".

For future Engines (Treasury, Mesh), mirror the same concept:
- plan file describing what will happen.

#### Step 2 — Execute → `state.json` + `outputs/`

A **runner** component walks through the contract and tracks reality.

**Typical commands**:
- `init contract.json` → `state.json`
- `next state.json` → show next stage/checklist
- `complete-stage <id> --outputs ...` → update `state.json`

The state file (e.g. `drill_state.json`) should contain:
- `drill_id` / `run_id`
- `status` (`pending` | `in_progress` | `completed` | `aborted`)
- `stages[]` with status, timestamps, attached outputs
- `created_at`, `updated_at`
- optional `tags` / `context`

**Example** (simplified):

```json
{
  "drill_id": "drill-1764691390",
  "status": "completed",
  "created_at": "2025-12-02T10:03:00Z",
  "updated_at": "2025-12-02T11:45:00Z",
  "stages": [
    {
      "id": "stage-1-iot-wireless-security",
      "status": "completed",
      "outputs": [
        "inventory.yaml",
        "topology.png",
        "findings.md"
      ]
    }
  ]
}
```

Runners exist today for Drills; the same pattern will apply to other Engines.

#### Step 3 — Seal → Receipts

A **sealer** takes:
- `contract.json`
- `state.json`
- `outputs/` (optional, usually via manifest or aggregated hash)

And produces a **receipt** in L3.

**Example** (Drills sealer behavior):
- Copies contract + state into `cases/drills/<drill-id>/`
- Mirrors `outputs/`
- Computes blake3 or similar hash over `drill_state.json` (and later outputs manifest)
- Derives summary metrics:
  - `status`
  - `stages_total`
  - `stages_completed`
  - unique domains / workflows
- Appends a receipt entry to `receipts/drills/drill_runs.jsonl`
- Calls a generic receipts Merkle updater to update `ROOT.drills.txt`
- Optionally triggers ANCHOR via Guardian

This "seal" step is what promotes local execution into **civilization evidence**.

---

### 3.3 Ledger Layer (L3) — Scrolls, Roots, Anchors

L3 is shared by all subsystems; only field names differ.

#### 3.3.1 Scrolls

A **scroll** is just a logical ledger space for a domain.

**Examples**:
- `Drills` (security drills and exercises)
- `Compliance` (Oracle answers)
- `Guardian` (anchor events, healing proofs)
- `Treasury` (credit/debit/settlements)
- `Mesh` (topology & configuration changes)
- `OffSec` (real incident & red-team case receipts)
- `Identity` (DIDs, credentials, auth events)
- `Observability` (metrics, logs, traces, alerts)
- `Automation` (workflow executions, approvals)

Each scroll has:
- 1+ JSONL files under `receipts/<scroll>/`
- 1 Merkle root file `ROOT.<scroll>.txt`

#### 3.3.2 Receipts

Receipts are append-only JSON objects with at least:
- `type`: operation type (e.g. `security_drill_run`, `oracle_answer`)
- domain-specific fields
- one or more hash fields:
  - `root_hash` / `answer_hash` / etc.
- optional `tags`:
  - `tags: [ "drill", "iot", "kubernetes" ]`

**Drill Receipt** (shape):

```json
{
  "type": "security_drill_run",
  "drill_id": "drill-1764691390",
  "prompt": "IoT device bridging into OT network with weak detection",
  "timestamp_started": "2025-12-02T10:03:00Z",
  "timestamp_completed": "2025-12-02T11:45:00Z",
  "status": "completed",
  "stages_total": 3,
  "stages_completed": 3,
  "domains": ["iot-wireless-security", "ot-ics-security", "detection-defense-ir"],
  "workflows": [
    "IoT Device Recon and Fingerprinting",
    "OT Asset and Network Mapping",
    "IR Triage and Containment"
  ],
  "severity": "unknown",
  "tags": ["drill", "iot", "ot", "detection"],
  "root_hash": "<blake3(drill_state.json or bundle)>",
  "proof_path": "cases/drills/drill-1764691390/PROOF.json",
  "artifacts_manifest": "cases/drills/drill-1764691390/ARTIFACTS.sha256"
}
```

**Oracle Answer Receipt** (shape):

```json
{
  "scroll": "Compliance",
  "issuer": "did:vm:node:oracle-01",
  "body": {
    "op_type": "oracle_answer",
    "question": "Are we compliant with Annex IV?",
    "model_id": "gpt-4.1",
    "citations_used": ["VM-AI-TECHDOC-001 §4.2", "..."],
    "compliance_flags": {
      "insufficient_context": false,
      "ambiguous_requirements": true,
      "out_of_scope_question": false
    },
    "answer_hash": "blake3:...",
    "context_docs": ["VM-AI-TECHDOC-001_Annex_IV_Technical_Documentation.docx"],
    "frameworks": ["AI_Act"],
    "extra": {
      "version": "v0.5.0",
      "prompt_version": "vm_oracle_answer_v1"
    }
  }
}
```

#### 3.3.3 ProofChain & Guardian Anchors

- A receipts update tool (or ProofChain engine) computes Merkle roots over each scroll's JSONL.
- Guardian sees the new root via `ProofChain.current_root_hex()`.
- Guardian's Anchor module:
  - Submits `root_hex` → anchor backend (HTTP/CLI/blockchain/OTS)
  - Keeps an internal `AnchorStatus` (`last_root`, `last_anchor_id`, `count`).
  - Emits `SecurityEvents` (`AnchorSuccess`, `AnchorFailure`, `AnchorDivergence`).

---

## 4. Existing Subsystems Mapped to the Pattern

### 4.1 Security Drills (Security Lab Suite)

**Experience (L1)**:
- `security_lab_router.py` (select skill)
- `security_lab_chain_engine.py` (multi-skill chain)
- CLI usage:
  - `security_lab_chain_engine.py --contract "prompt"` → `contract.json`
  - `security_lab_drill_runner.py init/next/complete-stage`

**Engine (L2)**:
- `contract.json` (drill plan)
- `drill_state.json` (progress)
- Runners hydrate stages from runbooks (actions, expected_outputs).

**Ledger (L3)**:
- `security_drill_seal_run.py`:
  - Syncs case directory
  - Hashes state
  - Appends drill receipt → `receipts/drills/drill_runs.jsonl`
  - Updates `ROOT.drills.txt`
  - Optionally auto-anchors using existing anchor scripts.

---

### 4.2 Oracle Node (Compliance Appliance)

**Experience (L1)**:
- MCP server exposing `oracle_answer` tool.

**Engine (L2)**:
- Corpus loader/search: `corpus/loader.py`, `corpus/search.py`
- Prompt + schema: `prompts/` (`vm_oracle_answer_v1`, `build_oracle_prompt()`)
- LLM abstraction: `oracle/llm.py`
- End-to-end:
  - question → context → prompt → LLM JSON → schema validation.

**Ledger (L3)**:
- `emit_oracle_answer_receipt()`
- Hash:
  - `answer_hash = "blake3:" + blake3(canonical_answer_json).hexdigest()`
- Receipts POSTed to `VAULTMESH_RECEIPT_ENDPOINT` (e.g. `/api/receipts/oracle`).
- Scroll: `Compliance`.

---

### 4.3 Guardian (Anchor-Integrated Sentinel)

**Experience (L1)**:
- `guardian_cli`:
  - `guardian anchor-status`
  - `guardian anchor-now` (with capability)
- Portal HTTP routes:
  - `GET /guardian/anchor-status`
  - `POST /guardian/anchor-now`

**Engine (L2)**:
- Rust crate `guardian`:
  - Holds `ProofChain`, `AnchorClient`, `AnchorVerifier`, config.
  - `run_anchor_cycle(&ProofChain)` → `AnchorVerdict`
  - `spawn_anchor_task()` for periodic anchoring.

**Ledger (L3)**:
- Anchors + anchor events:
  - `anchor_success`/`failure`/`divergence` events.
  - Can be streamed into `receipts/guardian/anchor_events.jsonl` with `ROOT.guardian.txt` and anchored further (if desired).

---

## 5. Adding New Domains (Treasury, Mesh, OffSec, etc.)

When adding a new subsystem "X" (e.g. Treasury, Mesh), follow this checklist.

### 5.1 Scroll Definition

1. **Pick a scroll name**:
   - Treasury / Mesh / OffSec / Identity / Observability / Automation, etc.

2. **Define**:
   - JSONL path: `receipts/<scroll>/<file>.jsonl`
   - Root file: `ROOT.<scroll>.txt`

3. **Define 1–3 receipt types**:
   - Treasury:
     - `treasury_credit`, `treasury_debit`, `treasury_settlement`
   - Mesh:
     - `mesh_route_change`, `mesh_node_join`, `mesh_node_leave`

### 5.2 Engine API

For each engine:
- **Define a Plan API**:
  - `*_plan_*.py` → produce `contract.json`
- **Define a Runner**:
  - `*_runner.py` → manage `state.json` + `outputs/`
- **Define a Sealer**:
  - `*_seal_*.py` → write receipts, update roots, maybe anchor.

### 5.3 Query CLI

Add a small query layer:
- Treasury:
  - `treasury_query_runs.py`:
    - filters: node, asset, date range, tags.
- Mesh:
  - `mesh_query_changes.py`:
    - filters: node, segment, change type, date.

This makes scrolls self-explaining and agent-friendly.

---

## 6. Design Gate: "Is It Aligned With the Eternal Pattern?"

Use this quick checklist whenever you design a new feature or refactor an old one.

### 6.1 Experience Layer

- [ ] Is there a clear entrypoint (CLI, MCP tool, HTTP route)?
- [ ] Is the intent clearly represented in a structured form (arguments, payload, contract)?

### 6.2 Engine Layer

- [ ] Does the subsystem produce a contract (even if implicit)?
- [ ] Is there a state object tracking progress or outcomes?
- [ ] Are the actions and outputs visible and inspectable (e.g. via JSON + files)?

### 6.3 Ledger Layer

- [ ] Does the subsystem emit a receipt for its important operations?
- [ ] Are receipts written to an append-only JSONL file?
- [ ] Is the JSONL covered by a Merkle root in `ROOT.<scroll>.txt`?
- [ ] Does Guardian have a way to anchor the relevant root(s)?
- [ ] Is there/will there be a simple query tool for this scroll?

**If any of these is "no", you have a clear next step.**

---

## 7. Future Extensions (Stable Pattern, Evolving Domains)

The Eternal Pattern is deliberately minimal:
- It doesn't care what chain you anchor to.
- It doesn't care which LLM model you use.
- It doesn't care whether the Runner is human-driven or fully autonomous.

As VaultMesh evolves, you can:
- **Swap LLMs** → Oracle stays the same; receipts remain valid.
- **Swap anchor backends** (OTS, Ethereum, Bitcoin, custom chain) → roots remain valid.
- **Add automated agents** (vm-copilot, OffSec agents, Mesh guardians) → they all just become more Experience Layer clients of the same Engine + Ledger.

**The shape does not change.**

---

## 8. Short Human Explanation (for README / Auditors)

VaultMesh treats every serious operation — a security drill, a compliance answer, an anchor event, a treasury transfer — as a small story with a beginning, middle, and end:

1. A **human or agent** expresses intent
2. An **engine** plans and executes the work, tracking state
3. The outcome is **sealed** into an append-only ledger, hashed, merklized, and anchored

This pattern — **Intent → Engine → Receipt → Scroll → Anchor** — is the same across all domains.

It's what makes VaultMesh composable, auditable, and explainable to both humans and machines.

---

## 9. Engine Specifications Index

The following engine specifications implement the Eternal Pattern:

| Engine | Scroll | Description | Receipt Types |
|--------|--------|-------------|---------------|
| [VAULTMESH-CONSOLE-ENGINE.md](./VAULTMESH-CONSOLE-ENGINE.md) | `Console` | AI agent sessions, code operations, sovereign development | `console_session_start`, `console_session_end`, `console_command`, `console_file_edit`, `console_tool_call`, `console_approval`, `console_git_commit`, `console_agent_spawn` |
| [VAULTMESH-MESH-ENGINE.md](./VAULTMESH-MESH-ENGINE.md) | `Mesh` | Federation topology, node management, routes, capabilities | `mesh_node_join`, `mesh_node_leave`, `mesh_route_change`, `mesh_capability_grant`, `mesh_capability_revoke`, `mesh_topology_snapshot` |
| [VAULTMESH-OFFSEC-ENGINE.md](./VAULTMESH-OFFSEC-ENGINE.md) | `OffSec` | Security incidents, red team engagements, vulnerability tracking | `offsec_incident`, `offsec_redteam`, `offsec_vuln_discovery`, `offsec_remediation`, `offsec_threat_intel`, `offsec_forensic_snapshot` |
| [VAULTMESH-IDENTITY-ENGINE.md](./VAULTMESH-IDENTITY-ENGINE.md) | `Identity` | DIDs, verifiable credentials, authentication, authorization | `identity_did_create`, `identity_did_rotate`, `identity_did_revoke`, `identity_credential_issue`, `identity_credential_revoke`, `identity_auth_event`, `identity_authz_decision` |
| [VAULTMESH-OBSERVABILITY-ENGINE.md](./VAULTMESH-OBSERVABILITY-ENGINE.md) | `Observability` | Metrics, logs, traces, alerts, SLOs | `obs_metric_snapshot`, `obs_log_batch`, `obs_trace_complete`, `obs_alert_fired`, `obs_alert_resolved`, `obs_slo_report`, `obs_anomaly_detected` |
| [VAULTMESH-AUTOMATION-ENGINE.md](./VAULTMESH-AUTOMATION-ENGINE.md) | `Automation` | n8n workflows, schedules, triggers, approvals | `auto_workflow_register`, `auto_workflow_execute`, `auto_workflow_complete`, `auto_schedule_create`, `auto_trigger_fire`, `auto_approval_request`, `auto_approval_decision` |
| [VAULTMESH-PSI-FIELD-ENGINE.md](./VAULTMESH-PSI-FIELD-ENGINE.md) | `PsiField` | Alchemical consciousness, phase transitions, transmutations | `psi_phase_transition`, `psi_emergence_event`, `psi_transmutation`, `psi_resonance`, `psi_integration`, `psi_oracle_insight` |
| [VAULTMESH-FEDERATION-PROTOCOL.md](./VAULTMESH-FEDERATION-PROTOCOL.md) | `Federation` | Cross-mesh trust, witness verification, cross-anchoring | `fed_trust_proposal`, `fed_trust_established`, `fed_trust_revoked`, `fed_witness_event`, `fed_cross_anchor`, `fed_schema_sync` |
| [VAULTMESH-CONSTITUTIONAL-GOVERNANCE.md](./VAULTMESH-CONSTITUTIONAL-GOVERNANCE.md) | `Governance` | Constitutional rules, amendments, enforcement, violations | `gov_proposal`, `gov_vote`, `gov_ratification`, `gov_amendment`, `gov_executive_order`, `gov_violation`, `gov_enforcement` |

**Implementation Reference**:
| Document | Description |
|----------|-------------|
| [VAULTMESH-IMPLEMENTATION-SCAFFOLDS.md](./VAULTMESH-IMPLEMENTATION-SCAFFOLDS.md) | Rust structs, Python CLI, directory structure |
| [VAULTMESH-MCP-SERVERS.md](./VAULTMESH-MCP-SERVERS.md) | MCP server implementations for Claude integration, tool definitions, gateway |
| [VAULTMESH-DEPLOYMENT-MANIFESTS.md](./VAULTMESH-DEPLOYMENT-MANIFESTS.md) | Kubernetes manifests, Docker Compose, infrastructure-as-code |
| [VAULTMESH-MONITORING-STACK.md](./VAULTMESH-MONITORING-STACK.md) | Prometheus config, Grafana dashboards, alerting rules, metrics |
| [VAULTMESH-TESTING-FRAMEWORK.md](./VAULTMESH-TESTING-FRAMEWORK.md) | Property-based tests, integration tests, chaos tests, fixtures |
| [VAULTMESH-MIGRATION-GUIDE.md](./VAULTMESH-MIGRATION-GUIDE.md) | Version upgrades, migration scripts, rollback procedures |

Each engine specification follows the same structure:
1. **Scroll Definition** (JSONL path, root file, receipt types)
2. **Core Concepts** (domain-specific entities)
3. **Mapping to Eternal Pattern** (L1, L2, L3)
4. **Query Interface**
5. **Design Gate Checklist**
6. **Integration Points**
7. **Future Extensions**