vm-core/docs/VAULTMESH-ETERNAL-PATTERN.md

VAULTMESH-ETERNAL-PATTERN.md

Canonical Design Pattern for All VaultMesh Subsystems

Every serious subsystem in VaultMesh should feel different in flavor, but identical in shape.

This document defines that shared shape — the Eternal Pattern.

It is the architectural law that binds Drills, Oracle, Guardian, Treasury, Mesh, and any future module into one Civilization Ledger.

---

1. Core Idea (One-Line Contract)

All VaultMesh subsystems follow this arc:

Real-world intent → Engine → Structured JSON → Receipt → Scroll → Guardian Anchor

If a new feature does not fit this pattern, it's either:

• not finished yet, or
• not part of the Ledger core.

---

2. Three-Layer VaultMesh Stack

At the highest level, VaultMesh is three stacked layers:

┌───────────────────────────────────────────────┐
│  L1 — Experience Layer                        │
│  (Humans & Agents)                            │
│  • CLI / UI / MCP tools / agents              │
│  • "Ask a question", "start a drill",         │
│    "anchor now", "run settlement"             │
└───────────────────────────────────────────────┘
                 │
                 ▼
┌───────────────────────────────────────────────┐
│  L2 — Engine Layer                            │
│  (Domain Engines & Contracts)                 │
│  • Domain logics: Drills, Oracle, Guardian,   │
│    Treasury, Mesh, OffSec, etc.               │
│  • Contracts (plans)                          │
│  • Runners (state machines)                   │
│  • State JSON (progress)                      │
└───────────────────────────────────────────────┘
                 │
                 ▼
┌───────────────────────────────────────────────┐
│  L3 — Ledger Layer                            │
│  (Receipts, Scrolls, ProofChain, Anchors)     │
│  • Receipts in append-only JSONL files        │
│  • Scrolls per domain (Drills, Compliance,    │
│    Guardian, Treasury, Mesh, etc.)            │
│  • Merkle roots (ROOT.<scroll>.txt)           │
│  • Guardian anchor cycles (local/OTS/chain)   │
└───────────────────────────────────────────────┘

Everything you build plugs into this stack.

---

3. Eternal Pattern — Generic Lifecycle

This is the reusable template for any new subsystem "X".

3.1 Experience Layer (L1) — Intent In

Goal: Take messy human/agent intent and normalize it.

Typical surfaces:

• CLI (vm-drills, vm-oracle, guardian, vm-treasury, etc.)
• MCP tools (e.g. oracle_answer)
• Web / TUI / dashboard
• Automation hooks (cron, CI, schedulers)

Typical inputs:

• "Run a security drill for IoT ↔ OT"
• "Are we compliant with Annex IV today?"
• "Anchor this ProofChain root"
• "Reconcile treasury balances between nodes"
• "Apply this mesh topology change"

L1 should:

• Capture the raw intent
• Attach minimal context (who, when, where)
• Hand it off to the appropriate Engine in L2

---

3.2 Engine Layer (L2) — Plan and Execute

Every Engine follows the same internal shape:

Step 1 — Plan → contract.json

An Engine takes the intent and creates a contract:

contract.json (or equivalent JSON struct) contains:

• id: unique contract / drill / run id
• title: short human title
• severity / priority (optional, domain-specific)
• stages[] / steps[]:
  • ordered id, skill / module, workflow, role, objective
• high-level objectives[]

Example (Drill contract snippet):

{
  "id": "drill-1764691390",
  "title": "IoT device bridging into OT with weak detection",
  "stages": [
    {
      "id": "stage-1-iot-wireless-security",
      "order": 1,
      "skill": "iot-wireless-security",
      "workflow": "IoT Device Recon and Fingerprinting",
      "role": "primary"
    },
    {
      "id": "stage-2-ot-ics-security",
      "order": 2,
      "skill": "ot-ics-security",
      "workflow": "OT Asset and Network Mapping",
      "role": "supporting"
    }
  ]
}

For Oracle, the "contract" can be implicit:

• the vm_oracle_answer_v1 payload is itself the "answer contract".

For future Engines (Treasury, Mesh), mirror the same concept:

• plan file describing what will happen.

Step 2 — Execute → state.json + outputs/

A runner component walks through the contract and tracks reality.

Typical commands:

• init contract.json → state.json
• next state.json → show next stage/checklist
• complete-stage <id> --outputs ... → update state.json

The state file (e.g. drill_state.json) should contain:

• drill_id / run_id
• status (pending | in_progress | completed | aborted)
• stages[] with status, timestamps, attached outputs
• created_at, updated_at
• optional tags / context

Example (simplified):

{
  "drill_id": "drill-1764691390",
  "status": "completed",
  "created_at": "2025-12-02T10:03:00Z",
  "updated_at": "2025-12-02T11:45:00Z",
  "stages": [
    {
      "id": "stage-1-iot-wireless-security",
      "status": "completed",
      "outputs": [
        "inventory.yaml",
        "topology.png",
        "findings.md"
      ]
    }
  ]
}

Runners exist today for Drills; the same pattern will apply to other Engines.

Step 3 — Seal → Receipts

A sealer takes:

• contract.json
• state.json
• outputs/ (optional, usually via manifest or aggregated hash)

And produces a receipt in L3.

Example (Drills sealer behavior):

• Copies contract + state into cases/drills/<drill-id>/
• Mirrors outputs/
• Computes blake3 or similar hash over drill_state.json (and later outputs manifest)
• Derives summary metrics:
  • status
  • stages_total
  • stages_completed
  • unique domains / workflows
• Appends a receipt entry to receipts/drills/drill_runs.jsonl
• Calls a generic receipts Merkle updater to update ROOT.drills.txt
• Optionally triggers ANCHOR via Guardian

This "seal" step is what promotes local execution into civilization evidence.

---

3.3 Ledger Layer (L3) — Scrolls, Roots, Anchors

L3 is shared by all subsystems; only field names differ.

3.3.1 Scrolls

A scroll is just a logical ledger space for a domain.

Examples:

• Drills (security drills and exercises)
• Compliance (Oracle answers)
• Guardian (anchor events, healing proofs)
• Treasury (credit/debit/settlements)
• Mesh (topology & configuration changes)
• OffSec (real incident & red-team case receipts)
• Identity (DIDs, credentials, auth events)
• Observability (metrics, logs, traces, alerts)
• Automation (workflow executions, approvals)

Each scroll has:

• 1+ JSONL files under receipts/<scroll>/
• 1 Merkle root file ROOT.<scroll>.txt

3.3.2 Receipts

Receipts are append-only JSON objects with at least:

• type: operation type (e.g. security_drill_run, oracle_answer)
• domain-specific fields
• one or more hash fields:
  • root_hash / answer_hash / etc.
• optional tags:
  • tags: [ "drill", "iot", "kubernetes" ]

Drill Receipt (shape):

{
  "type": "security_drill_run",
  "drill_id": "drill-1764691390",
  "prompt": "IoT device bridging into OT network with weak detection",
  "timestamp_started": "2025-12-02T10:03:00Z",
  "timestamp_completed": "2025-12-02T11:45:00Z",
  "status": "completed",
  "stages_total": 3,
  "stages_completed": 3,
  "domains": ["iot-wireless-security", "ot-ics-security", "detection-defense-ir"],
  "workflows": [
    "IoT Device Recon and Fingerprinting",
    "OT Asset and Network Mapping",
    "IR Triage and Containment"
  ],
  "severity": "unknown",
  "tags": ["drill", "iot", "ot", "detection"],
  "root_hash": "<blake3(drill_state.json or bundle)>",
  "proof_path": "cases/drills/drill-1764691390/PROOF.json",
  "artifacts_manifest": "cases/drills/drill-1764691390/ARTIFACTS.sha256"
}

Oracle Answer Receipt (shape):

{
  "scroll": "Compliance",
  "issuer": "did:vm:node:oracle-01",
  "body": {
    "op_type": "oracle_answer",
    "question": "Are we compliant with Annex IV?",
    "model_id": "gpt-4.1",
    "citations_used": ["VM-AI-TECHDOC-001 §4.2", "..."],
    "compliance_flags": {
      "insufficient_context": false,
      "ambiguous_requirements": true,
      "out_of_scope_question": false
    },
    "answer_hash": "blake3:...",
    "context_docs": ["VM-AI-TECHDOC-001_Annex_IV_Technical_Documentation.docx"],
    "frameworks": ["AI_Act"],
    "extra": {
      "version": "v0.5.0",
      "prompt_version": "vm_oracle_answer_v1"
    }
  }
}

3.3.3 ProofChain & Guardian Anchors

• A receipts update tool (or ProofChain engine) computes Merkle roots over each scroll's JSONL.
• Guardian sees the new root via ProofChain.current_root_hex().
• Guardian's Anchor module:
  • Submits root_hex → anchor backend (HTTP/CLI/blockchain/OTS)
  • Keeps an internal AnchorStatus (last_root, last_anchor_id, count).
  • Emits SecurityEvents (AnchorSuccess, AnchorFailure, AnchorDivergence).

---

4. Existing Subsystems Mapped to the Pattern

4.1 Security Drills (Security Lab Suite)

Experience (L1):

• security_lab_router.py (select skill)
• security_lab_chain_engine.py (multi-skill chain)
• CLI usage:
  • security_lab_chain_engine.py --contract "prompt" → contract.json
  • security_lab_drill_runner.py init/next/complete-stage

Engine (L2):

• contract.json (drill plan)
• drill_state.json (progress)
• Runners hydrate stages from runbooks (actions, expected_outputs).

Ledger (L3):

• security_drill_seal_run.py:
  • Syncs case directory
  • Hashes state
  • Appends drill receipt → receipts/drills/drill_runs.jsonl
  • Updates ROOT.drills.txt
  • Optionally auto-anchors using existing anchor scripts.

---

4.2 Oracle Node (Compliance Appliance)

Experience (L1):

• MCP server exposing oracle_answer tool.

Engine (L2):

• Corpus loader/search: corpus/loader.py, corpus/search.py
• Prompt + schema: prompts/ (vm_oracle_answer_v1, build_oracle_prompt())
• LLM abstraction: oracle/llm.py
• End-to-end:
  • question → context → prompt → LLM JSON → schema validation.

Ledger (L3):

• emit_oracle_answer_receipt()
• Hash:
  • answer_hash = "blake3:" + blake3(canonical_answer_json).hexdigest()
• Receipts POSTed to VAULTMESH_RECEIPT_ENDPOINT (e.g. /api/receipts/oracle).
• Scroll: Compliance.

---

4.3 Guardian (Anchor-Integrated Sentinel)

Experience (L1):

• guardian_cli:
  • guardian anchor-status
  • guardian anchor-now (with capability)
• Portal HTTP routes:
  • GET /guardian/anchor-status
  • POST /guardian/anchor-now

Engine (L2):

• Rust crate guardian:
  • Holds ProofChain, AnchorClient, AnchorVerifier, config.
  • run_anchor_cycle(&ProofChain) → AnchorVerdict
  • spawn_anchor_task() for periodic anchoring.

Ledger (L3):

• Anchors + anchor events:
  • anchor_success/failure/divergence events.
  • Can be streamed into receipts/guardian/anchor_events.jsonl with ROOT.guardian.txt and anchored further (if desired).

---

5. Adding New Domains (Treasury, Mesh, OffSec, etc.)

When adding a new subsystem "X" (e.g. Treasury, Mesh), follow this checklist.

5.1 Scroll Definition

1. Pick a scroll name:

   • Treasury / Mesh / OffSec / Identity / Observability / Automation, etc.

2. Define:

   • JSONL path: receipts/<scroll>/<file>.jsonl
   • Root file: ROOT.<scroll>.txt

3. Define 1–3 receipt types:

   • Treasury:
     • treasury_credit, treasury_debit, treasury_settlement
   • Mesh:
     • mesh_route_change, mesh_node_join, mesh_node_leave

5.2 Engine API

For each engine:

• Define a Plan API:
  • *_plan_*.py → produce contract.json
• Define a Runner:
  • *_runner.py → manage state.json + outputs/
• Define a Sealer:
  • *_seal_*.py → write receipts, update roots, maybe anchor.

5.3 Query CLI

Add a small query layer:

• Treasury:
  • treasury_query_runs.py:
    • filters: node, asset, date range, tags.
• Mesh:
  • mesh_query_changes.py:
    • filters: node, segment, change type, date.

This makes scrolls self-explaining and agent-friendly.

---

6. Design Gate: "Is It Aligned With the Eternal Pattern?"

Use this quick checklist whenever you design a new feature or refactor an old one.

6.1 Experience Layer

• Is there a clear entrypoint (CLI, MCP tool, HTTP route)?
• Is the intent clearly represented in a structured form (arguments, payload, contract)?

6.2 Engine Layer

• Does the subsystem produce a contract (even if implicit)?
• Is there a state object tracking progress or outcomes?
• Are the actions and outputs visible and inspectable (e.g. via JSON + files)?

6.3 Ledger Layer

• Does the subsystem emit a receipt for its important operations?
• Are receipts written to an append-only JSONL file?
• Is the JSONL covered by a Merkle root in ROOT.<scroll>.txt?
• Does Guardian have a way to anchor the relevant root(s)?
• Is there/will there be a simple query tool for this scroll?

If any of these is "no", you have a clear next step.

---

7. Future Extensions (Stable Pattern, Evolving Domains)

The Eternal Pattern is deliberately minimal:

• It doesn't care what chain you anchor to.
• It doesn't care which LLM model you use.
• It doesn't care whether the Runner is human-driven or fully autonomous.

As VaultMesh evolves, you can:

• Swap LLMs → Oracle stays the same; receipts remain valid.
• Swap anchor backends (OTS, Ethereum, Bitcoin, custom chain) → roots remain valid.
• Add automated agents (vm-copilot, OffSec agents, Mesh guardians) → they all just become more Experience Layer clients of the same Engine + Ledger.

The shape does not change.

---

8. Short Human Explanation (for README / Auditors)

VaultMesh treats every serious operation — a security drill, a compliance answer, an anchor event, a treasury transfer — as a small story with a beginning, middle, and end:

1. A human or agent expresses intent
2. An engine plans and executes the work, tracking state
3. The outcome is sealed into an append-only ledger, hashed, merklized, and anchored

This pattern — Intent → Engine → Receipt → Scroll → Anchor — is the same across all domains.

It's what makes VaultMesh composable, auditable, and explainable to both humans and machines.

---

9. Engine Specifications Index

The following engine specifications implement the Eternal Pattern:

Engine	Scroll	Description	Receipt Types
VAULTMESH-CONSOLE-ENGINE.md	Console	AI agent sessions, code operations, sovereign development	console_session_start, console_session_end, console_command, console_file_edit, console_tool_call, console_approval, console_git_commit, console_agent_spawn
VAULTMESH-MESH-ENGINE.md	Mesh	Federation topology, node management, routes, capabilities	mesh_node_join, mesh_node_leave, mesh_route_change, mesh_capability_grant, mesh_capability_revoke, mesh_topology_snapshot
VAULTMESH-OFFSEC-ENGINE.md	OffSec	Security incidents, red team engagements, vulnerability tracking	offsec_incident, offsec_redteam, offsec_vuln_discovery, offsec_remediation, offsec_threat_intel, offsec_forensic_snapshot
VAULTMESH-IDENTITY-ENGINE.md	Identity	DIDs, verifiable credentials, authentication, authorization	identity_did_create, identity_did_rotate, identity_did_revoke, identity_credential_issue, identity_credential_revoke, identity_auth_event, identity_authz_decision
VAULTMESH-OBSERVABILITY-ENGINE.md	Observability	Metrics, logs, traces, alerts, SLOs	obs_metric_snapshot, obs_log_batch, obs_trace_complete, obs_alert_fired, obs_alert_resolved, obs_slo_report, obs_anomaly_detected
VAULTMESH-AUTOMATION-ENGINE.md	Automation	n8n workflows, schedules, triggers, approvals	auto_workflow_register, auto_workflow_execute, auto_workflow_complete, auto_schedule_create, auto_trigger_fire, auto_approval_request, auto_approval_decision
VAULTMESH-PSI-FIELD-ENGINE.md	PsiField	Alchemical consciousness, phase transitions, transmutations	psi_phase_transition, psi_emergence_event, psi_transmutation, psi_resonance, psi_integration, psi_oracle_insight
VAULTMESH-FEDERATION-PROTOCOL.md	Federation	Cross-mesh trust, witness verification, cross-anchoring	fed_trust_proposal, fed_trust_established, fed_trust_revoked, fed_witness_event, fed_cross_anchor, fed_schema_sync
VAULTMESH-CONSTITUTIONAL-GOVERNANCE.md	Governance	Constitutional rules, amendments, enforcement, violations	gov_proposal, gov_vote, gov_ratification, gov_amendment, gov_executive_order, gov_violation, gov_enforcement

Implementation Reference:

Document	Description
VAULTMESH-IMPLEMENTATION-SCAFFOLDS.md	Rust structs, Python CLI, directory structure
VAULTMESH-MCP-SERVERS.md	MCP server implementations for Claude integration, tool definitions, gateway
VAULTMESH-DEPLOYMENT-MANIFESTS.md	Kubernetes manifests, Docker Compose, infrastructure-as-code
VAULTMESH-MONITORING-STACK.md	Prometheus config, Grafana dashboards, alerting rules, metrics
VAULTMESH-TESTING-FRAMEWORK.md	Property-based tests, integration tests, chaos tests, fixtures
VAULTMESH-MIGRATION-GUIDE.md	Version upgrades, migration scripts, rollback procedures

Each engine specification follows the same structure:

1. Scroll Definition (JSONL path, root file, receipt types)
2. Core Concepts (domain-specific entities)
3. Mapping to Eternal Pattern (L1, L2, L3)
4. Query Interface
5. Design Gate Checklist
6. Integration Points
7. Future Extensions