chore: initial import
This commit is contained in:
Sovereign
5
.gitignore
vendored
Normal file
5
.gitignore
vendored
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
/target
|
||||||
|
Cargo.lock
|
||||||
|
*.log
|
||||||
|
.env
|
||||||
|
/cc-ed25519.key
|
||||||
260
CHANGELOG.md
Normal file
260
CHANGELOG.md
Normal file
@@ -0,0 +1,260 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## V0.7.2 – Communication Layer
|
||||||
|
|
||||||
|
### Unified Event API
|
||||||
|
- Added `EventEnvelope` as the canonical message format for all comms events.
|
||||||
|
- New event kinds: `note`, `incident`, `ack`, `tag`, `resolve`.
|
||||||
|
- Events support optional `node_id` for node-specific or global scope.
|
||||||
|
- Author field tracks origin: "operator", "system", "vm-copilot", etc.
|
||||||
|
|
||||||
|
### New API Endpoints
|
||||||
|
- `POST /api/events` - Create a new event envelope.
|
||||||
|
- Server assigns `id` and `ts`.
|
||||||
|
- Author can be overridden via `X-VM-Author` header.
|
||||||
|
- `GET /api/events` - Query events with filtering:
|
||||||
|
- `?since=RFC3339` - Filter by timestamp.
|
||||||
|
- `?kind=note,incident` - Comma-separated kind filter.
|
||||||
|
- `?node_id=uuid` - Filter by node.
|
||||||
|
- `?limit=N` - Max results (default: 100).
|
||||||
|
|
||||||
|
### SSE Integration
|
||||||
|
- Envelope events broadcast via SSE using their `kind` as the event name.
|
||||||
|
- E.g., `event: note` for note envelopes.
|
||||||
|
|
||||||
|
### Persistence
|
||||||
|
- Events logged to `events.jsonl` in `$VAULTMESH_LOG_DIR`.
|
||||||
|
- Replayed on startup to restore in-memory state.
|
||||||
|
- Memory-bounded to 500 most recent envelopes.
|
||||||
|
|
||||||
|
### Usage
|
||||||
|
```bash
|
||||||
|
# Post a note
|
||||||
|
curl -X POST http://127.0.0.1:8088/api/events \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"kind":"note","node_id":"uuid","author":"operator","body":{"text":"Test","severity":"info"}}'
|
||||||
|
|
||||||
|
# Query events
|
||||||
|
curl "http://127.0.0.1:8088/api/events?since=2025-01-01T00:00:00Z&kind=note"
|
||||||
|
```
|
||||||
|
|
||||||
|
> V0.7.2 transforms CC from a control plane into a coordination plane.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## V0.7.1 – Mission Console
|
||||||
|
|
||||||
|
### NASA-Style Dashboard
|
||||||
|
- Added `GET /console` endpoint with 3-panel Mission Console UI.
|
||||||
|
- Global Mission Bar with fleet KPIs: Total, Healthy, Attention, Critical, Last Scan.
|
||||||
|
- Left panel: Clickable node list with live status pills (OK/ATTN/CRIT).
|
||||||
|
- Center panel: Selected node telemetry with metrics cards and per-node event timeline.
|
||||||
|
- Right panel: Attention summary chips, global scan findings, and global event feed.
|
||||||
|
|
||||||
|
### Live SSE Integration
|
||||||
|
- Real-time heartbeat indicator glow when nodes report in.
|
||||||
|
- Live status pill updates via `attention` events.
|
||||||
|
- Dynamic KPI recalculation without page refresh.
|
||||||
|
- Per-node and global event timelines populated from SSE stream.
|
||||||
|
|
||||||
|
### Visual Design
|
||||||
|
- Dark NASA-inspired theme (#05070a background).
|
||||||
|
- Monospace typography (JetBrains Mono).
|
||||||
|
- CSS Grid 3-column layout with sticky headers.
|
||||||
|
- Animated pulse for critical nodes.
|
||||||
|
- Color-coded severity indicators (green/yellow/red).
|
||||||
|
|
||||||
|
### Usage
|
||||||
|
```bash
|
||||||
|
# Open Mission Console
|
||||||
|
open http://127.0.0.1:8088/console
|
||||||
|
|
||||||
|
# Original table dashboard still at /
|
||||||
|
open http://127.0.0.1:8088/
|
||||||
|
```
|
||||||
|
|
||||||
|
> V0.7.1 transforms the basic table into an operator-grade control room.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## V0.7 – SSE Event Bus
|
||||||
|
|
||||||
|
### Real-Time Event Streaming
|
||||||
|
- Added `GET /events` endpoint for Server-Sent Events (SSE).
|
||||||
|
- Events are streamed in real-time as they occur (no page refresh needed).
|
||||||
|
- Broadcast channel (`tokio::sync::broadcast`) distributes events to all connected clients.
|
||||||
|
|
||||||
|
### Event Types
|
||||||
|
Four named SSE events are published:
|
||||||
|
- `heartbeat` – when a node sends a heartbeat.
|
||||||
|
- `scan` – when a sovereign-scan completes.
|
||||||
|
- `command` – when any command result is reported.
|
||||||
|
- `attention` – when a node's attention status changes.
|
||||||
|
|
||||||
|
### Wire Format
|
||||||
|
Each event is JSON-encoded with wire-efficient payloads:
|
||||||
|
```
|
||||||
|
event: heartbeat
|
||||||
|
data: {"ts":"2024-01-15T10:30:00Z","node_id":"...","hostname":"vault-01","cloudflare_ok":true,...}
|
||||||
|
|
||||||
|
event: attention
|
||||||
|
data: {"ts":"2024-01-15T10:30:00Z","node_id":"...","needs_attention":true,"reasons":["critical_findings"]}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Dashboard Integration
|
||||||
|
- Minimal JS probe added to dashboard and node detail pages.
|
||||||
|
- Events are logged to browser console (`[SSE][HB]`, `[SSE][SCAN]`, etc.) for debugging.
|
||||||
|
- Foundation for V0.7.1 Mission Console with live row updates.
|
||||||
|
|
||||||
|
### Keepalive
|
||||||
|
- SSE connection sends keepalive every 15 seconds to prevent timeouts.
|
||||||
|
- Clients that lag behind receive a warning and skip missed events.
|
||||||
|
|
||||||
|
### Testing
|
||||||
|
```bash
|
||||||
|
# Connect to SSE stream
|
||||||
|
curl -N http://127.0.0.1:8088/events
|
||||||
|
|
||||||
|
# Or open dashboard in browser, open DevTools Console
|
||||||
|
# Trigger heartbeat → see [SSE][HB] in console
|
||||||
|
```
|
||||||
|
|
||||||
|
> V0.7 provides the real-time infrastructure for live dashboards without page refresh.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## V0.6.1 – Log Tools
|
||||||
|
|
||||||
|
### CLI Log Commands
|
||||||
|
- Added `logs view` subcommand to query event logs with filters:
|
||||||
|
- `--kind heartbeats|scans|commands` – filter by event type
|
||||||
|
- `--node NODE_ID` – filter by node UUID
|
||||||
|
- `--since 1h|24h|7d|30m` – time-based filtering
|
||||||
|
- `--min-severity info|low|medium|high|critical` – severity filter (scans only)
|
||||||
|
- `-n|--limit N` – number of events to show (default: 20)
|
||||||
|
- Added `logs tail` subcommand for real-time log following
|
||||||
|
- Added `logs stats` subcommand for per-node and per-kind statistics
|
||||||
|
- Server mode unchanged: `vm-cc` or `vm-cc serve` starts HTTP server
|
||||||
|
|
||||||
|
### Files Added
|
||||||
|
- `command-center/src/cli.rs` – CLI argument parsing with clap
|
||||||
|
- `command-center/src/logs.rs` – Log reading and query logic
|
||||||
|
|
||||||
|
### Usage Examples
|
||||||
|
```bash
|
||||||
|
vm-cc logs view # last 20 events
|
||||||
|
vm-cc logs view --kind scans # last 20 scans
|
||||||
|
vm-cc logs view --since 1h --kind heartbeats
|
||||||
|
vm-cc logs view --kind scans --min-severity high
|
||||||
|
vm-cc logs stats # event counts by node
|
||||||
|
vm-cc logs tail # follow all logs
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## V0.6 – Append-Only Persistence
|
||||||
|
|
||||||
|
### Event Logging
|
||||||
|
- All CC state changes are now persisted to append-only JSONL log files.
|
||||||
|
- Log files are stored in `$VAULTMESH_LOG_DIR` (default: `/var/lib/vaultmesh/cc-logs`).
|
||||||
|
- Three event types:
|
||||||
|
- `heartbeats.jsonl` – HeartbeatEvent for each agent heartbeat.
|
||||||
|
- `scans.jsonl` – ScanEvent when sovereign-scan completes successfully.
|
||||||
|
- `commands.jsonl` – CommandEvent when agent reports any command result.
|
||||||
|
|
||||||
|
### Replay on Startup
|
||||||
|
- CC replays all log files on startup to reconstruct in-memory state.
|
||||||
|
- Nodes, heartbeat history, scan results, and command history survive restarts.
|
||||||
|
- Log replay counts are printed at startup for visibility.
|
||||||
|
|
||||||
|
### Ledger-Ready Format
|
||||||
|
- JSONL format is ledger-native; each line is a self-contained JSON object.
|
||||||
|
- Designed as foundation for V0.8 Ledger Bridge (Merkle tree over logs).
|
||||||
|
- No external database dependencies (SQLite not required).
|
||||||
|
|
||||||
|
### Configuration
|
||||||
|
|
||||||
|
| Variable | Default | Description |
|
||||||
|
|----------------------|------------------------------|----------------------------|
|
||||||
|
| `VAULTMESH_LOG_DIR` | `/var/lib/vaultmesh/cc-logs` | Directory for event logs |
|
||||||
|
|
||||||
|
> With V0.6, CC state is durable across restarts. The append-only design provides an audit trail of all fleet events.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## V0.5 – Fleet Orchestrator
|
||||||
|
|
||||||
|
### Scan Orchestrator
|
||||||
|
- Background scheduler runs every `VAULTMESH_SCHEDULER_TICK_SECONDS` (default: `300s`).
|
||||||
|
- Automatically queues `sovereign-scan` commands for nodes whose last scan is older than
|
||||||
|
`VAULTMESH_SCAN_INTERVAL_HOURS` (default: `24h`).
|
||||||
|
- Commands are signed with the CC Ed25519 key and added to the per-node command queue for agent pickup.
|
||||||
|
|
||||||
|
### Staleness / Drift Detection
|
||||||
|
- New **Attention** column on the main dashboard for at-a-glance fleet health.
|
||||||
|
- Attention reasons:
|
||||||
|
- `never_scanned` – node has no scan history.
|
||||||
|
- `scan_stale` – last scan older than `VAULTMESH_SCAN_STALE_HOURS`.
|
||||||
|
- `heartbeat_stale` – no heartbeat within `VAULTMESH_HEARTBEAT_STALE_MINUTES`.
|
||||||
|
- `critical_findings` – last scan reported critical vulnerabilities.
|
||||||
|
- `high_findings` – last scan reported high-severity vulnerabilities.
|
||||||
|
- `cloudflare_down` – Cloudflare service flag is `false`.
|
||||||
|
- `services_down` – services flag is `false`.
|
||||||
|
- Visual cues:
|
||||||
|
- `.attn-ok` – green background, label `OK` for healthy nodes.
|
||||||
|
- `.attn-bad` – red background, comma-separated reasons for nodes needing attention.
|
||||||
|
|
||||||
|
### Command Policy Enforcement
|
||||||
|
- Introduced `CommandPolicy` with:
|
||||||
|
- `global_allowed` commands.
|
||||||
|
- Optional `per_profile` allowlists keyed by OS profile.
|
||||||
|
- Default allowed commands:
|
||||||
|
- `service-status`
|
||||||
|
- `tail-journal`
|
||||||
|
- `sovereign-scan`
|
||||||
|
- `restart-service`
|
||||||
|
- Web UI returns HTTP `403 Forbidden` for disallowed commands.
|
||||||
|
- Scheduler respects policy and will not auto-queue scans if they are disallowed for the node's profile.
|
||||||
|
|
||||||
|
### Configuration (Environment Variables)
|
||||||
|
|
||||||
|
| Variable | Default | Description |
|
||||||
|
|------------------------------------|---------|-------------------------------------|
|
||||||
|
| `VAULTMESH_SCAN_INTERVAL_HOURS` | `24` | How often each node should be scanned |
|
||||||
|
| `VAULTMESH_SCAN_STALE_HOURS` | `48` | Scan staleness threshold |
|
||||||
|
| `VAULTMESH_HEARTBEAT_STALE_MINUTES`| `10` | Heartbeat staleness threshold |
|
||||||
|
| `VAULTMESH_SCHEDULER_TICK_SECONDS` | `300` | Scheduler loop interval |
|
||||||
|
|
||||||
|
> With V0.5, the fleet tends itself; operators only need to investigate nodes marked red in the Attention column.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## V0.4.1 – Scan Status UI
|
||||||
|
|
||||||
|
- Dashboard columns: Last Scan, Crit/High, Source (REAL/MOCK badges).
|
||||||
|
- Parse `sovereign-scan` stdout to track `LastScan` in state.
|
||||||
|
- Visual indicators for critical (red) and high (orange) findings.
|
||||||
|
|
||||||
|
## V0.4 – Sovereign Scan Integration
|
||||||
|
|
||||||
|
- Node agent executes `sovereign-scan` command.
|
||||||
|
- Scan results reported via command-result API.
|
||||||
|
- ProofChain receipts written to `/var/lib/vaultmesh/proofchain/`.
|
||||||
|
|
||||||
|
## V0.3 – Signed Commands
|
||||||
|
|
||||||
|
- Ed25519 command signing (CC signs, agent verifies).
|
||||||
|
- Command queue per node with nonce replay protection.
|
||||||
|
- Commands: `service-status`, `restart-service`, `tail-journal`.
|
||||||
|
|
||||||
|
## V0.2 – Node Metrics
|
||||||
|
|
||||||
|
- Added system metrics to heartbeat (load, memory, disk).
|
||||||
|
- Node detail page with metrics cards.
|
||||||
|
- Heartbeat history tracking.
|
||||||
|
|
||||||
|
## V0.1 – Initial Release
|
||||||
|
|
||||||
|
- Command Center with HTML dashboard.
|
||||||
|
- Node Agent with heartbeat loop.
|
||||||
|
- Basic health monitoring (`cloudflare_ok`, `services_ok`).
|
||||||
7
Cargo.toml
Normal file
7
Cargo.toml
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
[workspace]
|
||||||
|
members = [
|
||||||
|
"command-center",
|
||||||
|
"node-agent",
|
||||||
|
]
|
||||||
|
|
||||||
|
resolver = "2"
|
||||||
155
README.md
Normal file
155
README.md
Normal file
@@ -0,0 +1,155 @@
|
|||||||
|
# VaultMesh Command Center
|
||||||
|
|
||||||
|
Minimal, no-bloat control plane for VaultMesh nodes.
|
||||||
|
|
||||||
|
- Rust backend (Axum) with server-rendered HTML (HTMX-ready)
|
||||||
|
- Node agent (Rust daemon) that runs on each VaultMesh node
|
||||||
|
- Zero external infra deps (no Redis, Kafka, k8s)
|
||||||
|
- Cloudflare-native: fronted by Cloudflare Tunnel + Access
|
||||||
|
|
||||||
|
## Repository Layout
|
||||||
|
|
||||||
|
```text
|
||||||
|
vaultmesh-command-center/
|
||||||
|
├── Cargo.toml # Workspace manifest
|
||||||
|
├── README.md
|
||||||
|
├── .gitignore
|
||||||
|
├── command-center/ # Backend + Web UI
|
||||||
|
│ ├── Cargo.toml
|
||||||
|
│ └── src/
|
||||||
|
│ ├── main.rs # Entry point, server setup
|
||||||
|
│ ├── routes.rs # HTTP handlers
|
||||||
|
│ └── state.rs # AppState, in-memory node store
|
||||||
|
├── node-agent/ # Daemon for each VaultMesh node
|
||||||
|
│ ├── Cargo.toml
|
||||||
|
│ └── src/
|
||||||
|
│ ├── main.rs # Heartbeat loop
|
||||||
|
│ └── config.rs # Env config loader
|
||||||
|
├── docs/
|
||||||
|
│ ├── ARCHITECTURE.md # How it all fits together
|
||||||
|
│ └── NODE_AGENT_CONTRACT.md # Agent API spec
|
||||||
|
└── systemd/
|
||||||
|
├── vaultmesh-command-center.service
|
||||||
|
└── vaultmesh-node-agent.service
|
||||||
|
```
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Clone and build
|
||||||
|
git clone <your-url> vaultmesh-command-center
|
||||||
|
cd vaultmesh-command-center
|
||||||
|
|
||||||
|
# Run the command center locally
|
||||||
|
cd command-center
|
||||||
|
RUST_LOG=info cargo run
|
||||||
|
# listens on 127.0.0.1:8088
|
||||||
|
|
||||||
|
# In another terminal, run the agent (pointing at local CC)
|
||||||
|
cd ../node-agent
|
||||||
|
RUST_LOG=info VAULTMESH_OS_PROFILE=ArchVault cargo run
|
||||||
|
```
|
||||||
|
|
||||||
|
Then:
|
||||||
|
- Put the Command Center behind a Cloudflare Tunnel.
|
||||||
|
- Protect it with Cloudflare Access.
|
||||||
|
- Install the node agent as a systemd service on each VaultMesh node.
|
||||||
|
|
||||||
|
## Deployment
|
||||||
|
|
||||||
|
### Command Center
|
||||||
|
|
||||||
|
1. Build release binary:
|
||||||
|
```bash
|
||||||
|
cargo build --release -p vaultmesh-command-center
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Copy to `/usr/local/bin/`:
|
||||||
|
```bash
|
||||||
|
sudo cp target/release/vaultmesh-command-center /usr/local/bin/
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Install systemd unit:
|
||||||
|
```bash
|
||||||
|
sudo cp systemd/vaultmesh-command-center.service /etc/systemd/system/
|
||||||
|
sudo systemctl daemon-reload
|
||||||
|
sudo systemctl enable --now vaultmesh-command-center
|
||||||
|
```
|
||||||
|
|
||||||
|
4. Configure Cloudflare Tunnel to point at `http://127.0.0.1:8088`.
|
||||||
|
|
||||||
|
### Node Agent
|
||||||
|
|
||||||
|
1. Build release binary:
|
||||||
|
```bash
|
||||||
|
cargo build --release -p vaultmesh-node-agent
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Copy to each node:
|
||||||
|
```bash
|
||||||
|
sudo cp target/release/vaultmesh-node-agent /usr/local/bin/
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Create environment file `/etc/vaultmesh/agent.env`:
|
||||||
|
```bash
|
||||||
|
VAULTMESH_CC_URL=https://cc.your-domain.example
|
||||||
|
VAULTMESH_OS_PROFILE=ArchVault
|
||||||
|
VAULTMESH_ROOT=/var/lib/vaultmesh
|
||||||
|
VAULTMESH_HEARTBEAT_SECS=30
|
||||||
|
```
|
||||||
|
|
||||||
|
4. Install systemd unit:
|
||||||
|
```bash
|
||||||
|
sudo cp systemd/vaultmesh-node-agent.service /etc/systemd/system/
|
||||||
|
sudo systemctl daemon-reload
|
||||||
|
sudo systemctl enable --now vaultmesh-node-agent
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Endpoints
|
||||||
|
|
||||||
|
| Method | Path | Description |
|
||||||
|
|--------|------------------------------------|------------------------------------------|
|
||||||
|
| GET | `/` | HTML dashboard showing all nodes |
|
||||||
|
| GET | `/nodes` | JSON array of all node heartbeats |
|
||||||
|
| GET | `/nodes/:id` | Node detail page (HTML) |
|
||||||
|
| POST | `/nodes/:id/commands` | Queue command for node (web form or API) |
|
||||||
|
| POST | `/api/agent/heartbeat` | Agent heartbeat endpoint |
|
||||||
|
| GET | `/api/agent/commands?node_id=<uuid>` | Agent polls for pending commands |
|
||||||
|
| POST | `/api/agent/command-result` | Agent reports command execution result |
|
||||||
|
|
||||||
|
## Fleet Operation Model (V0.5)
|
||||||
|
|
||||||
|
**Green fleet:**
|
||||||
|
|
||||||
|
- Attention = `OK`.
|
||||||
|
- Heartbeats fresh (age < `VAULTMESH_HEARTBEAT_STALE_MINUTES`).
|
||||||
|
- Last scan age < `VAULTMESH_SCAN_STALE_HOURS`.
|
||||||
|
- No critical or high findings.
|
||||||
|
|
||||||
|
**Yellow/Red fleet:**
|
||||||
|
|
||||||
|
Check Attention reasons in order:
|
||||||
|
|
||||||
|
1. `heartbeat_stale` → connectivity / host issue.
|
||||||
|
2. `cloudflare_down` / `services_down` → control plane or local service failure.
|
||||||
|
3. `never_scanned` / `scan_stale` → wait for scheduler or trigger manual scan (if policy allows).
|
||||||
|
4. `critical_findings` / `high_findings` → prioritize remediation on that node.
|
||||||
|
|
||||||
|
**Policy guardrail:**
|
||||||
|
|
||||||
|
- Disallowed commands are blocked with HTTP 403.
|
||||||
|
- Scheduler respects policy; `sovereign-scan` must be allowed for a profile to be auto-queued.
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
| Variable | Default | Description |
|
||||||
|
|------------------------------------|-------------------|------------------------------|
|
||||||
|
| `VAULTMESH_SCAN_INTERVAL_HOURS` | `24` | Auto-scan interval |
|
||||||
|
| `VAULTMESH_SCAN_STALE_HOURS` | `48` | Scan staleness threshold |
|
||||||
|
| `VAULTMESH_HEARTBEAT_STALE_MINUTES`| `10` | Heartbeat staleness |
|
||||||
|
| `VAULTMESH_SCHEDULER_TICK_SECONDS` | `300` | Scheduler tick interval |
|
||||||
|
| `VAULTMESH_CC_KEY_PATH` | `cc-ed25519.key` | Command signing key path |
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
MIT
|
||||||
23
command-center/Cargo.toml
Normal file
23
command-center/Cargo.toml
Normal file
@@ -0,0 +1,23 @@
|
|||||||
|
[package]
|
||||||
|
name = "vaultmesh-command-center"
|
||||||
|
version = "0.1.0"
|
||||||
|
edition = "2021"
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
axum = { version = "0.7", features = ["macros", "json"] }
|
||||||
|
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
|
||||||
|
serde = { version = "1", features = ["derive"] }
|
||||||
|
serde_json = "1"
|
||||||
|
tower = "0.4"
|
||||||
|
tower-http = { version = "0.5", features = ["trace"] }
|
||||||
|
tracing = "0.1"
|
||||||
|
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
||||||
|
uuid = { version = "1", features = ["v4", "serde"] }
|
||||||
|
time = { version = "0.3", features = ["serde", "macros", "formatting", "parsing"] }
|
||||||
|
ed25519-dalek = { version = "2", features = ["rand_core"] }
|
||||||
|
rand = "0.8"
|
||||||
|
base64 = "0.22"
|
||||||
|
anyhow = "1"
|
||||||
|
clap = { version = "4", features = ["derive"] }
|
||||||
|
async-stream = "0.3"
|
||||||
|
futures-core = "0.3"
|
||||||
61
command-center/src/cli.rs
Normal file
61
command-center/src/cli.rs
Normal file
@@ -0,0 +1,61 @@
|
|||||||
|
use clap::{Parser, Subcommand};
|
||||||
|
|
||||||
|
#[derive(Parser)]
|
||||||
|
#[command(name = "vm-cc")]
|
||||||
|
#[command(about = "VaultMesh Command Center")]
|
||||||
|
pub struct Cli {
|
||||||
|
/// Subcommand to run. If omitted, starts the HTTP server.
|
||||||
|
#[command(subcommand)]
|
||||||
|
pub command: Option<Commands>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Subcommand)]
|
||||||
|
pub enum Commands {
|
||||||
|
/// View and query event logs
|
||||||
|
Logs {
|
||||||
|
#[command(subcommand)]
|
||||||
|
action: LogsAction,
|
||||||
|
},
|
||||||
|
/// Start the HTTP server (default if no subcommand is provided)
|
||||||
|
Serve,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Subcommand)]
|
||||||
|
pub enum LogsAction {
|
||||||
|
/// View recent log events
|
||||||
|
View {
|
||||||
|
/// Event kind: heartbeats, scans, commands (default: all)
|
||||||
|
#[arg(short, long)]
|
||||||
|
kind: Option<String>,
|
||||||
|
|
||||||
|
/// Filter by node ID (UUID)
|
||||||
|
#[arg(short, long)]
|
||||||
|
node: Option<String>,
|
||||||
|
|
||||||
|
/// Time filter: 1h, 24h, 7d, 30m, etc.
|
||||||
|
#[arg(short, long)]
|
||||||
|
since: Option<String>,
|
||||||
|
|
||||||
|
/// Minimum severity for scans: info, low, medium, high, critical
|
||||||
|
#[arg(long)]
|
||||||
|
min_severity: Option<String>,
|
||||||
|
|
||||||
|
/// Number of events to show (default: 20)
|
||||||
|
#[arg(short = 'n', long, default_value = "20")]
|
||||||
|
limit: usize,
|
||||||
|
},
|
||||||
|
|
||||||
|
/// Follow logs in real-time
|
||||||
|
Tail {
|
||||||
|
/// Event kind to follow
|
||||||
|
#[arg(short, long)]
|
||||||
|
kind: Option<String>,
|
||||||
|
|
||||||
|
/// Filter by node ID (UUID)
|
||||||
|
#[arg(short, long)]
|
||||||
|
node: Option<String>,
|
||||||
|
},
|
||||||
|
|
||||||
|
/// Show log statistics
|
||||||
|
Stats,
|
||||||
|
}
|
||||||
396
command-center/src/logs.rs
Normal file
396
command-center/src/logs.rs
Normal file
@@ -0,0 +1,396 @@
|
|||||||
|
// V0.6.1: CLI log reading and query commands
|
||||||
|
|
||||||
|
use crate::state::{CommandEvent, HeartbeatEvent, ScanEvent};
|
||||||
|
use std::collections::HashMap;
|
||||||
|
use std::io::{BufRead, BufReader};
|
||||||
|
use std::path::PathBuf;
|
||||||
|
use time::{Duration, OffsetDateTime};
|
||||||
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
/// Reader for JSONL log files.
|
||||||
|
pub struct LogReader {
|
||||||
|
dir: PathBuf,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl LogReader {
|
||||||
|
pub fn new() -> Self {
|
||||||
|
let dir = std::env::var("VAULTMESH_LOG_DIR")
|
||||||
|
.map(PathBuf::from)
|
||||||
|
.unwrap_or_else(|_| PathBuf::from("/var/lib/vaultmesh/cc-logs"));
|
||||||
|
LogReader { dir }
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn read_heartbeats(&self) -> Vec<HeartbeatEvent> {
|
||||||
|
self.read_jsonl("heartbeats.jsonl")
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn read_scans(&self) -> Vec<ScanEvent> {
|
||||||
|
self.read_jsonl("scans.jsonl")
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn read_commands(&self) -> Vec<CommandEvent> {
|
||||||
|
self.read_jsonl("commands.jsonl")
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_jsonl<T: serde::de::DeserializeOwned>(&self, filename: &str) -> Vec<T> {
|
||||||
|
let path = self.dir.join(filename);
|
||||||
|
if !path.exists() {
|
||||||
|
return vec![];
|
||||||
|
}
|
||||||
|
|
||||||
|
let file = match std::fs::File::open(&path) {
|
||||||
|
Ok(f) => f,
|
||||||
|
Err(_) => return vec![],
|
||||||
|
};
|
||||||
|
|
||||||
|
BufReader::new(file)
|
||||||
|
.lines()
|
||||||
|
.filter_map(|line| line.ok())
|
||||||
|
.filter(|line| !line.trim().is_empty())
|
||||||
|
.filter_map(|line| serde_json::from_str(&line).ok())
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Parse a duration string like "1h", "24h", "7d", "30m".
|
||||||
|
pub fn parse_duration(s: &str) -> Option<Duration> {
|
||||||
|
let s = s.trim().to_lowercase();
|
||||||
|
if s.ends_with('h') {
|
||||||
|
s[..s.len() - 1].parse::<i64>().ok().map(Duration::hours)
|
||||||
|
} else if s.ends_with('d') {
|
||||||
|
s[..s.len() - 1].parse::<i64>().ok().map(Duration::days)
|
||||||
|
} else if s.ends_with('m') {
|
||||||
|
s[..s.len() - 1].parse::<i64>().ok().map(Duration::minutes)
|
||||||
|
} else {
|
||||||
|
None
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Convert severity string to a rank for filtering.
|
||||||
|
pub fn severity_to_rank(s: &str) -> u8 {
|
||||||
|
match s.to_lowercase().as_str() {
|
||||||
|
"critical" => 5,
|
||||||
|
"high" => 4,
|
||||||
|
"medium" => 3,
|
||||||
|
"low" => 2,
|
||||||
|
"info" => 1,
|
||||||
|
_ => 0,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// View recent log events with filtering.
|
||||||
|
pub fn cmd_logs_view(
|
||||||
|
kind: Option<String>,
|
||||||
|
node: Option<String>,
|
||||||
|
since: Option<String>,
|
||||||
|
min_severity: Option<String>,
|
||||||
|
limit: usize,
|
||||||
|
) {
|
||||||
|
let reader = LogReader::new();
|
||||||
|
let now = OffsetDateTime::now_utc();
|
||||||
|
|
||||||
|
let since_ts = since.as_ref().and_then(|s| parse_duration(s)).map(|d| now - d);
|
||||||
|
|
||||||
|
let node_filter: Option<Uuid> = node.as_ref().and_then(|s| s.parse().ok());
|
||||||
|
|
||||||
|
let min_sev_rank = min_severity.as_ref().map(|s| severity_to_rank(s)).unwrap_or(0);
|
||||||
|
|
||||||
|
// Collect events based on kind
|
||||||
|
let mut events: Vec<(OffsetDateTime, String)> = vec![];
|
||||||
|
|
||||||
|
if kind.is_none() || kind.as_deref() == Some("heartbeats") {
|
||||||
|
for e in reader.read_heartbeats() {
|
||||||
|
if let Some(ts) = since_ts {
|
||||||
|
if e.ts < ts {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(nf) = node_filter {
|
||||||
|
if e.node_id != nf {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let line = format!(
|
||||||
|
"[HB] {} {} {} cf={} svc={}",
|
||||||
|
format_ts(e.ts),
|
||||||
|
e.hostname,
|
||||||
|
e.os_profile,
|
||||||
|
if e.heartbeat.cloudflare_ok { "OK" } else { "DOWN" },
|
||||||
|
if e.heartbeat.services_ok { "OK" } else { "DOWN" }
|
||||||
|
);
|
||||||
|
events.push((e.ts, line));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if kind.is_none() || kind.as_deref() == Some("scans") {
|
||||||
|
for e in reader.read_scans() {
|
||||||
|
if let Some(ts) = since_ts {
|
||||||
|
if e.ts < ts {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(nf) = node_filter {
|
||||||
|
if e.node_id != nf {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let max_sev = if e.summary.critical > 0 {
|
||||||
|
5
|
||||||
|
} else if e.summary.high > 0 {
|
||||||
|
4
|
||||||
|
} else if e.summary.medium > 0 {
|
||||||
|
3
|
||||||
|
} else if e.summary.low > 0 {
|
||||||
|
2
|
||||||
|
} else {
|
||||||
|
1
|
||||||
|
};
|
||||||
|
if max_sev < min_sev_rank {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let source = if e.findings_real { "REAL" } else { "MOCK" };
|
||||||
|
let line = format!(
|
||||||
|
"[SCAN] {} C:{} H:{} M:{} L:{} [{}]",
|
||||||
|
format_ts(e.ts),
|
||||||
|
e.summary.critical,
|
||||||
|
e.summary.high,
|
||||||
|
e.summary.medium,
|
||||||
|
e.summary.low,
|
||||||
|
source
|
||||||
|
);
|
||||||
|
events.push((e.ts, line));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if kind.is_none() || kind.as_deref() == Some("commands") {
|
||||||
|
for e in reader.read_commands() {
|
||||||
|
if let Some(ts) = since_ts {
|
||||||
|
if e.ts < ts {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(nf) = node_filter {
|
||||||
|
if e.node_id != nf {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let line = format!(
|
||||||
|
"[CMD] {} {} status={} exit={:?}",
|
||||||
|
format_ts(e.ts),
|
||||||
|
e.cmd,
|
||||||
|
e.status,
|
||||||
|
e.exit_code
|
||||||
|
);
|
||||||
|
events.push((e.ts, line));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if events.is_empty() {
|
||||||
|
println!("No events found.");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sort by timestamp descending, take limit
|
||||||
|
events.sort_by(|a, b| b.0.cmp(&a.0));
|
||||||
|
events.truncate(limit);
|
||||||
|
|
||||||
|
// Print in chronological order
|
||||||
|
for (_, line) in events.into_iter().rev() {
|
||||||
|
println!("{}", line);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Show log statistics.
|
||||||
|
pub fn cmd_logs_stats() {
|
||||||
|
let reader = LogReader::new();
|
||||||
|
|
||||||
|
let heartbeats = reader.read_heartbeats();
|
||||||
|
let scans = reader.read_scans();
|
||||||
|
let commands = reader.read_commands();
|
||||||
|
|
||||||
|
println!("=== Log Statistics ===\n");
|
||||||
|
println!("Heartbeats: {}", heartbeats.len());
|
||||||
|
println!("Scans: {}", scans.len());
|
||||||
|
println!("Commands: {}", commands.len());
|
||||||
|
|
||||||
|
// Count by node
|
||||||
|
let mut by_node: HashMap<Uuid, (usize, usize, usize, String)> = HashMap::new();
|
||||||
|
|
||||||
|
for e in &heartbeats {
|
||||||
|
let entry = by_node.entry(e.node_id).or_insert((0, 0, 0, e.hostname.clone()));
|
||||||
|
entry.0 += 1;
|
||||||
|
}
|
||||||
|
for e in &scans {
|
||||||
|
let entry = by_node.entry(e.node_id).or_insert((0, 0, 0, String::new()));
|
||||||
|
entry.1 += 1;
|
||||||
|
}
|
||||||
|
for e in &commands {
|
||||||
|
let entry = by_node.entry(e.node_id).or_insert((0, 0, 0, String::new()));
|
||||||
|
entry.2 += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
if !by_node.is_empty() {
|
||||||
|
println!("\n=== By Node ===\n");
|
||||||
|
println!(
|
||||||
|
"{:<38} {:<16} {:>8} {:>8} {:>8}",
|
||||||
|
"Node ID", "Hostname", "HB", "Scans", "Cmds"
|
||||||
|
);
|
||||||
|
println!("{:-<80}", "");
|
||||||
|
for (node_id, (hb, sc, cmd, hostname)) in by_node {
|
||||||
|
println!(
|
||||||
|
"{:<38} {:<16} {:>8} {:>8} {:>8}",
|
||||||
|
node_id, hostname, hb, sc, cmd
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Follow logs in real-time (tail -f style).
|
||||||
|
pub fn cmd_logs_tail(kind: Option<String>, node: Option<String>) {
|
||||||
|
use std::io::Seek;
|
||||||
|
use std::thread;
|
||||||
|
use std::time::Duration as StdDuration;
|
||||||
|
|
||||||
|
let node_filter: Option<Uuid> = node.as_ref().and_then(|s| s.parse().ok());
|
||||||
|
|
||||||
|
// Track file positions
|
||||||
|
let mut hb_pos: u64 = 0;
|
||||||
|
let mut scan_pos: u64 = 0;
|
||||||
|
let mut cmd_pos: u64 = 0;
|
||||||
|
|
||||||
|
// Seek to end of existing files
|
||||||
|
let log_dir = std::env::var("VAULTMESH_LOG_DIR")
|
||||||
|
.map(PathBuf::from)
|
||||||
|
.unwrap_or_else(|_| PathBuf::from("/var/lib/vaultmesh/cc-logs"));
|
||||||
|
|
||||||
|
if let Ok(mut f) = std::fs::File::open(log_dir.join("heartbeats.jsonl")) {
|
||||||
|
hb_pos = f.seek(std::io::SeekFrom::End(0)).unwrap_or(0);
|
||||||
|
}
|
||||||
|
if let Ok(mut f) = std::fs::File::open(log_dir.join("scans.jsonl")) {
|
||||||
|
scan_pos = f.seek(std::io::SeekFrom::End(0)).unwrap_or(0);
|
||||||
|
}
|
||||||
|
if let Ok(mut f) = std::fs::File::open(log_dir.join("commands.jsonl")) {
|
||||||
|
cmd_pos = f.seek(std::io::SeekFrom::End(0)).unwrap_or(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
println!("Following logs (Ctrl+C to stop)...\n");
|
||||||
|
|
||||||
|
loop {
|
||||||
|
// Check heartbeats
|
||||||
|
if kind.is_none() || kind.as_deref() == Some("heartbeats") {
|
||||||
|
if let Some((events, new_pos)) =
|
||||||
|
read_new_lines::<HeartbeatEvent>(&log_dir.join("heartbeats.jsonl"), hb_pos)
|
||||||
|
{
|
||||||
|
hb_pos = new_pos;
|
||||||
|
for e in events {
|
||||||
|
if let Some(nf) = node_filter {
|
||||||
|
if e.node_id != nf {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
println!(
|
||||||
|
"[HB] {} {} {}",
|
||||||
|
format_ts(e.ts),
|
||||||
|
e.hostname,
|
||||||
|
e.os_profile
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check scans
|
||||||
|
if kind.is_none() || kind.as_deref() == Some("scans") {
|
||||||
|
if let Some((events, new_pos)) =
|
||||||
|
read_new_lines::<ScanEvent>(&log_dir.join("scans.jsonl"), scan_pos)
|
||||||
|
{
|
||||||
|
scan_pos = new_pos;
|
||||||
|
for e in events {
|
||||||
|
if let Some(nf) = node_filter {
|
||||||
|
if e.node_id != nf {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let source = if e.findings_real { "REAL" } else { "MOCK" };
|
||||||
|
println!(
|
||||||
|
"[SCAN] {} C:{} H:{} M:{} L:{} [{}]",
|
||||||
|
format_ts(e.ts),
|
||||||
|
e.summary.critical,
|
||||||
|
e.summary.high,
|
||||||
|
e.summary.medium,
|
||||||
|
e.summary.low,
|
||||||
|
source
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check commands
|
||||||
|
if kind.is_none() || kind.as_deref() == Some("commands") {
|
||||||
|
if let Some((events, new_pos)) =
|
||||||
|
read_new_lines::<CommandEvent>(&log_dir.join("commands.jsonl"), cmd_pos)
|
||||||
|
{
|
||||||
|
cmd_pos = new_pos;
|
||||||
|
for e in events {
|
||||||
|
if let Some(nf) = node_filter {
|
||||||
|
if e.node_id != nf {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
println!(
|
||||||
|
"[CMD] {} {} status={}",
|
||||||
|
format_ts(e.ts),
|
||||||
|
e.cmd,
|
||||||
|
e.status
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
thread::sleep(StdDuration::from_secs(1));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Read new lines from a file starting at a given position.
|
||||||
|
fn read_new_lines<T: serde::de::DeserializeOwned>(
|
||||||
|
path: &PathBuf,
|
||||||
|
start_pos: u64,
|
||||||
|
) -> Option<(Vec<T>, u64)> {
|
||||||
|
use std::io::Seek;
|
||||||
|
|
||||||
|
let mut file = std::fs::File::open(path).ok()?;
|
||||||
|
let end_pos = file.seek(std::io::SeekFrom::End(0)).ok()?;
|
||||||
|
|
||||||
|
if end_pos <= start_pos {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
|
||||||
|
file.seek(std::io::SeekFrom::Start(start_pos)).ok()?;
|
||||||
|
let reader = BufReader::new(file);
|
||||||
|
let mut events = Vec::new();
|
||||||
|
|
||||||
|
for line in reader.lines() {
|
||||||
|
if let Ok(line) = line {
|
||||||
|
if line.trim().is_empty() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if let Ok(event) = serde_json::from_str(&line) {
|
||||||
|
events.push(event);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Some((events, end_pos))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Format timestamp for display.
|
||||||
|
fn format_ts(ts: OffsetDateTime) -> String {
|
||||||
|
format!(
|
||||||
|
"{:04}-{:02}-{:02} {:02}:{:02}:{:02}",
|
||||||
|
ts.year(),
|
||||||
|
ts.month() as u8,
|
||||||
|
ts.day(),
|
||||||
|
ts.hour(),
|
||||||
|
ts.minute(),
|
||||||
|
ts.second()
|
||||||
|
)
|
||||||
|
}
|
||||||
174
command-center/src/main.rs
Normal file
174
command-center/src/main.rs
Normal file
@@ -0,0 +1,174 @@
|
|||||||
|
mod cli;
|
||||||
|
mod logs;
|
||||||
|
mod routes;
|
||||||
|
mod state;
|
||||||
|
|
||||||
|
use crate::cli::{Cli, Commands, LogsAction};
|
||||||
|
use crate::routes::app;
|
||||||
|
use crate::state::{AppState, CommandPayload, SignedCommand};
|
||||||
|
use base64::Engine;
|
||||||
|
use clap::Parser;
|
||||||
|
use ed25519_dalek::{Signer, SigningKey};
|
||||||
|
use std::net::SocketAddr;
|
||||||
|
use std::path::Path;
|
||||||
|
use std::time::Duration;
|
||||||
|
use time::OffsetDateTime;
|
||||||
|
use tokio::net::TcpListener;
|
||||||
|
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};
|
||||||
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
/// Load an existing Ed25519 signing key or generate a new one.
|
||||||
|
fn load_or_init_signing_key() -> SigningKey {
|
||||||
|
let key_path = std::env::var("VAULTMESH_CC_KEY_PATH")
|
||||||
|
.unwrap_or_else(|_| "cc-ed25519.key".into());
|
||||||
|
let path = Path::new(&key_path);
|
||||||
|
|
||||||
|
if path.exists() {
|
||||||
|
let bytes = std::fs::read(path).expect("Failed to read signing key");
|
||||||
|
let key_bytes: [u8; 32] = bytes
|
||||||
|
.try_into()
|
||||||
|
.expect("Signing key must be exactly 32 bytes");
|
||||||
|
SigningKey::from_bytes(&key_bytes)
|
||||||
|
} else {
|
||||||
|
// Generate a new key
|
||||||
|
let mut csprng = rand::thread_rng();
|
||||||
|
let signing_key = SigningKey::generate(&mut csprng);
|
||||||
|
std::fs::write(path, signing_key.to_bytes()).expect("Failed to write signing key");
|
||||||
|
tracing::info!("Generated new signing key at {}", key_path);
|
||||||
|
signing_key
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Background scheduler loop that auto-triggers scans on nodes.
|
||||||
|
async fn scheduler_loop(state: AppState) {
|
||||||
|
let tick_secs = std::env::var("VAULTMESH_SCHEDULER_TICK_SECONDS")
|
||||||
|
.ok()
|
||||||
|
.and_then(|s| s.parse::<u64>().ok())
|
||||||
|
.unwrap_or(300);
|
||||||
|
|
||||||
|
tracing::info!("Scheduler started with tick interval: {}s", tick_secs);
|
||||||
|
|
||||||
|
loop {
|
||||||
|
if let Err(e) = run_scheduler_tick(&state).await {
|
||||||
|
tracing::warn!("scheduler tick failed: {e}");
|
||||||
|
}
|
||||||
|
tokio::time::sleep(Duration::from_secs(tick_secs)).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Run a single scheduler tick: check each node and queue scans if needed.
|
||||||
|
async fn run_scheduler_tick(state: &AppState) -> anyhow::Result<()> {
|
||||||
|
let now = OffsetDateTime::now_utc();
|
||||||
|
let latest = state.list_latest().await;
|
||||||
|
let last_scans = state.list_last_scans().await;
|
||||||
|
|
||||||
|
for hb in latest.iter() {
|
||||||
|
// Check policy allows sovereign-scan for this profile
|
||||||
|
if !state.is_command_allowed(&hb.os_profile, &hb.node_id, "sovereign-scan") {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check if scan needed (never scanned or interval elapsed)
|
||||||
|
let needs_scan = match last_scans.get(&hb.node_id) {
|
||||||
|
None => true,
|
||||||
|
Some(scan) => {
|
||||||
|
let elapsed = now - scan.ts;
|
||||||
|
elapsed >= state.scheduler_cfg.scan_interval
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
if needs_scan {
|
||||||
|
// Build and sign the command
|
||||||
|
let payload = CommandPayload {
|
||||||
|
node_id: hb.node_id,
|
||||||
|
ts: now,
|
||||||
|
nonce: Uuid::new_v4().to_string(),
|
||||||
|
cmd: "sovereign-scan".into(),
|
||||||
|
args: serde_json::json!({}),
|
||||||
|
};
|
||||||
|
|
||||||
|
let payload_json = serde_json::to_vec(&payload)?;
|
||||||
|
let signature = state.signing_key.sign(&payload_json);
|
||||||
|
let signature_b64 =
|
||||||
|
base64::engine::general_purpose::STANDARD.encode(signature.to_bytes());
|
||||||
|
|
||||||
|
let signed_cmd = SignedCommand {
|
||||||
|
payload,
|
||||||
|
signature: signature_b64,
|
||||||
|
};
|
||||||
|
|
||||||
|
state.queue_command(signed_cmd).await;
|
||||||
|
tracing::info!(
|
||||||
|
"scheduler: queued sovereign-scan for {} ({})",
|
||||||
|
hb.node_id,
|
||||||
|
hb.hostname
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::main]
|
||||||
|
async fn main() {
|
||||||
|
let cli = Cli::parse();
|
||||||
|
|
||||||
|
match cli.command {
|
||||||
|
Some(Commands::Logs { action }) => {
|
||||||
|
// CLI log commands - no tracing/server init needed
|
||||||
|
match action {
|
||||||
|
LogsAction::View {
|
||||||
|
kind,
|
||||||
|
node,
|
||||||
|
since,
|
||||||
|
min_severity,
|
||||||
|
limit,
|
||||||
|
} => {
|
||||||
|
logs::cmd_logs_view(kind, node, since, min_severity, limit);
|
||||||
|
}
|
||||||
|
LogsAction::Tail { kind, node } => {
|
||||||
|
logs::cmd_logs_tail(kind, node);
|
||||||
|
}
|
||||||
|
LogsAction::Stats => {
|
||||||
|
logs::cmd_logs_stats();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Some(Commands::Serve) | None => {
|
||||||
|
// Run the HTTP server (default)
|
||||||
|
run_server().await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn run_server() {
|
||||||
|
tracing_subscriber::registry()
|
||||||
|
.with(tracing_subscriber::EnvFilter::new(
|
||||||
|
std::env::var("RUST_LOG").unwrap_or_else(|_| "info".into()),
|
||||||
|
))
|
||||||
|
.with(tracing_subscriber::fmt::layer())
|
||||||
|
.init();
|
||||||
|
|
||||||
|
let signing_key = load_or_init_signing_key();
|
||||||
|
let state = AppState::new(signing_key);
|
||||||
|
|
||||||
|
// V0.6: Replay logs to reconstruct state
|
||||||
|
state.replay_from_logs().await;
|
||||||
|
|
||||||
|
// Log the public key for distribution to nodes
|
||||||
|
tracing::info!("CC public key (base64): {}", state.public_key_b64);
|
||||||
|
|
||||||
|
// Spawn the scheduler background task
|
||||||
|
let scheduler_state = state.clone();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
scheduler_loop(scheduler_state).await;
|
||||||
|
});
|
||||||
|
|
||||||
|
let app = app(state);
|
||||||
|
|
||||||
|
let addr: SocketAddr = "0.0.0.0:8088".parse().unwrap();
|
||||||
|
tracing::info!("VaultMesh Command Center listening on http://{}", addr);
|
||||||
|
|
||||||
|
let listener = TcpListener::bind(&addr).await.unwrap();
|
||||||
|
axum::serve(listener, app).await.unwrap();
|
||||||
|
}
|
||||||
2069
command-center/src/routes.rs
Normal file
2069
command-center/src/routes.rs
Normal file
File diff suppressed because it is too large
Load Diff
842
command-center/src/state.rs
Normal file
842
command-center/src/state.rs
Normal file
@@ -0,0 +1,842 @@
|
|||||||
|
use ed25519_dalek::SigningKey;
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use std::{
|
||||||
|
collections::{HashMap, HashSet, VecDeque},
|
||||||
|
fs::OpenOptions,
|
||||||
|
io::{BufRead, BufReader, Write},
|
||||||
|
path::PathBuf,
|
||||||
|
sync::Arc,
|
||||||
|
};
|
||||||
|
use time::{Duration, OffsetDateTime};
|
||||||
|
use tokio::sync::{broadcast, RwLock};
|
||||||
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
/// How many heartbeats we keep per node (for history).
|
||||||
|
const MAX_HEARTBEATS_PER_NODE: usize = 50;
|
||||||
|
|
||||||
|
/// How many command records we keep per node.
|
||||||
|
const MAX_COMMAND_RECORDS_PER_NODE: usize = 50;
|
||||||
|
|
||||||
|
/// Metrics reported by the node agent.
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||||
|
pub struct NodeMetrics {
|
||||||
|
pub uptime_seconds: Option<u64>,
|
||||||
|
pub load1: Option<f32>,
|
||||||
|
pub load5: Option<f32>,
|
||||||
|
pub load15: Option<f32>,
|
||||||
|
pub mem_total_mb: Option<u64>,
|
||||||
|
pub mem_used_mb: Option<u64>,
|
||||||
|
pub disk_root_total_gb: Option<f32>,
|
||||||
|
pub disk_root_used_gb: Option<f32>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Heartbeat data reported by each node.
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct NodeHeartbeat {
|
||||||
|
pub node_id: Uuid,
|
||||||
|
pub hostname: String,
|
||||||
|
pub os_profile: String,
|
||||||
|
pub cloudflare_ok: bool,
|
||||||
|
pub services_ok: bool,
|
||||||
|
pub vaultmesh_root: String,
|
||||||
|
#[serde(with = "time::serde::rfc3339")]
|
||||||
|
pub timestamp: OffsetDateTime,
|
||||||
|
#[serde(default)]
|
||||||
|
pub metrics: NodeMetrics,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A single node's heartbeat history.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct NodeHistory {
|
||||||
|
pub latest: NodeHeartbeat,
|
||||||
|
pub history: VecDeque<NodeHeartbeat>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Command payload to be signed and sent to agents.
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct CommandPayload {
|
||||||
|
pub node_id: Uuid,
|
||||||
|
#[serde(with = "time::serde::rfc3339")]
|
||||||
|
pub ts: OffsetDateTime,
|
||||||
|
pub nonce: String,
|
||||||
|
pub cmd: String,
|
||||||
|
pub args: serde_json::Value,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Signed command delivered to the agent.
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct SignedCommand {
|
||||||
|
pub payload: CommandPayload,
|
||||||
|
/// Base64-encoded Ed25519 signature of `payload` JSON bytes.
|
||||||
|
pub signature: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Result of executing a command on the node.
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct CommandResult {
|
||||||
|
pub node_id: Uuid,
|
||||||
|
#[serde(with = "time::serde::rfc3339")]
|
||||||
|
pub ts: OffsetDateTime,
|
||||||
|
pub nonce: String,
|
||||||
|
pub cmd: String,
|
||||||
|
pub status: String,
|
||||||
|
pub exit_code: Option<i32>,
|
||||||
|
pub stdout: String,
|
||||||
|
pub stderr: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Summary counts from a sovereign scan.
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct ScanSummary {
|
||||||
|
pub critical: u32,
|
||||||
|
pub high: u32,
|
||||||
|
pub medium: u32,
|
||||||
|
pub low: u32,
|
||||||
|
pub info: u32,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Last scan result for a node.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct LastScan {
|
||||||
|
pub ts: OffsetDateTime,
|
||||||
|
pub summary: ScanSummary,
|
||||||
|
pub findings_real: bool,
|
||||||
|
pub receipt_hash: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Scheduler configuration for automatic scans.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct SchedulerConfig {
|
||||||
|
pub scan_interval: Duration,
|
||||||
|
pub scan_stale: Duration,
|
||||||
|
pub heartbeat_stale: Duration,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Node attention status for dashboard.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct NodeAttentionStatus {
|
||||||
|
pub needs_attention: bool,
|
||||||
|
pub reasons: Vec<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
// ============================================================================
|
||||||
|
// V0.7: SSE Event Bus payloads
|
||||||
|
// ============================================================================
|
||||||
|
|
||||||
|
/// SSE payload for heartbeat events (wire-efficient subset).
|
||||||
|
#[derive(Debug, Clone, Serialize)]
|
||||||
|
pub struct HeartbeatEventPayload {
|
||||||
|
#[serde(with = "time::serde::rfc3339")]
|
||||||
|
pub ts: OffsetDateTime,
|
||||||
|
pub node_id: Uuid,
|
||||||
|
pub hostname: String,
|
||||||
|
pub os_profile: String,
|
||||||
|
pub cloudflare_ok: bool,
|
||||||
|
pub services_ok: bool,
|
||||||
|
pub vaultmesh_root: String,
|
||||||
|
pub uptime_secs: Option<u64>,
|
||||||
|
pub load_1: Option<f32>,
|
||||||
|
pub load_5: Option<f32>,
|
||||||
|
pub load_15: Option<f32>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// SSE payload for scan events.
|
||||||
|
#[derive(Debug, Clone, Serialize)]
|
||||||
|
pub struct ScanEventPayload {
|
||||||
|
#[serde(with = "time::serde::rfc3339")]
|
||||||
|
pub ts: OffsetDateTime,
|
||||||
|
pub node_id: Uuid,
|
||||||
|
pub hostname: String,
|
||||||
|
pub os_profile: String,
|
||||||
|
pub summary: ScanSummary,
|
||||||
|
pub findings_real: bool,
|
||||||
|
pub receipt_hash: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// SSE payload for command events.
|
||||||
|
#[derive(Debug, Clone, Serialize)]
|
||||||
|
pub struct CommandEventPayload {
|
||||||
|
#[serde(with = "time::serde::rfc3339")]
|
||||||
|
pub ts: OffsetDateTime,
|
||||||
|
pub node_id: Uuid,
|
||||||
|
pub hostname: String,
|
||||||
|
pub os_profile: String,
|
||||||
|
pub cmd: String,
|
||||||
|
pub status: String,
|
||||||
|
pub exit_code: Option<i32>,
|
||||||
|
pub nonce: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// SSE payload for attention state changes.
|
||||||
|
#[derive(Debug, Clone, Serialize)]
|
||||||
|
pub struct AttentionEventPayload {
|
||||||
|
#[serde(with = "time::serde::rfc3339")]
|
||||||
|
pub ts: OffsetDateTime,
|
||||||
|
pub node_id: Uuid,
|
||||||
|
pub hostname: String,
|
||||||
|
pub os_profile: String,
|
||||||
|
pub needs_attention: bool,
|
||||||
|
pub reasons: Vec<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Server event types for SSE broadcast.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub enum ServerEvent {
|
||||||
|
Heartbeat(HeartbeatEventPayload),
|
||||||
|
Scan(ScanEventPayload),
|
||||||
|
Command(CommandEventPayload),
|
||||||
|
Attention(AttentionEventPayload),
|
||||||
|
/// V0.7.2: Generic envelope for comms events (note, incident, ack, tag, resolve)
|
||||||
|
Envelope(EventEnvelope),
|
||||||
|
}
|
||||||
|
|
||||||
|
// ============================================================================
|
||||||
|
// V0.7.2: Communication Layer - EventEnvelope
|
||||||
|
// ============================================================================
|
||||||
|
|
||||||
|
/// Canonical message format for all comms events.
|
||||||
|
/// Used for notes, incidents, acknowledgements, tags, and resolutions.
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct EventEnvelope {
|
||||||
|
/// Unique event ID (server-assigned)
|
||||||
|
pub id: Uuid,
|
||||||
|
/// Event kind: "note", "incident", "ack", "tag", "resolve"
|
||||||
|
pub kind: String,
|
||||||
|
/// Timestamp (server-assigned)
|
||||||
|
#[serde(with = "time::serde::rfc3339")]
|
||||||
|
pub ts: OffsetDateTime,
|
||||||
|
/// Node this event relates to (optional for global events)
|
||||||
|
#[serde(skip_serializing_if = "Option::is_none")]
|
||||||
|
pub node_id: Option<Uuid>,
|
||||||
|
/// Author: "operator", "system", "vm-copilot", "scheduler", agent name
|
||||||
|
pub author: String,
|
||||||
|
/// Structured payload (kind-specific)
|
||||||
|
pub body: serde_json::Value,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Log entry wrapper for EventEnvelope (versioned for future compatibility).
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct EventEnvelopeLogEntry {
|
||||||
|
pub version: u8,
|
||||||
|
pub event: EventEnvelope,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Maximum number of envelopes to keep in memory.
|
||||||
|
const MAX_ENVELOPES_IN_MEMORY: usize = 500;
|
||||||
|
|
||||||
|
// ============================================================================
|
||||||
|
// V0.6: Append-only log events
|
||||||
|
// ============================================================================
|
||||||
|
|
||||||
|
/// Event logged when a heartbeat is received.
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct HeartbeatEvent {
|
||||||
|
#[serde(with = "time::serde::rfc3339")]
|
||||||
|
pub ts: OffsetDateTime,
|
||||||
|
pub node_id: Uuid,
|
||||||
|
pub hostname: String,
|
||||||
|
pub os_profile: String,
|
||||||
|
pub heartbeat: NodeHeartbeat,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Event logged when a sovereign-scan completes successfully.
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct ScanEvent {
|
||||||
|
#[serde(with = "time::serde::rfc3339")]
|
||||||
|
pub ts: OffsetDateTime,
|
||||||
|
pub node_id: Uuid,
|
||||||
|
pub os_profile: String,
|
||||||
|
pub summary: ScanSummary,
|
||||||
|
pub findings_real: bool,
|
||||||
|
pub receipt_hash: String,
|
||||||
|
#[serde(skip_serializing_if = "Option::is_none")]
|
||||||
|
pub receipt_path: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Event logged when any command result is reported.
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct CommandEvent {
|
||||||
|
#[serde(with = "time::serde::rfc3339")]
|
||||||
|
pub ts: OffsetDateTime,
|
||||||
|
pub node_id: Uuid,
|
||||||
|
pub cmd: String,
|
||||||
|
pub status: String,
|
||||||
|
pub exit_code: Option<i32>,
|
||||||
|
pub nonce: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Append-only log sink for durable event storage.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct LogSink {
|
||||||
|
pub dir: PathBuf,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl LogSink {
|
||||||
|
/// Create a new LogSink, initializing the directory if needed.
|
||||||
|
pub fn new_from_env() -> std::io::Result<Self> {
|
||||||
|
let dir = std::env::var("VAULTMESH_LOG_DIR")
|
||||||
|
.map(PathBuf::from)
|
||||||
|
.unwrap_or_else(|_| PathBuf::from("/var/lib/vaultmesh/cc-logs"));
|
||||||
|
|
||||||
|
std::fs::create_dir_all(&dir)?;
|
||||||
|
Ok(LogSink { dir })
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Get the full path for a log file.
|
||||||
|
pub fn file_path(&self, name: &str) -> PathBuf {
|
||||||
|
self.dir.join(name)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Append a JSON line to a log file.
|
||||||
|
pub fn append_json_line<T: Serialize>(&self, file: &str, value: &T) -> std::io::Result<()> {
|
||||||
|
let path = self.file_path(file);
|
||||||
|
let mut f = OpenOptions::new()
|
||||||
|
.create(true)
|
||||||
|
.append(true)
|
||||||
|
.open(path)?;
|
||||||
|
serde_json::to_writer(&mut f, value).map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?;
|
||||||
|
f.write_all(b"\n")?;
|
||||||
|
f.flush()?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Command policy for allowed commands per profile.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct CommandPolicy {
|
||||||
|
/// Per-profile allowed command names.
|
||||||
|
pub per_profile: HashMap<String, HashSet<String>>,
|
||||||
|
/// Global fallback set (applied if no per-profile entry).
|
||||||
|
pub global_allowed: HashSet<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl CommandPolicy {
|
||||||
|
pub fn default_policy() -> Self {
|
||||||
|
let mut global = HashSet::new();
|
||||||
|
global.insert("service-status".to_string());
|
||||||
|
global.insert("tail-journal".to_string());
|
||||||
|
global.insert("sovereign-scan".to_string());
|
||||||
|
global.insert("restart-service".to_string());
|
||||||
|
|
||||||
|
CommandPolicy {
|
||||||
|
per_profile: HashMap::new(),
|
||||||
|
global_allowed: global,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn is_allowed(&self, profile: &str, cmd: &str) -> bool {
|
||||||
|
if let Some(set) = self.per_profile.get(profile) {
|
||||||
|
set.contains(cmd)
|
||||||
|
} else {
|
||||||
|
self.global_allowed.contains(cmd)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Shared application state.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct AppState {
|
||||||
|
pub nodes: Arc<RwLock<HashMap<Uuid, NodeHistory>>>,
|
||||||
|
pub commands_pending: Arc<RwLock<HashMap<Uuid, VecDeque<SignedCommand>>>>,
|
||||||
|
pub command_results: Arc<RwLock<HashMap<Uuid, VecDeque<CommandResult>>>>,
|
||||||
|
pub last_scans: Arc<RwLock<HashMap<Uuid, LastScan>>>,
|
||||||
|
pub signing_key: Arc<SigningKey>,
|
||||||
|
pub public_key_b64: String,
|
||||||
|
pub scheduler_cfg: SchedulerConfig,
|
||||||
|
pub policy: CommandPolicy,
|
||||||
|
/// V0.6: Append-only log sink for persistence
|
||||||
|
pub logs: LogSink,
|
||||||
|
/// V0.7: SSE event broadcast channel
|
||||||
|
pub events_tx: broadcast::Sender<ServerEvent>,
|
||||||
|
/// V0.7.2: In-memory event envelope store
|
||||||
|
pub events: Arc<RwLock<Vec<EventEnvelope>>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl AppState {
|
||||||
|
pub fn new(signing_key: SigningKey) -> Self {
|
||||||
|
use base64::Engine;
|
||||||
|
let pk_bytes = signing_key.verifying_key().to_bytes();
|
||||||
|
let public_key_b64 = base64::engine::general_purpose::STANDARD.encode(pk_bytes);
|
||||||
|
|
||||||
|
// Parse scheduler config from env (with sane defaults)
|
||||||
|
let scan_interval_hours = std::env::var("VAULTMESH_SCAN_INTERVAL_HOURS")
|
||||||
|
.ok()
|
||||||
|
.and_then(|s| s.parse::<i64>().ok())
|
||||||
|
.unwrap_or(24);
|
||||||
|
|
||||||
|
let scan_stale_hours = std::env::var("VAULTMESH_SCAN_STALE_HOURS")
|
||||||
|
.ok()
|
||||||
|
.and_then(|s| s.parse::<i64>().ok())
|
||||||
|
.unwrap_or(48);
|
||||||
|
|
||||||
|
let heartbeat_stale_minutes = std::env::var("VAULTMESH_HEARTBEAT_STALE_MINUTES")
|
||||||
|
.ok()
|
||||||
|
.and_then(|s| s.parse::<i64>().ok())
|
||||||
|
.unwrap_or(10);
|
||||||
|
|
||||||
|
let scheduler_cfg = SchedulerConfig {
|
||||||
|
scan_interval: Duration::hours(scan_interval_hours),
|
||||||
|
scan_stale: Duration::hours(scan_stale_hours),
|
||||||
|
heartbeat_stale: Duration::minutes(heartbeat_stale_minutes),
|
||||||
|
};
|
||||||
|
|
||||||
|
tracing::info!(
|
||||||
|
"Scheduler config: scan_interval={}h, scan_stale={}h, heartbeat_stale={}m",
|
||||||
|
scan_interval_hours,
|
||||||
|
scan_stale_hours,
|
||||||
|
heartbeat_stale_minutes
|
||||||
|
);
|
||||||
|
|
||||||
|
// V0.6: Initialize log sink
|
||||||
|
let logs = LogSink::new_from_env()
|
||||||
|
.expect("failed to initialize log directory");
|
||||||
|
tracing::info!("Log directory: {:?}", logs.dir);
|
||||||
|
|
||||||
|
// V0.7: Initialize SSE broadcast channel
|
||||||
|
let (events_tx, _) = broadcast::channel(1024);
|
||||||
|
|
||||||
|
AppState {
|
||||||
|
nodes: Arc::new(RwLock::new(HashMap::new())),
|
||||||
|
commands_pending: Arc::new(RwLock::new(HashMap::new())),
|
||||||
|
command_results: Arc::new(RwLock::new(HashMap::new())),
|
||||||
|
last_scans: Arc::new(RwLock::new(HashMap::new())),
|
||||||
|
signing_key: Arc::new(signing_key),
|
||||||
|
public_key_b64,
|
||||||
|
scheduler_cfg,
|
||||||
|
policy: CommandPolicy::default_policy(),
|
||||||
|
logs,
|
||||||
|
events_tx,
|
||||||
|
events: Arc::new(RwLock::new(Vec::new())),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Check if a command is allowed by policy.
|
||||||
|
pub fn is_command_allowed(&self, profile: &str, node_id: &Uuid, cmd: &str) -> bool {
|
||||||
|
let allowed = self.policy.is_allowed(profile, cmd);
|
||||||
|
if !allowed {
|
||||||
|
tracing::warn!(
|
||||||
|
"policy: command {} denied for node {} (profile={})",
|
||||||
|
cmd,
|
||||||
|
node_id,
|
||||||
|
profile
|
||||||
|
);
|
||||||
|
}
|
||||||
|
allowed
|
||||||
|
}
|
||||||
|
|
||||||
|
// ========================================================================
|
||||||
|
// V0.7: SSE Event Bus
|
||||||
|
// ========================================================================
|
||||||
|
|
||||||
|
/// Publish an event to all SSE subscribers.
|
||||||
|
pub fn publish(&self, event: ServerEvent) {
|
||||||
|
if let Err(e) = self.events_tx.send(event) {
|
||||||
|
tracing::debug!("no SSE listeners: {e}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Recompute and publish attention status for a node.
|
||||||
|
pub async fn recompute_and_publish_attention(&self, node_id: Uuid) {
|
||||||
|
let now = OffsetDateTime::now_utc();
|
||||||
|
let latest = self.list_latest().await;
|
||||||
|
let last_scans = self.list_last_scans().await;
|
||||||
|
|
||||||
|
if let Some(hb) = latest.iter().find(|h| h.node_id == node_id) {
|
||||||
|
let attn = compute_attention(now, hb, last_scans.get(&node_id), &self.scheduler_cfg);
|
||||||
|
let payload = AttentionEventPayload {
|
||||||
|
ts: now,
|
||||||
|
node_id: hb.node_id,
|
||||||
|
hostname: hb.hostname.clone(),
|
||||||
|
os_profile: hb.os_profile.clone(),
|
||||||
|
needs_attention: attn.needs_attention,
|
||||||
|
reasons: attn.reasons,
|
||||||
|
};
|
||||||
|
self.publish(ServerEvent::Attention(payload));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ========================================================================
|
||||||
|
// V0.7.2: Communication Layer
|
||||||
|
// ========================================================================
|
||||||
|
|
||||||
|
/// Record an EventEnvelope: log to JSONL, store in memory, broadcast via SSE.
|
||||||
|
pub async fn record_envelope(&self, ev: EventEnvelope) {
|
||||||
|
// 1) Log to JSONL
|
||||||
|
let entry = EventEnvelopeLogEntry { version: 1, event: ev.clone() };
|
||||||
|
if let Err(e) = self.logs.append_json_line("events.jsonl", &entry) {
|
||||||
|
tracing::warn!("failed to append events log: {e}");
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2) In-memory store with cap
|
||||||
|
{
|
||||||
|
let mut events = self.events.write().await;
|
||||||
|
events.push(ev.clone());
|
||||||
|
if events.len() > MAX_ENVELOPES_IN_MEMORY {
|
||||||
|
let overflow = events.len() - MAX_ENVELOPES_IN_MEMORY;
|
||||||
|
events.drain(0..overflow);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 3) SSE broadcast
|
||||||
|
self.publish(ServerEvent::Envelope(ev));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Replay envelopes from events.jsonl on startup.
|
||||||
|
pub async fn replay_envelopes(&self) -> std::io::Result<()> {
|
||||||
|
let path = self.logs.file_path("events.jsonl");
|
||||||
|
if !path.exists() {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
let file = std::fs::File::open(&path)?;
|
||||||
|
let reader = BufReader::new(file);
|
||||||
|
let mut count = 0;
|
||||||
|
|
||||||
|
for line in reader.lines() {
|
||||||
|
let line = line?;
|
||||||
|
if line.trim().is_empty() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
let entry: EventEnvelopeLogEntry = match serde_json::from_str(&line) {
|
||||||
|
Ok(v) => v,
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!("invalid events line: {e}");
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// Store in memory (no broadcast during replay)
|
||||||
|
let mut events = self.events.write().await;
|
||||||
|
events.push(entry.event);
|
||||||
|
if events.len() > MAX_ENVELOPES_IN_MEMORY {
|
||||||
|
let overflow = events.len() - MAX_ENVELOPES_IN_MEMORY;
|
||||||
|
events.drain(0..overflow);
|
||||||
|
}
|
||||||
|
|
||||||
|
count += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
tracing::info!("replayed {} events from {:?}", count, path);
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Insert a new heartbeat for a node, updating history.
|
||||||
|
/// Logs the event to heartbeats.jsonl.
|
||||||
|
pub async fn upsert_heartbeat(&self, hb: NodeHeartbeat) {
|
||||||
|
// V0.6: Log first
|
||||||
|
let event = HeartbeatEvent {
|
||||||
|
ts: hb.timestamp,
|
||||||
|
node_id: hb.node_id,
|
||||||
|
hostname: hb.hostname.clone(),
|
||||||
|
os_profile: hb.os_profile.clone(),
|
||||||
|
heartbeat: hb.clone(),
|
||||||
|
};
|
||||||
|
if let Err(e) = self.logs.append_json_line("heartbeats.jsonl", &event) {
|
||||||
|
tracing::warn!("failed to append heartbeat log: {e}");
|
||||||
|
}
|
||||||
|
|
||||||
|
self.upsert_heartbeat_no_log(hb).await;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Insert heartbeat without logging (used during replay).
|
||||||
|
async fn upsert_heartbeat_no_log(&self, hb: NodeHeartbeat) {
|
||||||
|
let mut nodes = self.nodes.write().await;
|
||||||
|
let entry = nodes.entry(hb.node_id).or_insert_with(|| NodeHistory {
|
||||||
|
latest: hb.clone(),
|
||||||
|
history: VecDeque::new(),
|
||||||
|
});
|
||||||
|
|
||||||
|
entry.history.push_front(entry.latest.clone());
|
||||||
|
if entry.history.len() > MAX_HEARTBEATS_PER_NODE {
|
||||||
|
entry.history.pop_back();
|
||||||
|
}
|
||||||
|
|
||||||
|
entry.latest = hb;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Get a snapshot of all latest heartbeats.
|
||||||
|
pub async fn list_latest(&self) -> Vec<NodeHeartbeat> {
|
||||||
|
let nodes = self.nodes.read().await;
|
||||||
|
nodes.values().map(|h| h.latest.clone()).collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Queue a signed command for a node.
|
||||||
|
pub async fn queue_command(&self, cmd: SignedCommand) {
|
||||||
|
let mut pending = self.commands_pending.write().await;
|
||||||
|
let entry = pending
|
||||||
|
.entry(cmd.payload.node_id)
|
||||||
|
.or_insert_with(VecDeque::new);
|
||||||
|
entry.push_back(cmd);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Take (and clear) all pending commands for a node.
|
||||||
|
pub async fn take_pending_commands(&self, node_id: Uuid) -> Vec<SignedCommand> {
|
||||||
|
let mut pending = self.commands_pending.write().await;
|
||||||
|
if let Some(mut queue) = pending.remove(&node_id) {
|
||||||
|
queue.drain(..).collect()
|
||||||
|
} else {
|
||||||
|
Vec::new()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Record a command result.
|
||||||
|
/// Logs the event to commands.jsonl.
|
||||||
|
pub async fn record_command_result(&self, result: CommandResult) {
|
||||||
|
// V0.6: Log command event
|
||||||
|
let event = CommandEvent {
|
||||||
|
ts: result.ts,
|
||||||
|
node_id: result.node_id,
|
||||||
|
cmd: result.cmd.clone(),
|
||||||
|
status: result.status.clone(),
|
||||||
|
exit_code: result.exit_code,
|
||||||
|
nonce: result.nonce.clone(),
|
||||||
|
};
|
||||||
|
if let Err(e) = self.logs.append_json_line("commands.jsonl", &event) {
|
||||||
|
tracing::warn!("failed to append command log: {e}");
|
||||||
|
}
|
||||||
|
|
||||||
|
self.record_command_result_no_log(result).await;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Record command result without logging (used during replay).
|
||||||
|
async fn record_command_result_no_log(&self, result: CommandResult) {
|
||||||
|
let mut results = self.command_results.write().await;
|
||||||
|
let entry = results
|
||||||
|
.entry(result.node_id)
|
||||||
|
.or_insert_with(VecDeque::new);
|
||||||
|
entry.push_front(result);
|
||||||
|
if entry.len() > MAX_COMMAND_RECORDS_PER_NODE {
|
||||||
|
entry.pop_back();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Get recent command results for a node.
|
||||||
|
pub async fn list_command_results(&self, node_id: Uuid) -> Vec<CommandResult> {
|
||||||
|
let results = self.command_results.read().await;
|
||||||
|
results
|
||||||
|
.get(&node_id)
|
||||||
|
.map(|q| q.iter().cloned().collect())
|
||||||
|
.unwrap_or_default()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Update the last scan result for a node.
|
||||||
|
/// Note: ScanEvent logging is handled in routes.rs where we have os_profile context.
|
||||||
|
pub async fn update_last_scan(&self, node_id: Uuid, scan: LastScan) {
|
||||||
|
let mut scans = self.last_scans.write().await;
|
||||||
|
scans.insert(node_id, scan);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Update last scan without logging (used during replay).
|
||||||
|
pub async fn update_last_scan_no_log(&self, node_id: Uuid, scan: LastScan) {
|
||||||
|
let mut scans = self.last_scans.write().await;
|
||||||
|
scans.insert(node_id, scan);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Get the last scan result for a node.
|
||||||
|
pub async fn get_last_scan(&self, node_id: &Uuid) -> Option<LastScan> {
|
||||||
|
let scans = self.last_scans.read().await;
|
||||||
|
scans.get(node_id).cloned()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Get all last scan results.
|
||||||
|
pub async fn list_last_scans(&self) -> HashMap<Uuid, LastScan> {
|
||||||
|
let scans = self.last_scans.read().await;
|
||||||
|
scans.clone()
|
||||||
|
}
|
||||||
|
|
||||||
|
// ========================================================================
|
||||||
|
// V0.6: Log replay on startup
|
||||||
|
// ========================================================================
|
||||||
|
|
||||||
|
/// Replay all logs to reconstruct state on startup.
|
||||||
|
pub async fn replay_from_logs(&self) {
|
||||||
|
if let Err(e) = self.replay_heartbeats().await {
|
||||||
|
tracing::warn!("failed to replay heartbeats: {e}");
|
||||||
|
}
|
||||||
|
if let Err(e) = self.replay_scans().await {
|
||||||
|
tracing::warn!("failed to replay scans: {e}");
|
||||||
|
}
|
||||||
|
if let Err(e) = self.replay_commands().await {
|
||||||
|
tracing::warn!("failed to replay commands: {e}");
|
||||||
|
}
|
||||||
|
// V0.7.2: Replay event envelopes
|
||||||
|
if let Err(e) = self.replay_envelopes().await {
|
||||||
|
tracing::warn!("failed to replay envelopes: {e}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn replay_heartbeats(&self) -> std::io::Result<()> {
|
||||||
|
let path = self.logs.file_path("heartbeats.jsonl");
|
||||||
|
if !path.exists() {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
let file = std::fs::File::open(&path)?;
|
||||||
|
let reader = BufReader::new(file);
|
||||||
|
let mut count = 0;
|
||||||
|
|
||||||
|
for line in reader.lines() {
|
||||||
|
let line = line?;
|
||||||
|
if line.trim().is_empty() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
let event: HeartbeatEvent = match serde_json::from_str(&line) {
|
||||||
|
Ok(v) => v,
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!("invalid heartbeat line: {e}");
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// Reconstruct heartbeat from event
|
||||||
|
let hb = NodeHeartbeat {
|
||||||
|
node_id: event.node_id,
|
||||||
|
hostname: event.hostname,
|
||||||
|
os_profile: event.os_profile,
|
||||||
|
cloudflare_ok: event.heartbeat.cloudflare_ok,
|
||||||
|
services_ok: event.heartbeat.services_ok,
|
||||||
|
vaultmesh_root: event.heartbeat.vaultmesh_root,
|
||||||
|
timestamp: event.ts,
|
||||||
|
metrics: event.heartbeat.metrics,
|
||||||
|
};
|
||||||
|
|
||||||
|
self.upsert_heartbeat_no_log(hb).await;
|
||||||
|
count += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
tracing::info!("replayed {} heartbeats from {:?}", count, path);
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn replay_scans(&self) -> std::io::Result<()> {
|
||||||
|
let path = self.logs.file_path("scans.jsonl");
|
||||||
|
if !path.exists() {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
let file = std::fs::File::open(&path)?;
|
||||||
|
let reader = BufReader::new(file);
|
||||||
|
let mut count = 0;
|
||||||
|
|
||||||
|
for line in reader.lines() {
|
||||||
|
let line = line?;
|
||||||
|
if line.trim().is_empty() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
let event: ScanEvent = match serde_json::from_str(&line) {
|
||||||
|
Ok(v) => v,
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!("invalid scan line: {e}");
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
let scan = LastScan {
|
||||||
|
ts: event.ts,
|
||||||
|
summary: event.summary,
|
||||||
|
findings_real: event.findings_real,
|
||||||
|
receipt_hash: event.receipt_hash,
|
||||||
|
};
|
||||||
|
|
||||||
|
self.update_last_scan_no_log(event.node_id, scan).await;
|
||||||
|
count += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
tracing::info!("replayed {} scans from {:?}", count, path);
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn replay_commands(&self) -> std::io::Result<()> {
|
||||||
|
let path = self.logs.file_path("commands.jsonl");
|
||||||
|
if !path.exists() {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
let file = std::fs::File::open(&path)?;
|
||||||
|
let reader = BufReader::new(file);
|
||||||
|
let mut count = 0;
|
||||||
|
|
||||||
|
for line in reader.lines() {
|
||||||
|
let line = line?;
|
||||||
|
if line.trim().is_empty() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
let event: CommandEvent = match serde_json::from_str(&line) {
|
||||||
|
Ok(v) => v,
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!("invalid command line: {e}");
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// Build minimal CommandResult for history (stdout/stderr not stored in log)
|
||||||
|
let result = CommandResult {
|
||||||
|
node_id: event.node_id,
|
||||||
|
ts: event.ts,
|
||||||
|
nonce: event.nonce,
|
||||||
|
cmd: event.cmd,
|
||||||
|
status: event.status,
|
||||||
|
exit_code: event.exit_code,
|
||||||
|
stdout: String::new(),
|
||||||
|
stderr: String::new(),
|
||||||
|
};
|
||||||
|
|
||||||
|
self.record_command_result_no_log(result).await;
|
||||||
|
count += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
tracing::info!("replayed {} commands from {:?}", count, path);
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ============================================================================
|
||||||
|
// V0.7: Attention computation (moved from routes.rs for SSE access)
|
||||||
|
// ============================================================================
|
||||||
|
|
||||||
|
/// Compute attention status for a node based on current state.
|
||||||
|
pub fn compute_attention(
|
||||||
|
now: OffsetDateTime,
|
||||||
|
hb: &NodeHeartbeat,
|
||||||
|
scan: Option<&LastScan>,
|
||||||
|
cfg: &SchedulerConfig,
|
||||||
|
) -> NodeAttentionStatus {
|
||||||
|
let mut reasons = Vec::new();
|
||||||
|
|
||||||
|
// Heartbeat staleness
|
||||||
|
if now - hb.timestamp > cfg.heartbeat_stale {
|
||||||
|
reasons.push("heartbeat_stale".to_string());
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scan staleness / findings
|
||||||
|
match scan {
|
||||||
|
None => reasons.push("never_scanned".to_string()),
|
||||||
|
Some(s) => {
|
||||||
|
if now - s.ts > cfg.scan_stale {
|
||||||
|
reasons.push("scan_stale".to_string());
|
||||||
|
}
|
||||||
|
if s.summary.critical > 0 {
|
||||||
|
reasons.push("critical_findings".to_string());
|
||||||
|
} else if s.summary.high > 0 {
|
||||||
|
reasons.push("high_findings".to_string());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Service flags
|
||||||
|
if !hb.cloudflare_ok {
|
||||||
|
reasons.push("cloudflare_down".to_string());
|
||||||
|
}
|
||||||
|
if !hb.services_ok {
|
||||||
|
reasons.push("services_down".to_string());
|
||||||
|
}
|
||||||
|
|
||||||
|
NodeAttentionStatus {
|
||||||
|
needs_attention: !reasons.is_empty(),
|
||||||
|
reasons,
|
||||||
|
}
|
||||||
|
}
|
||||||
182
docs/ARCHITECTURE.md
Normal file
182
docs/ARCHITECTURE.md
Normal file
@@ -0,0 +1,182 @@
|
|||||||
|
# VaultMesh Command Center Architecture
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
The VaultMesh Command Center is a minimal control plane for monitoring and managing VaultMesh nodes. It consists of two components:
|
||||||
|
|
||||||
|
1. **Command Center (CC)** - Central Rust/Axum web server
|
||||||
|
2. **Node Agent** - Lightweight daemon running on each VaultMesh node
|
||||||
|
|
||||||
|
## Communication Model
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────┐ HTTPS (Cloudflare Tunnel) ┌─────────────────┐
|
||||||
|
│ Node Agent │ ─────────────────────────────────▶│ Command Center │
|
||||||
|
│ (ArchVault) │ POST /api/agent/heartbeat │ (Axum) │
|
||||||
|
└─────────────────┘ └─────────────────┘
|
||||||
|
│
|
||||||
|
┌─────────────────┐ HTTPS (Cloudflare Tunnel) │
|
||||||
|
│ Node Agent │ ─────────────────────────────────────────┘
|
||||||
|
│ (DebianVault) │ POST /api/agent/heartbeat
|
||||||
|
└─────────────────┘
|
||||||
|
|
||||||
|
┌─────────────────────────────────┐
|
||||||
|
│ Cloudflare Access │
|
||||||
|
│ (Zero Trust Authentication) │
|
||||||
|
└─────────────────────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────┐
|
||||||
|
│ Admin Browser │
|
||||||
|
│ GET / (Dashboard) │
|
||||||
|
└─────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Security Model
|
||||||
|
|
||||||
|
### Outbound-Only from Nodes
|
||||||
|
|
||||||
|
Nodes only make **outbound** connections to the Command Center. No inbound ports are opened on VaultMesh nodes for CC communication.
|
||||||
|
|
||||||
|
### Cloudflare Zero Trust
|
||||||
|
|
||||||
|
- **Cloudflare Tunnel**: CC is exposed via `cloudflared` tunnel, not a public IP
|
||||||
|
- **Cloudflare Access**: Admin dashboard protected by Cloudflare Access policies
|
||||||
|
- **Agent Authentication**: Future: signed JWTs or shared secrets for agent auth
|
||||||
|
|
||||||
|
### Network Flow
|
||||||
|
|
||||||
|
1. Node agent wakes up every N seconds
|
||||||
|
2. Agent collects local health metrics
|
||||||
|
3. Agent POSTs heartbeat JSON to CC via Cloudflare Tunnel
|
||||||
|
4. CC stores heartbeat in memory (future: SQLite)
|
||||||
|
5. Admin views dashboard via Cloudflare Access-protected URL
|
||||||
|
|
||||||
|
## Data Model
|
||||||
|
|
||||||
|
### NodeHeartbeat
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"node_id": "550e8400-e29b-41d4-a716-446655440000",
|
||||||
|
"hostname": "vault-node-01",
|
||||||
|
"os_profile": "ArchVault",
|
||||||
|
"cloudflare_ok": true,
|
||||||
|
"services_ok": true,
|
||||||
|
"vaultmesh_root": "/var/lib/vaultmesh",
|
||||||
|
"timestamp": "2024-01-15T10:30:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Storage (V1)
|
||||||
|
|
||||||
|
In-memory `HashMap<Uuid, NodeHeartbeat>` wrapped in `Arc<RwLock<>>`.
|
||||||
|
|
||||||
|
Nodes are keyed by `node_id`. Each heartbeat overwrites the previous entry for that node.
|
||||||
|
|
||||||
|
## Component Details
|
||||||
|
|
||||||
|
### Command Center
|
||||||
|
|
||||||
|
| Component | Technology |
|
||||||
|
|-----------|------------|
|
||||||
|
| Web Framework | Axum 0.7 |
|
||||||
|
| Async Runtime | Tokio |
|
||||||
|
| Serialization | Serde + serde_json |
|
||||||
|
| Logging | tracing + tracing-subscriber |
|
||||||
|
| HTML | Server-rendered (HTMX-ready) |
|
||||||
|
|
||||||
|
### Node Agent
|
||||||
|
|
||||||
|
| Component | Technology |
|
||||||
|
|-----------|------------|
|
||||||
|
| HTTP Client | reqwest (rustls-tls) |
|
||||||
|
| Async Runtime | Tokio |
|
||||||
|
| System Checks | systemctl calls |
|
||||||
|
| Config | Environment variables |
|
||||||
|
|
||||||
|
## Deployment
|
||||||
|
|
||||||
|
### Command Center Deployment
|
||||||
|
|
||||||
|
1. Build: `cargo build --release -p vaultmesh-command-center`
|
||||||
|
2. Install binary to `/usr/local/bin/`
|
||||||
|
3. Install systemd unit
|
||||||
|
4. Configure Cloudflare Tunnel to `http://127.0.0.1:8088`
|
||||||
|
5. Configure Cloudflare Access policies
|
||||||
|
|
||||||
|
### Node Agent Deployment
|
||||||
|
|
||||||
|
1. Build: `cargo build --release -p vaultmesh-node-agent`
|
||||||
|
2. Install binary to `/usr/local/bin/`
|
||||||
|
3. Create `/etc/vaultmesh/agent.env` with:
|
||||||
|
- `VAULTMESH_CC_URL=https://cc.your-domain.example`
|
||||||
|
- `VAULTMESH_OS_PROFILE=ArchVault`
|
||||||
|
4. Install systemd unit
|
||||||
|
|
||||||
|
## Version History
|
||||||
|
|
||||||
|
### V0.7.2: Communication Layer (Current)
|
||||||
|
- Unified `EventEnvelope` as canonical message format for comms events.
|
||||||
|
- `POST /api/events` - Ingest endpoint for operators, agents, and bots.
|
||||||
|
- `GET /api/events` - Query endpoint with since/kind/node_id/limit filtering.
|
||||||
|
- New event kinds: `note`, `incident`, `ack`, `tag`, `resolve`.
|
||||||
|
- SSE broadcast of envelope events by their `kind` name.
|
||||||
|
- Durable persistence to `events.jsonl` with replay on startup.
|
||||||
|
- Memory-bounded in-memory store (500 most recent envelopes).
|
||||||
|
|
||||||
|
### V0.7.1: Mission Console
|
||||||
|
- NASA-style 3-panel dashboard at `GET /console`.
|
||||||
|
- Global Mission Bar with fleet KPIs (Total/Healthy/Attention/Critical).
|
||||||
|
- Left panel: Node list with status pills (OK/ATTN/CRIT), live heartbeat glow.
|
||||||
|
- Center panel: Selected node telemetry + per-node event timeline.
|
||||||
|
- Right panel: Attention summary, scan findings, global event feed.
|
||||||
|
- Full SSE wiring for real-time DOM updates without page refresh.
|
||||||
|
|
||||||
|
### V0.7: SSE Event Bus
|
||||||
|
- Real-time event streaming via `GET /events` (Server-Sent Events).
|
||||||
|
- Named events: `heartbeat`, `scan`, `command`, `attention`.
|
||||||
|
- Broadcast channel distributes events to all connected SSE clients.
|
||||||
|
- Keepalive every 15s to prevent connection timeouts.
|
||||||
|
- JS probe in dashboard for console.log debugging.
|
||||||
|
|
||||||
|
### V0.6.1: Log Tools
|
||||||
|
- CLI subcommands for querying JSONL event logs.
|
||||||
|
- `logs view` with filters: --kind, --node, --since, --min-severity, --limit.
|
||||||
|
- `logs tail` for real-time log following.
|
||||||
|
- `logs stats` for per-node event statistics.
|
||||||
|
|
||||||
|
### V0.6: Append-Only Persistence
|
||||||
|
- Event logging to JSONL files (heartbeats, scans, commands).
|
||||||
|
- State replay on startup from `$VAULTMESH_LOG_DIR`.
|
||||||
|
- Foundation for V0.8 Ledger Bridge (Merkle over logs).
|
||||||
|
|
||||||
|
### V0.5: Fleet Orchestrator
|
||||||
|
- Background scheduler for autonomous scans.
|
||||||
|
- Attention model with staleness / drift detection.
|
||||||
|
- Command policy enforcement (per-profile allowlists).
|
||||||
|
|
||||||
|
### V0.4: Commands & Sovereign Scan
|
||||||
|
- Push commands to nodes (Ed25519 signed).
|
||||||
|
- Command receipts with proof chain.
|
||||||
|
- Sovereign scan execution and reporting.
|
||||||
|
|
||||||
|
### V0.3: Signed Commands
|
||||||
|
- Ed25519 key generation and command signing.
|
||||||
|
- Agent signature verification.
|
||||||
|
- Nonce-based replay protection.
|
||||||
|
|
||||||
|
### V0.2: Node Metrics
|
||||||
|
- System metrics in heartbeat (load, memory, disk).
|
||||||
|
- Heartbeat history tracking.
|
||||||
|
|
||||||
|
## Future Extensions
|
||||||
|
|
||||||
|
### V0.7.1: Mission Console
|
||||||
|
- NASA-inspired dashboard layout with node tree sidebar.
|
||||||
|
- Live row updates via SSE (no page refresh).
|
||||||
|
- Attention summary panel with fleet health overview.
|
||||||
|
|
||||||
|
### V0.8: Ledger Bridge
|
||||||
|
- Merkle tree over `sovereign-scans.jsonl`.
|
||||||
|
- `ROOT.txt` + `PROOFCHAIN.json` for VaultMesh proof layer.
|
||||||
83
docs/EVENT_GENERATION.md
Normal file
83
docs/EVENT_GENERATION.md
Normal file
@@ -0,0 +1,83 @@
|
|||||||
|
# VaultMesh Command Center: Event Generation Mechanism
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
The VaultMesh Command Center generates events through a sophisticated, multi-layered mechanism designed for real-time monitoring and fleet management.
|
||||||
|
|
||||||
|
## Event Types
|
||||||
|
|
||||||
|
### 1. Heartbeat Events
|
||||||
|
- **Trigger**: Node heartbeat submission
|
||||||
|
- **Payload Includes**:
|
||||||
|
* Timestamp
|
||||||
|
* Node ID
|
||||||
|
* Hostname
|
||||||
|
* OS Profile
|
||||||
|
* Cloudflare Status
|
||||||
|
* Services Status
|
||||||
|
* VaultMesh Root Path
|
||||||
|
* System Metrics (uptime, load averages)
|
||||||
|
|
||||||
|
### 2. Scan Events
|
||||||
|
- **Trigger**: Successful scan result submission
|
||||||
|
- **Payload Includes**:
|
||||||
|
* Timestamp
|
||||||
|
* Node ID
|
||||||
|
* Hostname
|
||||||
|
* OS Profile
|
||||||
|
* Scan Summary (critical/high/medium/low findings)
|
||||||
|
* Real/Mock Findings Flag
|
||||||
|
* Receipt Hash
|
||||||
|
|
||||||
|
### 3. Command Events
|
||||||
|
- **Trigger**: Command execution result
|
||||||
|
- **Payload Includes**:
|
||||||
|
* Timestamp
|
||||||
|
* Node ID
|
||||||
|
* Hostname
|
||||||
|
* OS Profile
|
||||||
|
* Command Name
|
||||||
|
* Execution Status
|
||||||
|
* Exit Code
|
||||||
|
* Nonce (for replay protection)
|
||||||
|
|
||||||
|
## Event Generation Flow
|
||||||
|
|
||||||
|
1. **Data Collection**
|
||||||
|
- Node agents submit heartbeats and scan results
|
||||||
|
- Command results are reported back to the Command Center
|
||||||
|
|
||||||
|
2. **Event Processing**
|
||||||
|
- Raw data is transformed into structured event payloads
|
||||||
|
- Events are published to a broadcast channel
|
||||||
|
- Server-Sent Events (SSE) distribute events to connected clients
|
||||||
|
|
||||||
|
3. **State Management**
|
||||||
|
- Events trigger state updates (node history, last scan, etc.)
|
||||||
|
- Attention status is recomputed based on new events
|
||||||
|
|
||||||
|
## Advanced Features
|
||||||
|
|
||||||
|
- **Automatic Scan Scheduling**
|
||||||
|
- Periodic scans triggered based on node profile and last scan timestamp
|
||||||
|
- Configurable scan intervals
|
||||||
|
|
||||||
|
- **Attention Computation**
|
||||||
|
- Dynamic assessment of node health
|
||||||
|
- Tracks critical findings, heartbeat staleness, service status
|
||||||
|
|
||||||
|
## Security Considerations
|
||||||
|
|
||||||
|
- Ed25519 key signing for commands
|
||||||
|
- Nonce-based replay protection
|
||||||
|
- Configurable command policies per node profile
|
||||||
|
|
||||||
|
## Performance Characteristics
|
||||||
|
|
||||||
|
- In-memory event storage (500 most recent events)
|
||||||
|
- Optional JSONL log persistence
|
||||||
|
- Low-overhead event broadcasting
|
||||||
|
|
||||||
|
## Extensibility
|
||||||
|
|
||||||
|
The event system supports easy addition of new event types and payloads through the `ServerEvent` enum and corresponding payload structures.
|
||||||
170
docs/EVENT_PROCESSING.md
Normal file
170
docs/EVENT_PROCESSING.md
Normal file
@@ -0,0 +1,170 @@
|
|||||||
|
# VaultMesh Command Center: Event Processing Architecture
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
The Command Center implements a sophisticated, multi-layered event processing system designed for robust, real-time fleet management.
|
||||||
|
|
||||||
|
## Key Components
|
||||||
|
|
||||||
|
### 1. Event Types
|
||||||
|
- `NodeHeartbeat`: Node status updates
|
||||||
|
- `ScanEvent`: Scan findings and results
|
||||||
|
- `CommandEvent`: Command execution outcomes
|
||||||
|
- `EventEnvelope`: Generic communication events
|
||||||
|
|
||||||
|
### 2. Processing Stages
|
||||||
|
|
||||||
|
#### a. Ingestion
|
||||||
|
- Raw event data received via HTTP endpoints
|
||||||
|
- Validated and transformed into structured payloads
|
||||||
|
- Logged to append-only JSONL files for durability
|
||||||
|
|
||||||
|
#### b. State Update
|
||||||
|
- In-memory state updated with latest information
|
||||||
|
- Maintain sliding window of recent events (max 50 per node)
|
||||||
|
- Compute derived states (attention status, last scan)
|
||||||
|
|
||||||
|
#### c. Broadcasting
|
||||||
|
- Events published via Server-Sent Events (SSE)
|
||||||
|
- Broadcast to all connected clients
|
||||||
|
- Low-latency, real-time updates
|
||||||
|
|
||||||
|
## Event Processing Workflow
|
||||||
|
|
||||||
|
### Heartbeat Processing
|
||||||
|
1. Receive heartbeat data
|
||||||
|
2. Log to `heartbeats.jsonl`
|
||||||
|
3. Update node history
|
||||||
|
4. Publish heartbeat event
|
||||||
|
5. Recompute node attention status
|
||||||
|
6. Broadcast attention event
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub async fn upsert_heartbeat(&self, hb: NodeHeartbeat) {
|
||||||
|
// Log event
|
||||||
|
let event = HeartbeatEvent { ... };
|
||||||
|
self.logs.append_json_line("heartbeats.jsonl", &event);
|
||||||
|
|
||||||
|
// Update in-memory state
|
||||||
|
self.upsert_heartbeat_no_log(hb).await;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Scan Result Processing
|
||||||
|
1. Receive scan results
|
||||||
|
2. Log to `scans.jsonl`
|
||||||
|
3. Update last scan information
|
||||||
|
4. Publish scan event
|
||||||
|
5. Recompute node attention status
|
||||||
|
6. Broadcast attention event
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub async fn update_last_scan(&self, node_id: Uuid, scan: LastScan) {
|
||||||
|
// Update scan history
|
||||||
|
let mut scans = self.last_scans.write().await;
|
||||||
|
scans.insert(node_id, scan);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Command Result Processing
|
||||||
|
1. Receive command result
|
||||||
|
2. Log to `commands.jsonl`
|
||||||
|
3. Store command history
|
||||||
|
4. Publish command event
|
||||||
|
5. Optionally trigger additional actions
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub async fn record_command_result(&self, result: CommandResult) {
|
||||||
|
// Log command event
|
||||||
|
let event = CommandEvent { ... };
|
||||||
|
self.logs.append_json_line("commands.jsonl", &event);
|
||||||
|
|
||||||
|
// Update command result history
|
||||||
|
self.record_command_result_no_log(result).await;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Attention Computation
|
||||||
|
|
||||||
|
The system dynamically computes a node's attention status based on multiple factors:
|
||||||
|
|
||||||
|
- Heartbeat staleness
|
||||||
|
- Scan staleness
|
||||||
|
- Scan findings severity
|
||||||
|
- Service status
|
||||||
|
- Cloudflare tunnel status
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub fn compute_attention(
|
||||||
|
now: OffsetDateTime,
|
||||||
|
hb: &NodeHeartbeat,
|
||||||
|
scan: Option<&LastScan>,
|
||||||
|
cfg: &SchedulerConfig,
|
||||||
|
) -> NodeAttentionStatus {
|
||||||
|
let mut reasons = Vec::new();
|
||||||
|
|
||||||
|
// Check heartbeat age
|
||||||
|
if now - hb.timestamp > cfg.heartbeat_stale {
|
||||||
|
reasons.push("heartbeat_stale");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check scan status
|
||||||
|
match scan {
|
||||||
|
None => reasons.push("never_scanned"),
|
||||||
|
Some(s) => {
|
||||||
|
if now - s.ts > cfg.scan_stale {
|
||||||
|
reasons.push("scan_stale");
|
||||||
|
}
|
||||||
|
if s.summary.critical > 0 {
|
||||||
|
reasons.push("critical_findings");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check service flags
|
||||||
|
if !hb.cloudflare_ok {
|
||||||
|
reasons.push("cloudflare_down");
|
||||||
|
}
|
||||||
|
|
||||||
|
NodeAttentionStatus {
|
||||||
|
needs_attention: !reasons.is_empty(),
|
||||||
|
reasons,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Persistence and State Replay
|
||||||
|
|
||||||
|
### Log Replay Mechanism
|
||||||
|
- On startup, reconstruct in-memory state from JSONL logs
|
||||||
|
- Replay events in chronological order
|
||||||
|
- Recreate node history, scan results, and command results
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub async fn replay_from_logs(&self) {
|
||||||
|
self.replay_heartbeats().await;
|
||||||
|
self.replay_scans().await;
|
||||||
|
self.replay_commands().await;
|
||||||
|
self.replay_envelopes().await;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Performance Characteristics
|
||||||
|
|
||||||
|
- In-memory event store: 500 most recent events
|
||||||
|
- Append-only logging for durability
|
||||||
|
- Non-blocking event processing
|
||||||
|
- Low-overhead broadcasting
|
||||||
|
|
||||||
|
## Security Considerations
|
||||||
|
|
||||||
|
- Ed25519 key signing for commands
|
||||||
|
- Configurable command policies
|
||||||
|
- Nonce-based replay protection
|
||||||
|
- No sensitive data stored in logs
|
||||||
|
|
||||||
|
## Extensibility
|
||||||
|
|
||||||
|
- Easy to add new event types
|
||||||
|
- Flexible attention computation
|
||||||
|
- Modular event processing pipeline
|
||||||
163
docs/NODE_AGENT_CONTRACT.md
Normal file
163
docs/NODE_AGENT_CONTRACT.md
Normal file
@@ -0,0 +1,163 @@
|
|||||||
|
# Node Agent Contract
|
||||||
|
|
||||||
|
This document defines the API contract between VaultMesh Node Agents and the Command Center.
|
||||||
|
|
||||||
|
## Heartbeat Endpoint
|
||||||
|
|
||||||
|
### Request
|
||||||
|
|
||||||
|
```
|
||||||
|
POST /api/agent/heartbeat
|
||||||
|
Content-Type: application/json
|
||||||
|
```
|
||||||
|
|
||||||
|
### Heartbeat Schema
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"node_id": "UUID v4",
|
||||||
|
"hostname": "string",
|
||||||
|
"os_profile": "string",
|
||||||
|
"cloudflare_ok": "boolean",
|
||||||
|
"services_ok": "boolean",
|
||||||
|
"vaultmesh_root": "string",
|
||||||
|
"timestamp": "ISO 8601 / RFC 3339"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Field Definitions
|
||||||
|
|
||||||
|
| Field | Type | Description |
|
||||||
|
|-------|------|-------------|
|
||||||
|
| `node_id` | UUID v4 | Unique identifier for this node. Should persist across reboots. |
|
||||||
|
| `hostname` | String | System hostname (from `hostname::get()` or `/etc/hostname`) |
|
||||||
|
| `os_profile` | String | VaultMesh profile name: `ArchVault`, `DebianVault`, etc. |
|
||||||
|
| `cloudflare_ok` | Boolean | `true` if `cloudflared` service is active |
|
||||||
|
| `services_ok` | Boolean | `true` if VAULTMESH_ROOT exists and is healthy |
|
||||||
|
| `vaultmesh_root` | String | Path to VAULTMESH_ROOT (e.g., `/var/lib/vaultmesh`) |
|
||||||
|
| `timestamp` | RFC 3339 | UTC timestamp when heartbeat was generated |
|
||||||
|
|
||||||
|
### Response
|
||||||
|
|
||||||
|
**Success (200 OK)**:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"status": "ok"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Error (4xx/5xx)**:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"error": "description"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Environment Variables
|
||||||
|
|
||||||
|
The node agent is configured via environment variables, typically set in `/etc/vaultmesh/agent.env`.
|
||||||
|
|
||||||
|
| Variable | Required | Default | Description |
|
||||||
|
|----------|----------|---------|-------------|
|
||||||
|
| `VAULTMESH_NODE_ID` | No | Auto-generated UUID v4 | Persistent node identifier |
|
||||||
|
| `VAULTMESH_CC_URL` | No | `http://127.0.0.1:8088` | Command Center base URL |
|
||||||
|
| `VAULTMESH_OS_PROFILE` | No | `ArchVault` | OS profile name to report |
|
||||||
|
| `VAULTMESH_ROOT` | No | `/var/lib/vaultmesh` | Path to check for services_ok |
|
||||||
|
| `VAULTMESH_HEARTBEAT_SECS` | No | `30` | Seconds between heartbeats |
|
||||||
|
| `RUST_LOG` | No | `info` | Log level (trace, debug, info, warn, error) |
|
||||||
|
|
||||||
|
### Example `/etc/vaultmesh/agent.env`
|
||||||
|
|
||||||
|
```bash
|
||||||
|
VAULTMESH_NODE_ID=550e8400-e29b-41d4-a716-446655440000
|
||||||
|
VAULTMESH_CC_URL=https://cc.vaultmesh.example
|
||||||
|
VAULTMESH_OS_PROFILE=ArchVault
|
||||||
|
VAULTMESH_ROOT=/var/lib/vaultmesh
|
||||||
|
VAULTMESH_HEARTBEAT_SECS=30
|
||||||
|
RUST_LOG=info
|
||||||
|
```
|
||||||
|
|
||||||
|
## Node Registration
|
||||||
|
|
||||||
|
Nodes self-register on first heartbeat. There is no explicit registration endpoint.
|
||||||
|
|
||||||
|
When the Command Center receives a heartbeat with a new `node_id`, it creates a new entry. Subsequent heartbeats update the existing entry.
|
||||||
|
|
||||||
|
### Node ID Persistence
|
||||||
|
|
||||||
|
For consistent tracking, the `VAULTMESH_NODE_ID` should be persisted. Options:
|
||||||
|
|
||||||
|
1. **Environment file**: Set in `/etc/vaultmesh/agent.env`
|
||||||
|
2. **Machine ID**: Could derive from `/etc/machine-id`
|
||||||
|
3. **Auto-generated**: If not set, agent generates a new UUID on each start (not recommended for production)
|
||||||
|
|
||||||
|
**Recommended**: Generate a UUID once during node bootstrap and store in `agent.env`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# During node bootstrap
|
||||||
|
echo "VAULTMESH_NODE_ID=$(uuidgen)" >> /etc/vaultmesh/agent.env
|
||||||
|
```
|
||||||
|
|
||||||
|
## Health Checks
|
||||||
|
|
||||||
|
### cloudflare_ok
|
||||||
|
|
||||||
|
The agent runs:
|
||||||
|
```bash
|
||||||
|
systemctl is-active --quiet cloudflared
|
||||||
|
```
|
||||||
|
|
||||||
|
Returns `true` if exit code is 0 (service active).
|
||||||
|
|
||||||
|
### services_ok
|
||||||
|
|
||||||
|
The agent checks if `VAULTMESH_ROOT` exists and is a directory:
|
||||||
|
```rust
|
||||||
|
std::path::Path::new(vaultmesh_root).is_dir()
|
||||||
|
```
|
||||||
|
|
||||||
|
Future versions may add additional checks:
|
||||||
|
- Disk space
|
||||||
|
- Key services running
|
||||||
|
- ProofChain integrity
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
### Network Errors
|
||||||
|
|
||||||
|
If the agent cannot reach the Command Center:
|
||||||
|
- Log error at WARN/ERROR level
|
||||||
|
- Sleep for heartbeat interval
|
||||||
|
- Retry on next cycle
|
||||||
|
|
||||||
|
No exponential backoff in V1. The agent will retry every `VAULTMESH_HEARTBEAT_SECS` seconds indefinitely.
|
||||||
|
|
||||||
|
### Invalid Response
|
||||||
|
|
||||||
|
If CC returns non-2xx status:
|
||||||
|
- Log warning with status code
|
||||||
|
- Continue normal operation
|
||||||
|
- Retry on next cycle
|
||||||
|
|
||||||
|
## Security Considerations
|
||||||
|
|
||||||
|
### Transport Security
|
||||||
|
|
||||||
|
- Agent should connect to CC via HTTPS (Cloudflare Tunnel)
|
||||||
|
- `reqwest` configured with `rustls-tls` (no OpenSSL dependency)
|
||||||
|
|
||||||
|
### Authentication (Future)
|
||||||
|
|
||||||
|
V1 has no agent authentication. Future versions may add:
|
||||||
|
- Signed JWTs
|
||||||
|
- Shared secrets
|
||||||
|
- mTLS
|
||||||
|
|
||||||
|
### Data Sensitivity
|
||||||
|
|
||||||
|
Heartbeat data is low-sensitivity:
|
||||||
|
- No secrets or credentials
|
||||||
|
- No PII
|
||||||
|
- No file contents
|
||||||
|
|
||||||
|
The `vaultmesh_root` path and hostname are the most identifying fields.
|
||||||
19
node-agent/Cargo.toml
Normal file
19
node-agent/Cargo.toml
Normal file
@@ -0,0 +1,19 @@
|
|||||||
|
[package]
|
||||||
|
name = "vaultmesh-node-agent"
|
||||||
|
version = "0.1.0"
|
||||||
|
edition = "2021"
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
reqwest = { version = "0.12", features = ["json", "rustls-tls"], default-features = false }
|
||||||
|
tokio = { version = "1", features = ["rt-multi-thread", "macros", "time", "process"] }
|
||||||
|
serde = { version = "1", features = ["derive"] }
|
||||||
|
serde_json = "1"
|
||||||
|
uuid = { version = "1", features = ["v4", "serde"] }
|
||||||
|
time = { version = "0.3", features = ["serde", "macros", "formatting", "parsing"] }
|
||||||
|
tracing = "0.1"
|
||||||
|
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
||||||
|
hostname = "0.4"
|
||||||
|
anyhow = "1"
|
||||||
|
ed25519-dalek = "2"
|
||||||
|
base64 = "0.22"
|
||||||
|
blake3 = "1"
|
||||||
52
node-agent/src/config.rs
Normal file
52
node-agent/src/config.rs
Normal file
@@ -0,0 +1,52 @@
|
|||||||
|
use std::env;
|
||||||
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
/// Agent configuration loaded from environment variables.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct Config {
|
||||||
|
pub node_id: Uuid,
|
||||||
|
pub cc_url: String,
|
||||||
|
pub os_profile: String,
|
||||||
|
pub vaultmesh_root: String,
|
||||||
|
pub heartbeat_secs: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Config {
|
||||||
|
/// Load configuration from environment variables.
|
||||||
|
///
|
||||||
|
/// | Variable | Default |
|
||||||
|
/// |---------------------------|----------------------------|
|
||||||
|
/// | VAULTMESH_NODE_ID | auto-generated UUID |
|
||||||
|
/// | VAULTMESH_CC_URL | http://127.0.0.1:8088 |
|
||||||
|
/// | VAULTMESH_OS_PROFILE | ArchVault |
|
||||||
|
/// | VAULTMESH_ROOT | /var/lib/vaultmesh |
|
||||||
|
/// | VAULTMESH_HEARTBEAT_SECS | 30 |
|
||||||
|
pub fn from_env() -> Self {
|
||||||
|
let node_id = env::var("VAULTMESH_NODE_ID")
|
||||||
|
.ok()
|
||||||
|
.and_then(|s| s.parse().ok())
|
||||||
|
.unwrap_or_else(Uuid::new_v4);
|
||||||
|
|
||||||
|
let cc_url = env::var("VAULTMESH_CC_URL")
|
||||||
|
.unwrap_or_else(|_| "http://127.0.0.1:8088".into());
|
||||||
|
|
||||||
|
let os_profile = env::var("VAULTMESH_OS_PROFILE")
|
||||||
|
.unwrap_or_else(|_| "ArchVault".into());
|
||||||
|
|
||||||
|
let vaultmesh_root = env::var("VAULTMESH_ROOT")
|
||||||
|
.unwrap_or_else(|_| "/var/lib/vaultmesh".into());
|
||||||
|
|
||||||
|
let heartbeat_secs = env::var("VAULTMESH_HEARTBEAT_SECS")
|
||||||
|
.ok()
|
||||||
|
.and_then(|s| s.parse().ok())
|
||||||
|
.unwrap_or(30);
|
||||||
|
|
||||||
|
Self {
|
||||||
|
node_id,
|
||||||
|
cc_url,
|
||||||
|
os_profile,
|
||||||
|
vaultmesh_root,
|
||||||
|
heartbeat_secs,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
818
node-agent/src/main.rs
Normal file
818
node-agent/src/main.rs
Normal file
@@ -0,0 +1,818 @@
|
|||||||
|
mod config;
|
||||||
|
|
||||||
|
use crate::config::Config;
|
||||||
|
use anyhow::{bail, Result};
|
||||||
|
use ed25519_dalek::{Signature, Verifier, VerifyingKey};
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use std::{fs, path::Path, process::Command as StdCommand};
|
||||||
|
use time::OffsetDateTime;
|
||||||
|
use tokio::time::{sleep, Duration};
|
||||||
|
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};
|
||||||
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
/// Metrics collected from the node.
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||||
|
pub struct NodeMetrics {
|
||||||
|
pub uptime_seconds: Option<u64>,
|
||||||
|
pub load1: Option<f32>,
|
||||||
|
pub load5: Option<f32>,
|
||||||
|
pub load15: Option<f32>,
|
||||||
|
pub mem_total_mb: Option<u64>,
|
||||||
|
pub mem_used_mb: Option<u64>,
|
||||||
|
pub disk_root_total_gb: Option<f32>,
|
||||||
|
pub disk_root_used_gb: Option<f32>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Heartbeat data sent to the Command Center.
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct NodeHeartbeat {
|
||||||
|
pub node_id: Uuid,
|
||||||
|
pub hostname: String,
|
||||||
|
pub os_profile: String,
|
||||||
|
pub cloudflare_ok: bool,
|
||||||
|
pub services_ok: bool,
|
||||||
|
pub vaultmesh_root: String,
|
||||||
|
#[serde(with = "time::serde::rfc3339")]
|
||||||
|
pub timestamp: OffsetDateTime,
|
||||||
|
pub metrics: NodeMetrics,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Command payload from CC.
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct CommandPayload {
|
||||||
|
pub node_id: Uuid,
|
||||||
|
#[serde(with = "time::serde::rfc3339")]
|
||||||
|
pub ts: OffsetDateTime,
|
||||||
|
pub nonce: String,
|
||||||
|
pub cmd: String,
|
||||||
|
pub args: serde_json::Value,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Signed command from CC.
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct SignedCommand {
|
||||||
|
pub payload: CommandPayload,
|
||||||
|
pub signature: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Command execution result sent back to CC.
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct CommandResult {
|
||||||
|
pub node_id: Uuid,
|
||||||
|
#[serde(with = "time::serde::rfc3339")]
|
||||||
|
pub ts: OffsetDateTime,
|
||||||
|
pub nonce: String,
|
||||||
|
pub cmd: String,
|
||||||
|
pub status: String,
|
||||||
|
pub exit_code: Option<i32>,
|
||||||
|
pub stdout: String,
|
||||||
|
pub stderr: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Summary of scan findings by severity.
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||||
|
pub struct ScanSummary {
|
||||||
|
pub critical: u32,
|
||||||
|
pub high: u32,
|
||||||
|
pub medium: u32,
|
||||||
|
pub low: u32,
|
||||||
|
pub info: u32,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ScanSummary {
|
||||||
|
fn merge(&mut self, other: &ScanSummary) {
|
||||||
|
self.critical += other.critical;
|
||||||
|
self.high += other.high;
|
||||||
|
self.medium += other.medium;
|
||||||
|
self.low += other.low;
|
||||||
|
self.info += other.info;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Result from running a single scanner.
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||||
|
pub struct ScannerResult {
|
||||||
|
pub available: bool,
|
||||||
|
pub ran: bool,
|
||||||
|
pub version: Option<String>,
|
||||||
|
pub findings_real: bool,
|
||||||
|
pub findings: u32,
|
||||||
|
pub output_file: Option<String>,
|
||||||
|
pub error: Option<String>,
|
||||||
|
pub summary: ScanSummary,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Check if cloudflared service is active via systemctl.
|
||||||
|
fn check_cloudflared() -> bool {
|
||||||
|
StdCommand::new("systemctl")
|
||||||
|
.args(["is-active", "--quiet", "cloudflared"])
|
||||||
|
.status()
|
||||||
|
.map(|s| s.success())
|
||||||
|
.unwrap_or(false)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Check if essential VaultMesh services are healthy.
|
||||||
|
fn check_services(vaultmesh_root: &str) -> bool {
|
||||||
|
Path::new(vaultmesh_root).is_dir()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Gather system metrics from /proc and df.
|
||||||
|
fn gather_metrics() -> NodeMetrics {
|
||||||
|
NodeMetrics {
|
||||||
|
uptime_seconds: read_uptime_seconds(),
|
||||||
|
load1: read_loadavg_index(0),
|
||||||
|
load5: read_loadavg_index(1),
|
||||||
|
load15: read_loadavg_index(2),
|
||||||
|
mem_total_mb: read_meminfo_total_mb(),
|
||||||
|
mem_used_mb: read_meminfo_used_mb(),
|
||||||
|
disk_root_total_gb: read_disk_root_total_gb(),
|
||||||
|
disk_root_used_gb: read_disk_root_used_gb(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_uptime_seconds() -> Option<u64> {
|
||||||
|
let contents = fs::read_to_string("/proc/uptime").ok()?;
|
||||||
|
let first = contents.split_whitespace().next()?;
|
||||||
|
let secs_str = first.split('.').next().unwrap_or(first);
|
||||||
|
secs_str.parse().ok()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_loadavg_index(idx: usize) -> Option<f32> {
|
||||||
|
let contents = fs::read_to_string("/proc/loadavg").ok()?;
|
||||||
|
let mut parts = contents.split_whitespace();
|
||||||
|
parts.nth(idx)?.parse().ok()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_meminfo_total_mb() -> Option<u64> {
|
||||||
|
read_meminfo_field_kb("MemTotal").map(|kb| kb / 1024)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_meminfo_used_mb() -> Option<u64> {
|
||||||
|
let total_kb = read_meminfo_field_kb("MemTotal")?;
|
||||||
|
let avail_kb = read_meminfo_field_kb("MemAvailable")?;
|
||||||
|
let used_kb = total_kb.saturating_sub(avail_kb);
|
||||||
|
Some(used_kb / 1024)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_meminfo_field_kb(field: &str) -> Option<u64> {
|
||||||
|
let contents = fs::read_to_string("/proc/meminfo").ok()?;
|
||||||
|
for line in contents.lines() {
|
||||||
|
if line.starts_with(field) {
|
||||||
|
let val = line.split_whitespace().nth(1)?;
|
||||||
|
return val.parse().ok();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
None
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_disk_root_total_gb() -> Option<f32> {
|
||||||
|
read_df_root_column(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_disk_root_used_gb() -> Option<f32> {
|
||||||
|
read_df_root_column(2)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_df_root_column(col_index: usize) -> Option<f32> {
|
||||||
|
let output = StdCommand::new("df")
|
||||||
|
.arg("-B1")
|
||||||
|
.arg("/")
|
||||||
|
.output()
|
||||||
|
.ok()?;
|
||||||
|
if !output.status.success() {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
let stdout = String::from_utf8_lossy(&output.stdout);
|
||||||
|
let mut lines = stdout.lines();
|
||||||
|
lines.next()?;
|
||||||
|
let line = lines.next()?;
|
||||||
|
let cols: Vec<&str> = line.split_whitespace().collect();
|
||||||
|
let val: f64 = cols.get(col_index)?.parse().ok()?;
|
||||||
|
Some((val / (1024.0 * 1024.0 * 1024.0)) as f32)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Load CC public key from file.
|
||||||
|
fn load_cc_public_key(path: &str) -> Result<VerifyingKey> {
|
||||||
|
use base64::Engine;
|
||||||
|
let p = Path::new(path);
|
||||||
|
if !p.exists() {
|
||||||
|
bail!("CC public key file not found: {}", path);
|
||||||
|
}
|
||||||
|
let contents = fs::read_to_string(p)?.trim().to_string();
|
||||||
|
let bytes = base64::engine::general_purpose::STANDARD.decode(&contents)?;
|
||||||
|
let key_bytes: [u8; 32] = bytes
|
||||||
|
.try_into()
|
||||||
|
.map_err(|_| anyhow::anyhow!("Public key must be 32 bytes"))?;
|
||||||
|
Ok(VerifyingKey::from_bytes(&key_bytes)?)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Verify a signed command.
|
||||||
|
fn verify_command(cmd: &SignedCommand, pubkey: &VerifyingKey) -> Result<()> {
|
||||||
|
use base64::Engine;
|
||||||
|
let payload_bytes = serde_json::to_vec(&cmd.payload)?;
|
||||||
|
let sig_bytes = base64::engine::general_purpose::STANDARD.decode(&cmd.signature)?;
|
||||||
|
let sig_arr: [u8; 64] = sig_bytes
|
||||||
|
.try_into()
|
||||||
|
.map_err(|_| anyhow::anyhow!("Signature must be 64 bytes"))?;
|
||||||
|
let signature = Signature::from_bytes(&sig_arr);
|
||||||
|
pubkey.verify(&payload_bytes, &signature)?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Execute a whitelisted command.
|
||||||
|
fn execute_command(cmd: &SignedCommand, config: &Config) -> CommandResult {
|
||||||
|
let payload = &cmd.payload;
|
||||||
|
let allowed = ["restart-service", "service-status", "tail-journal", "sovereign-scan"];
|
||||||
|
|
||||||
|
if !allowed.contains(&payload.cmd.as_str()) {
|
||||||
|
return CommandResult {
|
||||||
|
node_id: payload.node_id,
|
||||||
|
ts: OffsetDateTime::now_utc(),
|
||||||
|
nonce: payload.nonce.clone(),
|
||||||
|
cmd: payload.cmd.clone(),
|
||||||
|
status: "rejected".into(),
|
||||||
|
exit_code: None,
|
||||||
|
stdout: String::new(),
|
||||||
|
stderr: format!("Command '{}' not in whitelist", payload.cmd),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
match payload.cmd.as_str() {
|
||||||
|
"service-status" => {
|
||||||
|
let service = payload.args["service"].as_str().unwrap_or("cloudflared");
|
||||||
|
let output = StdCommand::new("systemctl")
|
||||||
|
.args(["status", service])
|
||||||
|
.output();
|
||||||
|
make_result(payload, output)
|
||||||
|
}
|
||||||
|
"restart-service" => {
|
||||||
|
let service = payload.args["service"].as_str().unwrap_or("cloudflared");
|
||||||
|
let output = StdCommand::new("systemctl")
|
||||||
|
.args(["restart", service])
|
||||||
|
.output();
|
||||||
|
make_result(payload, output)
|
||||||
|
}
|
||||||
|
"tail-journal" => {
|
||||||
|
let unit = payload.args["unit"].as_str().unwrap_or("cloudflared");
|
||||||
|
let lines = payload.args["lines"].as_u64().unwrap_or(50);
|
||||||
|
let output = StdCommand::new("journalctl")
|
||||||
|
.args(["-u", unit, "-n", &lines.to_string(), "--no-pager"])
|
||||||
|
.output();
|
||||||
|
make_result(payload, output)
|
||||||
|
}
|
||||||
|
"sovereign-scan" => {
|
||||||
|
run_sovereign_scan(payload, config)
|
||||||
|
}
|
||||||
|
_ => CommandResult {
|
||||||
|
node_id: payload.node_id,
|
||||||
|
ts: OffsetDateTime::now_utc(),
|
||||||
|
nonce: payload.nonce.clone(),
|
||||||
|
cmd: payload.cmd.clone(),
|
||||||
|
status: "error".into(),
|
||||||
|
exit_code: None,
|
||||||
|
stdout: String::new(),
|
||||||
|
stderr: "Unknown command".into(),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Convert command output to CommandResult.
|
||||||
|
fn make_result(payload: &CommandPayload, output: std::io::Result<std::process::Output>) -> CommandResult {
|
||||||
|
match output {
|
||||||
|
Ok(out) => CommandResult {
|
||||||
|
node_id: payload.node_id,
|
||||||
|
ts: OffsetDateTime::now_utc(),
|
||||||
|
nonce: payload.nonce.clone(),
|
||||||
|
cmd: payload.cmd.clone(),
|
||||||
|
status: if out.status.success() { "success".into() } else { "failed".into() },
|
||||||
|
exit_code: out.status.code(),
|
||||||
|
stdout: String::from_utf8_lossy(&out.stdout).into_owned(),
|
||||||
|
stderr: String::from_utf8_lossy(&out.stderr).into_owned(),
|
||||||
|
},
|
||||||
|
Err(e) => CommandResult {
|
||||||
|
node_id: payload.node_id,
|
||||||
|
ts: OffsetDateTime::now_utc(),
|
||||||
|
nonce: payload.nonce.clone(),
|
||||||
|
cmd: payload.cmd.clone(),
|
||||||
|
status: "error".into(),
|
||||||
|
exit_code: None,
|
||||||
|
stdout: String::new(),
|
||||||
|
stderr: format!("Failed to execute: {}", e),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Check if a scanner binary is available on the system.
|
||||||
|
fn scanner_available(name: &str) -> bool {
|
||||||
|
StdCommand::new("which")
|
||||||
|
.arg(name)
|
||||||
|
.output()
|
||||||
|
.map(|o| o.status.success())
|
||||||
|
.unwrap_or(false)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Get scanner version string.
|
||||||
|
fn get_scanner_version(name: &str) -> Option<String> {
|
||||||
|
let output = StdCommand::new(name)
|
||||||
|
.arg("--version")
|
||||||
|
.output()
|
||||||
|
.ok()?;
|
||||||
|
if output.status.success() {
|
||||||
|
let stdout = String::from_utf8_lossy(&output.stdout);
|
||||||
|
Some(stdout.lines().next().unwrap_or("unknown").to_string())
|
||||||
|
} else {
|
||||||
|
None
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Run nuclei scanner or generate mock results.
|
||||||
|
fn run_nuclei(nonce: &str, profile: &str, receipts_dir: &str) -> ScannerResult {
|
||||||
|
if !scanner_available("nuclei") {
|
||||||
|
tracing::info!("nuclei not available, generating mock results");
|
||||||
|
return generate_mock_nuclei_result(nonce, receipts_dir);
|
||||||
|
}
|
||||||
|
|
||||||
|
let output_file = format!("{}/nuclei-{}.json", receipts_dir, nonce);
|
||||||
|
let templates = match profile {
|
||||||
|
"fast" => "cves/",
|
||||||
|
"deep" => ".",
|
||||||
|
_ => "cves/,vulnerabilities/",
|
||||||
|
};
|
||||||
|
|
||||||
|
let output = StdCommand::new("nuclei")
|
||||||
|
.args(["-t", templates, "-silent", "-json", "-o", &output_file])
|
||||||
|
.args(["-target", "localhost"])
|
||||||
|
.output();
|
||||||
|
|
||||||
|
match output {
|
||||||
|
Ok(out) if out.status.success() || out.status.code() == Some(0) => {
|
||||||
|
let summary = parse_nuclei_findings(&output_file);
|
||||||
|
ScannerResult {
|
||||||
|
available: true,
|
||||||
|
ran: true,
|
||||||
|
version: get_scanner_version("nuclei"),
|
||||||
|
findings_real: true,
|
||||||
|
findings: summary.critical + summary.high + summary.medium + summary.low + summary.info,
|
||||||
|
output_file: Some(format!("nuclei-{}.json", nonce)),
|
||||||
|
error: None,
|
||||||
|
summary,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(out) => ScannerResult {
|
||||||
|
available: true,
|
||||||
|
ran: true,
|
||||||
|
version: get_scanner_version("nuclei"),
|
||||||
|
findings_real: false,
|
||||||
|
findings: 0,
|
||||||
|
output_file: None,
|
||||||
|
error: Some(format!("nuclei exited with code {:?}: {}",
|
||||||
|
out.status.code(),
|
||||||
|
String::from_utf8_lossy(&out.stderr))),
|
||||||
|
summary: ScanSummary::default(),
|
||||||
|
},
|
||||||
|
Err(e) => ScannerResult {
|
||||||
|
available: true,
|
||||||
|
ran: false,
|
||||||
|
version: None,
|
||||||
|
findings_real: false,
|
||||||
|
findings: 0,
|
||||||
|
output_file: None,
|
||||||
|
error: Some(format!("Failed to run nuclei: {}", e)),
|
||||||
|
summary: ScanSummary::default(),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Generate mock nuclei results when scanner not available.
|
||||||
|
fn generate_mock_nuclei_result(nonce: &str, receipts_dir: &str) -> ScannerResult {
|
||||||
|
let output_file = format!("{}/nuclei-{}.json", receipts_dir, nonce);
|
||||||
|
let mock_findings = serde_json::json!([
|
||||||
|
{"info": {"severity": "high", "name": "MOCK: Example CVE Detection"}, "host": "localhost", "matched-at": "localhost:80"},
|
||||||
|
{"info": {"severity": "medium", "name": "MOCK: Outdated Software Version"}, "host": "localhost", "matched-at": "localhost:443"},
|
||||||
|
{"info": {"severity": "low", "name": "MOCK: Information Disclosure"}, "host": "localhost", "matched-at": "localhost:22"}
|
||||||
|
]);
|
||||||
|
|
||||||
|
fs::write(&output_file, serde_json::to_string_pretty(&mock_findings).unwrap()).ok();
|
||||||
|
|
||||||
|
ScannerResult {
|
||||||
|
available: false,
|
||||||
|
ran: false,
|
||||||
|
version: None,
|
||||||
|
findings_real: false,
|
||||||
|
findings: 3,
|
||||||
|
output_file: Some(format!("nuclei-{}.json", nonce)),
|
||||||
|
error: Some("Scanner not installed - mock data generated".into()),
|
||||||
|
summary: ScanSummary {
|
||||||
|
critical: 0,
|
||||||
|
high: 1,
|
||||||
|
medium: 1,
|
||||||
|
low: 1,
|
||||||
|
info: 0,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Parse nuclei JSON output for findings summary.
|
||||||
|
fn parse_nuclei_findings(output_file: &str) -> ScanSummary {
|
||||||
|
let mut summary = ScanSummary::default();
|
||||||
|
|
||||||
|
if let Ok(contents) = fs::read_to_string(output_file) {
|
||||||
|
for line in contents.lines() {
|
||||||
|
if let Ok(finding) = serde_json::from_str::<serde_json::Value>(line) {
|
||||||
|
if let Some(severity) = finding["info"]["severity"].as_str() {
|
||||||
|
match severity.to_lowercase().as_str() {
|
||||||
|
"critical" => summary.critical += 1,
|
||||||
|
"high" => summary.high += 1,
|
||||||
|
"medium" => summary.medium += 1,
|
||||||
|
"low" => summary.low += 1,
|
||||||
|
"info" => summary.info += 1,
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
summary
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Run trivy scanner or generate mock results.
|
||||||
|
fn run_trivy(nonce: &str, profile: &str, receipts_dir: &str) -> ScannerResult {
|
||||||
|
if !scanner_available("trivy") {
|
||||||
|
tracing::info!("trivy not available, generating mock results");
|
||||||
|
return generate_mock_trivy_result(nonce, receipts_dir);
|
||||||
|
}
|
||||||
|
|
||||||
|
let output_file = format!("{}/trivy-{}.json", receipts_dir, nonce);
|
||||||
|
let severity = match profile {
|
||||||
|
"fast" => "CRITICAL,HIGH",
|
||||||
|
"deep" => "CRITICAL,HIGH,MEDIUM,LOW",
|
||||||
|
_ => "CRITICAL,HIGH,MEDIUM",
|
||||||
|
};
|
||||||
|
|
||||||
|
let output = StdCommand::new("trivy")
|
||||||
|
.args(["fs", "/", "--format", "json", "--output", &output_file])
|
||||||
|
.args(["--severity", severity])
|
||||||
|
.args(["--scanners", "vuln"])
|
||||||
|
.output();
|
||||||
|
|
||||||
|
match output {
|
||||||
|
Ok(out) if out.status.success() || out.status.code() == Some(0) => {
|
||||||
|
let summary = parse_trivy_findings(&output_file);
|
||||||
|
ScannerResult {
|
||||||
|
available: true,
|
||||||
|
ran: true,
|
||||||
|
version: get_scanner_version("trivy"),
|
||||||
|
findings_real: true,
|
||||||
|
findings: summary.critical + summary.high + summary.medium + summary.low + summary.info,
|
||||||
|
output_file: Some(format!("trivy-{}.json", nonce)),
|
||||||
|
error: None,
|
||||||
|
summary,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(out) => ScannerResult {
|
||||||
|
available: true,
|
||||||
|
ran: true,
|
||||||
|
version: get_scanner_version("trivy"),
|
||||||
|
findings_real: false,
|
||||||
|
findings: 0,
|
||||||
|
output_file: None,
|
||||||
|
error: Some(format!("trivy exited with code {:?}: {}",
|
||||||
|
out.status.code(),
|
||||||
|
String::from_utf8_lossy(&out.stderr))),
|
||||||
|
summary: ScanSummary::default(),
|
||||||
|
},
|
||||||
|
Err(e) => ScannerResult {
|
||||||
|
available: true,
|
||||||
|
ran: false,
|
||||||
|
version: None,
|
||||||
|
findings_real: false,
|
||||||
|
findings: 0,
|
||||||
|
output_file: None,
|
||||||
|
error: Some(format!("Failed to run trivy: {}", e)),
|
||||||
|
summary: ScanSummary::default(),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Generate mock trivy results when scanner not available.
|
||||||
|
fn generate_mock_trivy_result(nonce: &str, receipts_dir: &str) -> ScannerResult {
|
||||||
|
let output_file = format!("{}/trivy-{}.json", receipts_dir, nonce);
|
||||||
|
let mock_findings = serde_json::json!({
|
||||||
|
"Results": [{
|
||||||
|
"Target": "/",
|
||||||
|
"Vulnerabilities": [
|
||||||
|
{"VulnerabilityID": "MOCK-CVE-2024-0001", "Severity": "CRITICAL", "Title": "MOCK: Critical System Vulnerability"},
|
||||||
|
{"VulnerabilityID": "MOCK-CVE-2024-0002", "Severity": "HIGH", "Title": "MOCK: High Risk Package"},
|
||||||
|
{"VulnerabilityID": "MOCK-CVE-2024-0003", "Severity": "HIGH", "Title": "MOCK: Outdated Dependency"},
|
||||||
|
{"VulnerabilityID": "MOCK-CVE-2024-0004", "Severity": "MEDIUM", "Title": "MOCK: Medium Risk Finding"},
|
||||||
|
{"VulnerabilityID": "MOCK-CVE-2024-0005", "Severity": "MEDIUM", "Title": "MOCK: Configuration Issue"}
|
||||||
|
]
|
||||||
|
}]
|
||||||
|
});
|
||||||
|
|
||||||
|
fs::write(&output_file, serde_json::to_string_pretty(&mock_findings).unwrap()).ok();
|
||||||
|
|
||||||
|
ScannerResult {
|
||||||
|
available: false,
|
||||||
|
ran: false,
|
||||||
|
version: None,
|
||||||
|
findings_real: false,
|
||||||
|
findings: 5,
|
||||||
|
output_file: Some(format!("trivy-{}.json", nonce)),
|
||||||
|
error: Some("Scanner not installed - mock data generated".into()),
|
||||||
|
summary: ScanSummary {
|
||||||
|
critical: 1,
|
||||||
|
high: 2,
|
||||||
|
medium: 2,
|
||||||
|
low: 0,
|
||||||
|
info: 0,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Parse trivy JSON output for findings summary.
|
||||||
|
fn parse_trivy_findings(output_file: &str) -> ScanSummary {
|
||||||
|
let mut summary = ScanSummary::default();
|
||||||
|
|
||||||
|
if let Ok(contents) = fs::read_to_string(output_file) {
|
||||||
|
if let Ok(report) = serde_json::from_str::<serde_json::Value>(&contents) {
|
||||||
|
if let Some(results) = report["Results"].as_array() {
|
||||||
|
for result in results {
|
||||||
|
if let Some(vulns) = result["Vulnerabilities"].as_array() {
|
||||||
|
for vuln in vulns {
|
||||||
|
if let Some(severity) = vuln["Severity"].as_str() {
|
||||||
|
match severity.to_uppercase().as_str() {
|
||||||
|
"CRITICAL" => summary.critical += 1,
|
||||||
|
"HIGH" => summary.high += 1,
|
||||||
|
"MEDIUM" => summary.medium += 1,
|
||||||
|
"LOW" => summary.low += 1,
|
||||||
|
_ => summary.info += 1,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
summary
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Execute the sovereign-scan command.
|
||||||
|
fn run_sovereign_scan(payload: &CommandPayload, config: &Config) -> CommandResult {
|
||||||
|
use time::format_description::well_known::Rfc3339;
|
||||||
|
|
||||||
|
let profile = payload.args["profile"].as_str().unwrap_or("default");
|
||||||
|
let scanners: Vec<&str> = payload.args["scanners"]
|
||||||
|
.as_array()
|
||||||
|
.map(|arr| arr.iter().filter_map(|v| v.as_str()).collect())
|
||||||
|
.unwrap_or_else(|| vec!["nuclei", "trivy"]);
|
||||||
|
|
||||||
|
tracing::info!("Running sovereign-scan: profile={}, scanners={:?}", profile, scanners);
|
||||||
|
|
||||||
|
// Create receipts directory
|
||||||
|
let receipts_dir = format!("{}/receipts/sovereign", config.vaultmesh_root);
|
||||||
|
if let Err(e) = fs::create_dir_all(&receipts_dir) {
|
||||||
|
return CommandResult {
|
||||||
|
node_id: payload.node_id,
|
||||||
|
ts: OffsetDateTime::now_utc(),
|
||||||
|
nonce: payload.nonce.clone(),
|
||||||
|
cmd: "sovereign-scan".into(),
|
||||||
|
status: "error".into(),
|
||||||
|
exit_code: None,
|
||||||
|
stdout: String::new(),
|
||||||
|
stderr: format!("Failed to create receipts directory: {}", e),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut scanner_results = serde_json::Map::new();
|
||||||
|
let mut summary = ScanSummary::default();
|
||||||
|
|
||||||
|
// Run each requested scanner
|
||||||
|
for scanner in &scanners {
|
||||||
|
match *scanner {
|
||||||
|
"nuclei" => {
|
||||||
|
let result = run_nuclei(&payload.nonce, profile, &receipts_dir);
|
||||||
|
summary.merge(&result.summary);
|
||||||
|
scanner_results.insert("nuclei".into(), serde_json::to_value(&result).unwrap());
|
||||||
|
}
|
||||||
|
"trivy" => {
|
||||||
|
let result = run_trivy(&payload.nonce, profile, &receipts_dir);
|
||||||
|
summary.merge(&result.summary);
|
||||||
|
scanner_results.insert("trivy".into(), serde_json::to_value(&result).unwrap());
|
||||||
|
}
|
||||||
|
_ => {
|
||||||
|
tracing::warn!("Unknown scanner requested: {}", scanner);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Build receipt (without hash first)
|
||||||
|
let mut receipt = serde_json::json!({
|
||||||
|
"kind": "sovereign-scan",
|
||||||
|
"node_id": payload.node_id,
|
||||||
|
"nonce": payload.nonce,
|
||||||
|
"timestamp": OffsetDateTime::now_utc().format(&Rfc3339).unwrap(),
|
||||||
|
"os_profile": config.os_profile,
|
||||||
|
"vaultmesh_root": config.vaultmesh_root,
|
||||||
|
"profile": profile,
|
||||||
|
"scanners": scanner_results,
|
||||||
|
"summary": summary
|
||||||
|
});
|
||||||
|
|
||||||
|
// Hash the receipt content (before adding hash field)
|
||||||
|
let receipt_bytes = serde_json::to_vec(&receipt).unwrap();
|
||||||
|
let hash = blake3::hash(&receipt_bytes);
|
||||||
|
let hash_hex = format!("blake3:{}", hash.to_hex());
|
||||||
|
receipt["receipt_hash"] = serde_json::Value::String(hash_hex.clone());
|
||||||
|
|
||||||
|
// Write receipt file
|
||||||
|
let receipt_path = format!("{}/SCAN-{}.json", receipts_dir, payload.nonce);
|
||||||
|
if let Err(e) = fs::write(&receipt_path, serde_json::to_string_pretty(&receipt).unwrap()) {
|
||||||
|
tracing::error!("Failed to write receipt: {}", e);
|
||||||
|
}
|
||||||
|
|
||||||
|
tracing::info!("Sovereign scan complete: receipt_hash={}", hash_hex);
|
||||||
|
|
||||||
|
// Append to proofchain log
|
||||||
|
let proofchain_dir = format!("{}/proofchain", config.vaultmesh_root);
|
||||||
|
fs::create_dir_all(&proofchain_dir).ok();
|
||||||
|
let proofchain_entry = serde_json::json!({
|
||||||
|
"ts": OffsetDateTime::now_utc().format(&Rfc3339).unwrap(),
|
||||||
|
"node_id": payload.node_id,
|
||||||
|
"nonce": payload.nonce,
|
||||||
|
"receipt_hash": hash_hex
|
||||||
|
});
|
||||||
|
let proofchain_path = format!("{}/sovereign-scans.jsonl", proofchain_dir);
|
||||||
|
if let Ok(mut file) = fs::OpenOptions::new()
|
||||||
|
.create(true)
|
||||||
|
.append(true)
|
||||||
|
.open(&proofchain_path)
|
||||||
|
{
|
||||||
|
use std::io::Write;
|
||||||
|
writeln!(file, "{}", serde_json::to_string(&proofchain_entry).unwrap()).ok();
|
||||||
|
}
|
||||||
|
|
||||||
|
// Determine if any scanner had real findings
|
||||||
|
let any_real = scanner_results.values().any(|v| {
|
||||||
|
v.get("findings_real").and_then(|f| f.as_bool()).unwrap_or(false)
|
||||||
|
});
|
||||||
|
|
||||||
|
// Return summary as stdout
|
||||||
|
let stdout = serde_json::to_string_pretty(&serde_json::json!({
|
||||||
|
"status": "completed",
|
||||||
|
"receipt_path": receipt_path,
|
||||||
|
"receipt_hash": hash_hex,
|
||||||
|
"summary": summary,
|
||||||
|
"findings_real": any_real
|
||||||
|
}))
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
CommandResult {
|
||||||
|
node_id: payload.node_id,
|
||||||
|
ts: OffsetDateTime::now_utc(),
|
||||||
|
nonce: payload.nonce.clone(),
|
||||||
|
cmd: "sovereign-scan".into(),
|
||||||
|
status: "success".into(),
|
||||||
|
exit_code: Some(0),
|
||||||
|
stdout,
|
||||||
|
stderr: String::new(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::main]
|
||||||
|
async fn main() {
|
||||||
|
tracing_subscriber::registry()
|
||||||
|
.with(tracing_subscriber::EnvFilter::new(
|
||||||
|
std::env::var("RUST_LOG").unwrap_or_else(|_| "info".into()),
|
||||||
|
))
|
||||||
|
.with(tracing_subscriber::fmt::layer())
|
||||||
|
.init();
|
||||||
|
|
||||||
|
let config = Config::from_env();
|
||||||
|
let hostname = hostname::get()
|
||||||
|
.map(|h| h.to_string_lossy().to_string())
|
||||||
|
.unwrap_or_else(|_| "unknown".into());
|
||||||
|
|
||||||
|
tracing::info!(
|
||||||
|
"VaultMesh Node Agent starting (node_id={}, hostname={})",
|
||||||
|
config.node_id,
|
||||||
|
hostname
|
||||||
|
);
|
||||||
|
tracing::info!("Command Center URL: {}", config.cc_url);
|
||||||
|
tracing::info!("Heartbeat interval: {}s", config.heartbeat_secs);
|
||||||
|
|
||||||
|
// Load CC public key for signature verification
|
||||||
|
let cc_pubkey_path = std::env::var("VAULTMESH_CC_PUBKEY")
|
||||||
|
.unwrap_or_else(|_| "/etc/vaultmesh/cc-ed25519.pub".into());
|
||||||
|
let cc_pubkey = match load_cc_public_key(&cc_pubkey_path) {
|
||||||
|
Ok(pk) => {
|
||||||
|
tracing::info!("Loaded CC public key from {}", cc_pubkey_path);
|
||||||
|
Some(pk)
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!("Could not load CC public key: {} - commands disabled", e);
|
||||||
|
None
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
let client = reqwest::Client::new();
|
||||||
|
let heartbeat_url = format!("{}/api/agent/heartbeat", config.cc_url);
|
||||||
|
let commands_url = format!("{}/api/agent/commands?node_id={}", config.cc_url, config.node_id);
|
||||||
|
let result_url = format!("{}/api/agent/command-result", config.cc_url);
|
||||||
|
|
||||||
|
loop {
|
||||||
|
// Send heartbeat
|
||||||
|
let metrics = gather_metrics();
|
||||||
|
let heartbeat = NodeHeartbeat {
|
||||||
|
node_id: config.node_id,
|
||||||
|
hostname: hostname.clone(),
|
||||||
|
os_profile: config.os_profile.clone(),
|
||||||
|
cloudflare_ok: check_cloudflared(),
|
||||||
|
services_ok: check_services(&config.vaultmesh_root),
|
||||||
|
vaultmesh_root: config.vaultmesh_root.clone(),
|
||||||
|
timestamp: OffsetDateTime::now_utc(),
|
||||||
|
metrics,
|
||||||
|
};
|
||||||
|
|
||||||
|
tracing::debug!("Sending heartbeat: {:?}", heartbeat);
|
||||||
|
|
||||||
|
match client.post(&heartbeat_url).json(&heartbeat).send().await {
|
||||||
|
Ok(resp) if resp.status().is_success() => {
|
||||||
|
tracing::info!(
|
||||||
|
"Heartbeat sent (cf={}, svc={}, load={:.2}, mem={}MiB, disk={:.1}GiB)",
|
||||||
|
heartbeat.cloudflare_ok,
|
||||||
|
heartbeat.services_ok,
|
||||||
|
heartbeat.metrics.load1.unwrap_or(0.0),
|
||||||
|
heartbeat.metrics.mem_used_mb.unwrap_or(0),
|
||||||
|
heartbeat.metrics.disk_root_used_gb.unwrap_or(0.0),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
Ok(resp) => {
|
||||||
|
tracing::warn!("Heartbeat returned non-success status: {}", resp.status());
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
tracing::error!("Failed to send heartbeat: {}", e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Poll for commands (only if we have the public key)
|
||||||
|
if let Some(ref pubkey) = cc_pubkey {
|
||||||
|
match client.get(&commands_url).send().await {
|
||||||
|
Ok(resp) if resp.status().is_success() => {
|
||||||
|
match resp.json::<Vec<SignedCommand>>().await {
|
||||||
|
Ok(commands) => {
|
||||||
|
for cmd in commands {
|
||||||
|
tracing::info!("Received command: {} (nonce={})", cmd.payload.cmd, cmd.payload.nonce);
|
||||||
|
|
||||||
|
// Verify signature
|
||||||
|
if let Err(e) = verify_command(&cmd, pubkey) {
|
||||||
|
tracing::error!("Signature verification failed: {}", e);
|
||||||
|
let result = CommandResult {
|
||||||
|
node_id: cmd.payload.node_id,
|
||||||
|
ts: OffsetDateTime::now_utc(),
|
||||||
|
nonce: cmd.payload.nonce.clone(),
|
||||||
|
cmd: cmd.payload.cmd.clone(),
|
||||||
|
status: "rejected".into(),
|
||||||
|
exit_code: None,
|
||||||
|
stdout: String::new(),
|
||||||
|
stderr: format!("Signature verification failed: {}", e),
|
||||||
|
};
|
||||||
|
let _ = client.post(&result_url).json(&result).send().await;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
tracing::info!("Signature verified, executing command");
|
||||||
|
let result = execute_command(&cmd, &config);
|
||||||
|
tracing::info!(
|
||||||
|
"Command '{}' completed: {} (exit={:?})",
|
||||||
|
result.cmd,
|
||||||
|
result.status,
|
||||||
|
result.exit_code
|
||||||
|
);
|
||||||
|
|
||||||
|
// Report result
|
||||||
|
if let Err(e) = client.post(&result_url).json(&result).send().await {
|
||||||
|
tracing::error!("Failed to report command result: {}", e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!("Failed to parse commands response: {}", e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(resp) => {
|
||||||
|
tracing::warn!("Commands endpoint returned: {}", resp.status());
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
tracing::debug!("Failed to poll commands: {}", e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
sleep(Duration::from_secs(config.heartbeat_secs)).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
113
scripts/event_generation_demo.py
Executable file
113
scripts/event_generation_demo.py
Executable file
@@ -0,0 +1,113 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
VaultMesh Command Center: Event Generation Demonstration
|
||||||
|
|
||||||
|
This script simulates the event generation mechanism of the VaultMesh Command Center.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import uuid
|
||||||
|
import json
|
||||||
|
import time
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
import requests
|
||||||
|
|
||||||
|
class EventGenerator:
|
||||||
|
def __init__(self, cc_url="http://localhost:8088"):
|
||||||
|
self.cc_url = cc_url
|
||||||
|
self.node_id = str(uuid.uuid4())
|
||||||
|
self.hostname = "demo-node"
|
||||||
|
self.os_profile = "DemoVault"
|
||||||
|
|
||||||
|
def generate_heartbeat(self):
|
||||||
|
"""Generate a heartbeat event payload."""
|
||||||
|
return {
|
||||||
|
"node_id": self.node_id,
|
||||||
|
"hostname": self.hostname,
|
||||||
|
"os_profile": self.os_profile,
|
||||||
|
"cloudflare_ok": True,
|
||||||
|
"services_ok": True,
|
||||||
|
"vaultmesh_root": "/var/lib/vaultmesh",
|
||||||
|
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||||
|
"metrics": {
|
||||||
|
"uptime_seconds": int(time.time()),
|
||||||
|
"load1": 0.5,
|
||||||
|
"load5": 0.3,
|
||||||
|
"load15": 0.2
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
def generate_scan_result(self):
|
||||||
|
"""Generate a mock scan result event payload."""
|
||||||
|
return {
|
||||||
|
"cmd": "sovereign-scan",
|
||||||
|
"node_id": self.node_id,
|
||||||
|
"ts": datetime.now(timezone.utc).isoformat(),
|
||||||
|
"status": "success",
|
||||||
|
"stdout": json.dumps({
|
||||||
|
"summary": {
|
||||||
|
"critical": 1,
|
||||||
|
"high": 3,
|
||||||
|
"medium": 5,
|
||||||
|
"low": 10,
|
||||||
|
"info": 15
|
||||||
|
},
|
||||||
|
"findings_real": False,
|
||||||
|
"receipt_hash": str(uuid.uuid4())
|
||||||
|
}),
|
||||||
|
"exit_code": 0
|
||||||
|
}
|
||||||
|
|
||||||
|
def generate_command_result(self):
|
||||||
|
"""Generate a mock command result event payload."""
|
||||||
|
return {
|
||||||
|
"node_id": self.node_id,
|
||||||
|
"cmd": "service-status",
|
||||||
|
"ts": datetime.now(timezone.utc).isoformat(),
|
||||||
|
"status": "success",
|
||||||
|
"stdout": "cloudflared: active\nnginx: active",
|
||||||
|
"exit_code": 0,
|
||||||
|
"nonce": str(uuid.uuid4())
|
||||||
|
}
|
||||||
|
|
||||||
|
def send_event(self, event_type, event_data):
|
||||||
|
"""Send an event to the Command Center."""
|
||||||
|
try:
|
||||||
|
response = requests.post(
|
||||||
|
f"{self.cc_url}/api/events",
|
||||||
|
json={
|
||||||
|
"kind": event_type,
|
||||||
|
"node_id": self.node_id,
|
||||||
|
"body": event_data
|
||||||
|
},
|
||||||
|
headers={"Content-Type": "application/json"}
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
print(f"[SUCCESS] Sent {event_type} event for node {self.node_id}")
|
||||||
|
return response.json()
|
||||||
|
except requests.RequestException as e:
|
||||||
|
print(f"[ERROR] Failed to send {event_type} event: {e}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
def main():
|
||||||
|
generator = EventGenerator()
|
||||||
|
|
||||||
|
print("🚀 VaultMesh Command Center Event Generation Demo")
|
||||||
|
print("---------------------------------------------")
|
||||||
|
|
||||||
|
# Simulate event generation
|
||||||
|
events = [
|
||||||
|
("heartbeat", generator.generate_heartbeat()),
|
||||||
|
("scan", generator.generate_scan_result()),
|
||||||
|
("command", generator.generate_command_result())
|
||||||
|
]
|
||||||
|
|
||||||
|
for event_type, event_data in events:
|
||||||
|
print(f"\nGenerating {event_type.upper()} event:")
|
||||||
|
print(json.dumps(event_data, indent=2))
|
||||||
|
result = generator.send_event(event_type, event_data)
|
||||||
|
if result:
|
||||||
|
print(f"Event ID: {result.get('id')}")
|
||||||
|
time.sleep(1) # Add a small delay between events
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
23
systemd/vaultmesh-command-center.service
Normal file
23
systemd/vaultmesh-command-center.service
Normal file
@@ -0,0 +1,23 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=VaultMesh Command Center
|
||||||
|
After=network-online.target cloudflared.service
|
||||||
|
Wants=network-online.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=exec
|
||||||
|
ExecStart=/usr/local/bin/vaultmesh-command-center
|
||||||
|
Restart=on-failure
|
||||||
|
RestartSec=5
|
||||||
|
Environment=RUST_LOG=info
|
||||||
|
|
||||||
|
# Hardening
|
||||||
|
NoNewPrivileges=yes
|
||||||
|
ProtectSystem=strict
|
||||||
|
ProtectHome=yes
|
||||||
|
PrivateTmp=yes
|
||||||
|
ProtectKernelTunables=yes
|
||||||
|
ProtectKernelModules=yes
|
||||||
|
ProtectControlGroups=yes
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
23
systemd/vaultmesh-node-agent.service
Normal file
23
systemd/vaultmesh-node-agent.service
Normal file
@@ -0,0 +1,23 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=VaultMesh Node Agent
|
||||||
|
After=network-online.target cloudflared.service
|
||||||
|
Wants=network-online.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=exec
|
||||||
|
ExecStart=/usr/local/bin/vaultmesh-node-agent
|
||||||
|
Restart=on-failure
|
||||||
|
RestartSec=10
|
||||||
|
EnvironmentFile=/etc/vaultmesh/agent.env
|
||||||
|
|
||||||
|
# Hardening
|
||||||
|
NoNewPrivileges=yes
|
||||||
|
ProtectSystem=strict
|
||||||
|
ProtectHome=yes
|
||||||
|
PrivateTmp=yes
|
||||||
|
ProtectKernelTunables=yes
|
||||||
|
ProtectKernelModules=yes
|
||||||
|
ProtectControlGroups=yes
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
1
vm-copilot/requirements.txt
Normal file
1
vm-copilot/requirements.txt
Normal file
@@ -0,0 +1 @@
|
|||||||
|
requests>=2.28.0
|
||||||
305
vm-copilot/vm_copilot_agent.py
Normal file
305
vm-copilot/vm_copilot_agent.py
Normal file
@@ -0,0 +1,305 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
VaultMesh vm-copilot Agent
|
||||||
|
|
||||||
|
A bot that monitors Command Center nodes and posts incidents/notes
|
||||||
|
when attention conditions are detected.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python vm_copilot_agent.py
|
||||||
|
|
||||||
|
Environment variables:
|
||||||
|
VAULTMESH_CC_URL - Command Center URL (default: http://127.0.0.1:8088)
|
||||||
|
VM_COPILOT_INTERVAL - Poll interval in seconds (default: 60)
|
||||||
|
VM_COPILOT_NODES - Comma-separated list of node IDs to watch
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import logging
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
import requests
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
# Configuration
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CC_URL = os.environ.get("VAULTMESH_CC_URL", "http://127.0.0.1:8088")
|
||||||
|
POLL_INTERVAL = int(os.environ.get("VM_COPILOT_INTERVAL", "60"))
|
||||||
|
|
||||||
|
# Node IDs to watch (can be overridden via env var)
|
||||||
|
DEFAULT_NODES = [
|
||||||
|
"b5d5a72e-71d9-46d7-a4d7-a1c89d53224b", # gamma (machine-id derived)
|
||||||
|
]
|
||||||
|
WATCH_NODES = os.environ.get("VM_COPILOT_NODES", ",".join(DEFAULT_NODES)).split(",")
|
||||||
|
WATCH_NODES = [n.strip() for n in WATCH_NODES if n.strip()]
|
||||||
|
|
||||||
|
AUTHOR = "vm-copilot"
|
||||||
|
|
||||||
|
# Logging
|
||||||
|
logging.basicConfig(
|
||||||
|
level=logging.INFO,
|
||||||
|
format="%(asctime)s [%(levelname)s] %(message)s",
|
||||||
|
datefmt="%Y-%m-%d %H:%M:%S",
|
||||||
|
)
|
||||||
|
log = logging.getLogger("vm-copilot")
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
# State Tracking
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# Track conditions we've already alerted on: node_id -> set of condition keys
|
||||||
|
alerted_conditions: dict[str, set[str]] = {}
|
||||||
|
|
||||||
|
# Track last event timestamp per node for incremental fetching
|
||||||
|
last_event_ts: dict[str, str] = {}
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
# HTTP Helpers
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def get_node(node_id: str) -> Optional[dict]:
|
||||||
|
"""Fetch a single node's state from CC by querying the nodes list."""
|
||||||
|
try:
|
||||||
|
# /nodes returns JSON array of all nodes
|
||||||
|
resp = requests.get(f"{CC_URL}/nodes", timeout=10)
|
||||||
|
if resp.status_code == 200:
|
||||||
|
nodes = resp.json()
|
||||||
|
# Find the node by ID
|
||||||
|
for node in nodes:
|
||||||
|
if node.get("node_id") == node_id:
|
||||||
|
return node
|
||||||
|
log.warning(f"Node {node_id} not found in nodes list")
|
||||||
|
return None
|
||||||
|
else:
|
||||||
|
log.error(f"Failed to get nodes: {resp.status_code}")
|
||||||
|
return None
|
||||||
|
except requests.RequestException as e:
|
||||||
|
log.error(f"Request error getting nodes: {e}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def get_events(
|
||||||
|
node_id: str,
|
||||||
|
since: Optional[str] = None,
|
||||||
|
kinds: Optional[list[str]] = None,
|
||||||
|
limit: int = 50,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Query events for a node from CC."""
|
||||||
|
params = {"node_id": node_id, "limit": str(limit)}
|
||||||
|
if since:
|
||||||
|
params["since"] = since
|
||||||
|
if kinds:
|
||||||
|
params["kind"] = ",".join(kinds)
|
||||||
|
|
||||||
|
try:
|
||||||
|
resp = requests.get(f"{CC_URL}/api/events", params=params, timeout=10)
|
||||||
|
if resp.status_code == 200:
|
||||||
|
return resp.json()
|
||||||
|
else:
|
||||||
|
log.error(f"Failed to get events: {resp.status_code}")
|
||||||
|
return []
|
||||||
|
except requests.RequestException as e:
|
||||||
|
log.error(f"Request error getting events: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def post_event(kind: str, node_id: str, body: dict) -> Optional[dict]:
|
||||||
|
"""Post an event to CC."""
|
||||||
|
payload = {
|
||||||
|
"kind": kind,
|
||||||
|
"node_id": node_id,
|
||||||
|
"author": AUTHOR,
|
||||||
|
"body": body,
|
||||||
|
}
|
||||||
|
try:
|
||||||
|
resp = requests.post(
|
||||||
|
f"{CC_URL}/api/events",
|
||||||
|
json=payload,
|
||||||
|
headers={"Content-Type": "application/json"},
|
||||||
|
timeout=10,
|
||||||
|
)
|
||||||
|
if resp.status_code == 200:
|
||||||
|
result = resp.json()
|
||||||
|
log.info(f"Posted {kind} event: {result.get('id', 'unknown')}")
|
||||||
|
return result
|
||||||
|
else:
|
||||||
|
log.error(f"Failed to post event: {resp.status_code} - {resp.text}")
|
||||||
|
return None
|
||||||
|
except requests.RequestException as e:
|
||||||
|
log.error(f"Request error posting event: {e}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
# Rule Engine
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def analyze_node(node: dict, recent_events: list[dict]) -> list[tuple[str, str, dict]]:
|
||||||
|
"""
|
||||||
|
Analyze a node's state and return events to emit.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of (kind, condition_key, body) tuples
|
||||||
|
"""
|
||||||
|
to_emit = []
|
||||||
|
attention = node.get("attention", [])
|
||||||
|
hostname = node.get("hostname", "unknown")
|
||||||
|
node_id = node.get("node_id", "unknown")
|
||||||
|
|
||||||
|
# Rule 1: critical_findings - needs attention
|
||||||
|
if "critical_findings" in attention:
|
||||||
|
scan_summary = node.get("last_scan_summary")
|
||||||
|
crit_count = 0
|
||||||
|
if scan_summary:
|
||||||
|
crit_count = scan_summary.get("critical", 0) + scan_summary.get("high", 0)
|
||||||
|
|
||||||
|
to_emit.append((
|
||||||
|
"incident",
|
||||||
|
"critical_findings",
|
||||||
|
{
|
||||||
|
"title": "Critical Findings Detected",
|
||||||
|
"description": f"Node {hostname} has {crit_count} critical/high scan findings requiring attention",
|
||||||
|
"severity": "high",
|
||||||
|
},
|
||||||
|
))
|
||||||
|
to_emit.append((
|
||||||
|
"note",
|
||||||
|
"critical_findings_note",
|
||||||
|
{
|
||||||
|
"text": f"Auto-detected critical_findings on node {hostname}. Recommend manual review of scan results.",
|
||||||
|
"severity": "warn",
|
||||||
|
},
|
||||||
|
))
|
||||||
|
|
||||||
|
# Rule 2: heartbeat_stale - node may be offline
|
||||||
|
if "heartbeat_stale" in attention:
|
||||||
|
last_seen = node.get("last_seen", "unknown")
|
||||||
|
to_emit.append((
|
||||||
|
"incident",
|
||||||
|
"heartbeat_stale",
|
||||||
|
{
|
||||||
|
"title": "Node Heartbeat Stale",
|
||||||
|
"description": f"Node {hostname} has not sent a heartbeat recently. Last seen: {last_seen}",
|
||||||
|
"severity": "high",
|
||||||
|
},
|
||||||
|
))
|
||||||
|
|
||||||
|
# Rule 3: cloudflare_down - informational note
|
||||||
|
if "cloudflare_down" in attention:
|
||||||
|
to_emit.append((
|
||||||
|
"note",
|
||||||
|
"cloudflare_down",
|
||||||
|
{
|
||||||
|
"text": f"Node {hostname} reports Cloudflare tunnel is down.",
|
||||||
|
"severity": "warn",
|
||||||
|
},
|
||||||
|
))
|
||||||
|
|
||||||
|
# Rule 4: never_scanned - needs initial scan
|
||||||
|
if "never_scanned" in attention:
|
||||||
|
to_emit.append((
|
||||||
|
"note",
|
||||||
|
"never_scanned",
|
||||||
|
{
|
||||||
|
"text": f"Node {hostname} has never been scanned. Recommend triggering initial sovereign scan.",
|
||||||
|
"severity": "info",
|
||||||
|
},
|
||||||
|
))
|
||||||
|
|
||||||
|
return to_emit
|
||||||
|
|
||||||
|
|
||||||
|
def has_recent_operator_ack(events: list[dict]) -> bool:
|
||||||
|
"""Check if there's a recent ack from someone other than vm-copilot."""
|
||||||
|
for ev in events:
|
||||||
|
if ev.get("kind") == "ack":
|
||||||
|
author = ev.get("author", "")
|
||||||
|
if author and author != AUTHOR:
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
# Main Loop
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def run_cycle():
|
||||||
|
"""Run a single monitoring cycle."""
|
||||||
|
for node_id in WATCH_NODES:
|
||||||
|
log.debug(f"Checking node {node_id}")
|
||||||
|
|
||||||
|
# Fetch node state
|
||||||
|
node = get_node(node_id)
|
||||||
|
if not node:
|
||||||
|
continue
|
||||||
|
|
||||||
|
hostname = node.get("hostname", node_id[:8])
|
||||||
|
attention = node.get("attention", [])
|
||||||
|
log.info(f"Node {hostname}: attention={attention}")
|
||||||
|
|
||||||
|
# Fetch recent events
|
||||||
|
since = last_event_ts.get(node_id)
|
||||||
|
events = get_events(
|
||||||
|
node_id,
|
||||||
|
since=since,
|
||||||
|
kinds=["note", "incident", "ack", "resolve"],
|
||||||
|
)
|
||||||
|
|
||||||
|
# Update last seen timestamp
|
||||||
|
if events:
|
||||||
|
# Events are returned oldest-first by default, get the latest
|
||||||
|
latest_ts = max(ev.get("ts", "") for ev in events)
|
||||||
|
if latest_ts:
|
||||||
|
last_event_ts[node_id] = latest_ts
|
||||||
|
|
||||||
|
# Check for operator acks
|
||||||
|
recent_ack = has_recent_operator_ack(events)
|
||||||
|
if recent_ack:
|
||||||
|
log.info(f"Node {hostname}: operator ack detected, resetting alerts")
|
||||||
|
alerted_conditions.pop(node_id, None)
|
||||||
|
|
||||||
|
# Analyze and emit events
|
||||||
|
to_emit = analyze_node(node, events)
|
||||||
|
|
||||||
|
for kind, condition_key, body in to_emit:
|
||||||
|
# Skip if we've already alerted on this condition
|
||||||
|
node_alerts = alerted_conditions.setdefault(node_id, set())
|
||||||
|
if condition_key in node_alerts:
|
||||||
|
log.debug(f"Already alerted on {condition_key} for {hostname}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Post the event
|
||||||
|
result = post_event(kind, node_id, body)
|
||||||
|
if result:
|
||||||
|
node_alerts.add(condition_key)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""Main entry point."""
|
||||||
|
log.info("=" * 60)
|
||||||
|
log.info("VaultMesh vm-copilot starting")
|
||||||
|
log.info(f" CC URL: {CC_URL}")
|
||||||
|
log.info(f" Poll interval: {POLL_INTERVAL}s")
|
||||||
|
log.info(f" Watching nodes: {WATCH_NODES}")
|
||||||
|
log.info("=" * 60)
|
||||||
|
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
run_cycle()
|
||||||
|
except Exception as e:
|
||||||
|
log.exception(f"Error in monitoring cycle: {e}")
|
||||||
|
|
||||||
|
log.info(f"Sleeping {POLL_INTERVAL}s...")
|
||||||
|
time.sleep(POLL_INTERVAL)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
log.info("Shutting down...")
|
||||||
|
sys.exit(0)
|
||||||
Reference in New Issue
Block a user