Initial commit: Cloudflare infrastructure with WAF Intelligence
- Complete Cloudflare Terraform configuration (DNS, WAF, tunnels, access) - WAF Intelligence MCP server with threat analysis and ML classification - GitOps automation with PR workflows and drift detection - Observatory monitoring stack with Prometheus/Grafana - IDE operator rules for governed development - Security playbooks and compliance frameworks - Autonomous remediation and state reconciliation
This commit is contained in:
Vault Sovereign
386
archive_docs/CLEANUP_COMPLETE.md
Normal file
386
archive_docs/CLEANUP_COMPLETE.md
Normal file
@@ -0,0 +1,386 @@
|
||||
# CLEANUP COMPLETE: B + C Refactoring Summary
|
||||
|
||||
**Status:** ✅ All 6 cleanup tasks completed
|
||||
**Date:** December 8, 2025
|
||||
**Purpose:** Eliminate code chaos and establish guardrails for agent automation
|
||||
|
||||
---
|
||||
|
||||
## What Was The Problem?
|
||||
|
||||
During Phase 6, autonomous agents (Cline/Claude) were making **patch edits** to files without understanding the whole context:
|
||||
|
||||
- **oracle_answer_mcp.py** ended up with **duplicate argparse definitions** (`--question` defined twice)
|
||||
- This caused: `argparse.ArgumentError: argument --question: conflicting option string`
|
||||
- Root cause: Agent appended code without reading the entire file
|
||||
|
||||
Result: **Code drift** — multiple similar code blocks, unclear which is the "real" version.
|
||||
|
||||
---
|
||||
|
||||
## Solution: B + C Strategy
|
||||
|
||||
### B — Restructure oracle_answer around proper MCP package layout
|
||||
### C — Establish guardrails so agents stop auto-patching blind
|
||||
|
||||
---
|
||||
|
||||
## B: Clean Package Structure
|
||||
|
||||
### Before (Chaos)
|
||||
```
|
||||
CLOUDFLARE/
|
||||
├── oracle_answer_mcp.py # Monolithic, 332 lines, mixed concerns
|
||||
├── oracle_runner.py # Separate oracle logic
|
||||
├── mcp/
|
||||
│ ├── oracle_answer/
|
||||
│ │ └── __init__.py # Just __version__, missing exports
|
||||
│ └── (empty)
|
||||
└── (no clear separation)
|
||||
```
|
||||
|
||||
**Problem:** Three different places doing similar things. Agents don't know which is authoritative.
|
||||
|
||||
### After (Clean)
|
||||
|
||||
```
|
||||
CLOUDFLARE/
|
||||
├── mcp/
|
||||
│ ├── __init__.py # Package marker
|
||||
│ └── oracle_answer/
|
||||
│ ├── __init__.py # Exports OracleAnswerTool, ToolResponse
|
||||
│ ├── tool.py # Core logic (OracleAnswerTool class)
|
||||
│ └── cli.py # CLI wrapper (optional entry point)
|
||||
│
|
||||
├── oracle_answer_mcp.py # DEPRECATED: backward compat wrapper
|
||||
├── oracle_runner.py # Separate concern (document search)
|
||||
├── AGENT_GUARDRAILS.md # NEW: Rules for agents (C1)
|
||||
└── STRUCTURE.md # Architecture documentation
|
||||
```
|
||||
|
||||
**Benefit:** Clear separation of concerns. Agents know exactly where to edit.
|
||||
|
||||
---
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
### ✅ B1: mcp/__init__.py
|
||||
```python
|
||||
"""
|
||||
MCP tools for the CLOUDFLARE workspace.
|
||||
Currently:
|
||||
- oracle_answer: compliance / security oracle
|
||||
"""
|
||||
```
|
||||
**Purpose:** Package marker. Nothing fancy.
|
||||
|
||||
### ✅ B2: mcp/oracle_answer/__init__.py (Rewritten)
|
||||
```python
|
||||
from .tool import OracleAnswerTool, ToolResponse
|
||||
|
||||
__version__ = "0.2.0"
|
||||
__all__ = ["OracleAnswerTool", "ToolResponse", "__version__"]
|
||||
```
|
||||
**Before:** Missing exports (pyright error)
|
||||
**After:** Proper exports that are actually defined in tool.py
|
||||
|
||||
### ✅ B3: mcp/oracle_answer/tool.py (New)
|
||||
```python
|
||||
@dataclass
|
||||
class ToolResponse:
|
||||
answer: str
|
||||
framework_hits: Dict[str, List[str]]
|
||||
reasoning: Optional[str] = None
|
||||
|
||||
class OracleAnswerTool:
|
||||
async def answer(self, question: str, ...) -> ToolResponse:
|
||||
"""Main entry point for MCP / clients."""
|
||||
# Core logic here
|
||||
```
|
||||
**Purpose:** Single responsibility — answer compliance questions.
|
||||
**Benefit:** Easy to test, easy to plug into MCP server or CLI.
|
||||
|
||||
### ✅ B4: mcp/oracle_answer/cli.py (New)
|
||||
```python
|
||||
# NOTE FOR AUTOMATION:
|
||||
# - All CLI arguments must be defined ONLY in build_parser().
|
||||
# - When changing CLI flags, rewrite build_parser() entirely.
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
"""Single source of truth for CLI args."""
|
||||
parser = argparse.ArgumentParser(...)
|
||||
parser.add_argument("--question", required=True)
|
||||
parser.add_argument("--frameworks", nargs="*")
|
||||
parser.add_argument("--mode", choices=["strict", "advisory"])
|
||||
parser.add_argument("--json", action="store_true")
|
||||
return parser
|
||||
|
||||
async def main_async(args: Optional[List[str]] = None) -> int:
|
||||
tool = OracleAnswerTool(...)
|
||||
resp = await tool.answer(...)
|
||||
print(...)
|
||||
return 0
|
||||
```
|
||||
**Purpose:** CLI wrapper (optional). Separates argument handling from logic.
|
||||
**Key:** `build_parser()` is the single source of truth for all CLI args.
|
||||
**Benefit:** Agents can't accidentally add duplicate `--question` flags anymore.
|
||||
|
||||
### ✅ C1: AGENT_GUARDRAILS.md (New)
|
||||
305 lines of explicit rules:
|
||||
|
||||
1. **Argparse Rule:** All args defined ONLY in `build_parser()`, never elsewhere
|
||||
2. **Duplicate Rule:** Check for duplicates before editing
|
||||
3. **Read First Rule:** Read ENTIRE file before making edits
|
||||
4. **SRP Rule:** Each file has one responsibility
|
||||
5. **Type Hints Rule:** All functions must have type annotations
|
||||
6. **Docstring Rule:** Every module/class/function needs docs
|
||||
|
||||
**Purpose:** Paste this into Cline before asking it to edit code.
|
||||
|
||||
### ✅ C2: oracle_answer_mcp.py (Deprecated)
|
||||
```python
|
||||
"""
|
||||
DEPRECATED: Use mcp.oracle_answer instead
|
||||
This file is kept for backward compatibility only.
|
||||
"""
|
||||
|
||||
warnings.warn(
|
||||
"oracle_answer_mcp.py is deprecated. "
|
||||
"Use 'from mcp.oracle_answer import OracleAnswerTool' instead."
|
||||
)
|
||||
|
||||
# For backward compatibility, re-export from new location
|
||||
from mcp.oracle_answer import OracleAnswerTool, ToolResponse
|
||||
```
|
||||
**Purpose:** Soft migration. Old code still works but gets warned.
|
||||
**Timeline:** Can be deleted after 30 days (once all code migrated).
|
||||
|
||||
---
|
||||
|
||||
## Key Improvements
|
||||
|
||||
| Aspect | Before | After |
|
||||
|--------|--------|-------|
|
||||
| **Organization** | oracle_answer_mcp.py at root (monolithic) | Proper mcp/ package structure |
|
||||
| **Separation** | CLI + tool logic mixed in one 332-line file | tool.py (logic) + cli.py (wrapper) |
|
||||
| **Exports** | `__all__ = [undefined names]` | Proper exports from tool.py |
|
||||
| **Argparse** | No guard against duplicate flags | Single build_parser() + guardrails |
|
||||
| **Agent safety** | No rules; chaos ensues | AGENT_GUARDRAILS.md provides clear rules |
|
||||
| **Backward compat** | Breakage when moving files | Deprecation wrapper + 30-day migration |
|
||||
| **Type hints** | Mixed coverage | All functions properly typed |
|
||||
|
||||
---
|
||||
|
||||
## How to Use The New Structure
|
||||
|
||||
### 1. CLI Usage
|
||||
```bash
|
||||
# Old way (deprecated)
|
||||
python3 oracle_answer_mcp.py --question "GDPR?"
|
||||
|
||||
# New way
|
||||
python3 -m mcp.oracle_answer.cli --question "GDPR?"
|
||||
|
||||
# Or as Python import
|
||||
from mcp.oracle_answer import OracleAnswerTool
|
||||
tool = OracleAnswerTool()
|
||||
response = await tool.answer("GDPR?")
|
||||
```
|
||||
|
||||
### 2. For MCP Integration
|
||||
```python
|
||||
from mcp.oracle_answer import OracleAnswerTool, ToolResponse
|
||||
|
||||
# In your MCP server handler:
|
||||
tool = OracleAnswerTool()
|
||||
response = await tool.answer(question, frameworks=["ISO-27001"])
|
||||
# Returns ToolResponse with answer, framework_hits, reasoning
|
||||
```
|
||||
|
||||
### 3. For Testing
|
||||
```python
|
||||
import asyncio
|
||||
from mcp.oracle_answer import OracleAnswerTool
|
||||
|
||||
async def test():
|
||||
tool = OracleAnswerTool()
|
||||
resp = await tool.answer("Test question")
|
||||
assert resp.answer is not None
|
||||
print(resp.reasoning)
|
||||
|
||||
asyncio.run(test())
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Agent Guardrails (Copy This Into Cline)
|
||||
|
||||
Before asking Cline to edit Python files in this repo, paste:
|
||||
|
||||
```
|
||||
SESSION GUARDRAILS (CLOUDFLARE)
|
||||
|
||||
Follow AGENT_GUARDRAILS.md in the repo root.
|
||||
|
||||
1. CLI Arguments:
|
||||
- All CLI args defined ONLY in build_parser()
|
||||
- Rewrite build_parser() entirely when changing args
|
||||
- DO NOT append add_argument() calls elsewhere
|
||||
|
||||
2. File Layout:
|
||||
- New tools go in mcp/<tool_name>/
|
||||
- New scripts go in scripts/
|
||||
- New observability code goes in observatory/
|
||||
- DO NOT create new files at repo root without explicit request
|
||||
|
||||
3. __all__ / Exports:
|
||||
- If modifying __init__.py, ensure all names in __all__ are imported
|
||||
- Example: if __all__ = ["X", "Y"], then X and Y must be defined or imported
|
||||
|
||||
4. Refactoring:
|
||||
- Rewrite whole functions, not line-by-line patches
|
||||
- Read entire file before editing
|
||||
- Check for duplicates (grep for function name, arg name, etc.)
|
||||
|
||||
5. Type Hints:
|
||||
- All functions must have parameter types and return types
|
||||
- Use Optional[T] for optional values
|
||||
|
||||
6. Safety:
|
||||
- Do not modify .env, secrets, or Cloudflare/DNS constants
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing The New Structure
|
||||
|
||||
```bash
|
||||
# Verify imports work
|
||||
python3 -c "from mcp.oracle_answer import OracleAnswerTool; print('✓')"
|
||||
|
||||
# Verify CLI works
|
||||
python3 -m mcp.oracle_answer.cli --help
|
||||
|
||||
# Verify backward compat
|
||||
python3 -c "from oracle_answer_mcp import OracleAnswerTool; print('✓ deprecated')"
|
||||
|
||||
# Verify package structure
|
||||
ls -R CLOUDFLARE/mcp/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Migration Timeline
|
||||
|
||||
### Now (Dec 8, 2025)
|
||||
- ✅ New structure deployed
|
||||
- ✅ Backward compat wrapper in place
|
||||
- ✅ Guardrails documented
|
||||
|
||||
### Week 1
|
||||
- Update any local scripts that import oracle_answer_mcp.py
|
||||
- Change to: `from mcp.oracle_answer import OracleAnswerTool`
|
||||
|
||||
### Week 2
|
||||
- Update CI/CD, docs, examples
|
||||
- Verify no code imports from oracle_answer_mcp.py
|
||||
|
||||
### Week 3+
|
||||
- Delete oracle_answer_mcp.py (safe, been replaced for 2+ weeks)
|
||||
- Deprecation warning goes away
|
||||
|
||||
---
|
||||
|
||||
## What This Prevents
|
||||
|
||||
### Problem 1: Duplicate Argparse Definitions
|
||||
**Before:**
|
||||
```python
|
||||
parser.add_argument("--question", required=False) # Line 50
|
||||
...
|
||||
parser.add_argument("--question", required=True) # Line 200
|
||||
# Error: conflicting option string --question
|
||||
```
|
||||
|
||||
**After:**
|
||||
```python
|
||||
def build_parser(): # SINGLE SOURCE OF TRUTH
|
||||
parser.add_argument("--question", required=False)
|
||||
return parser
|
||||
```
|
||||
With guardrails: Agent knows to rewrite build_parser() as a whole, not patch random lines.
|
||||
|
||||
### Problem 2: Code Drift
|
||||
**Before:** Different versions of the same logic scattered across files.
|
||||
|
||||
**After:** Clear ownership:
|
||||
- `tool.py` = oracle logic (one place)
|
||||
- `cli.py` = argument handling (one place)
|
||||
- `__init__.py` = exports (one place)
|
||||
|
||||
### Problem 3: Agent Blind Patching
|
||||
**Before:** Agent would insert lines without reading context.
|
||||
|
||||
**After:** Guardrails + clear structure means:
|
||||
1. Agent knows which file to edit (tool.py for logic, cli.py for CLI)
|
||||
2. Agent reads ENTIRE file first (guardrails enforce this)
|
||||
3. Agent rewrites whole function (not patch)
|
||||
4. Guardrails prevent duplicates by design
|
||||
|
||||
---
|
||||
|
||||
## File Stats
|
||||
|
||||
| File | Lines | Purpose |
|
||||
|------|-------|---------|
|
||||
| mcp/__init__.py | 6 | Package marker |
|
||||
| mcp/oracle_answer/__init__.py | 10 | Exports |
|
||||
| mcp/oracle_answer/tool.py | 75 | Core logic |
|
||||
| mcp/oracle_answer/cli.py | 95 | CLI wrapper |
|
||||
| AGENT_GUARDRAILS.md | 305 | Rules for agents |
|
||||
| oracle_answer_mcp.py | 27 | Deprecation wrapper |
|
||||
| **Total** | **518** | Clean, modular code |
|
||||
|
||||
**Compared to before:** 332-line monolith → 186 lines of focused code + 305 lines of guardrails.
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Test the new structure:**
|
||||
```bash
|
||||
python3 -m mcp.oracle_answer.cli --question "Test?" --json
|
||||
```
|
||||
|
||||
2. **Update your imports:**
|
||||
- Old: `from oracle_answer_mcp import OracleAnswerTool`
|
||||
- New: `from mcp.oracle_answer import OracleAnswerTool`
|
||||
|
||||
3. **Use guardrails with agents:**
|
||||
- Paste AGENT_GUARDRAILS.md into Cline before editing
|
||||
- Agents will follow the rules
|
||||
|
||||
4. **Plan for Phase 7 (WAF Intelligence):**
|
||||
- New MCP tool: `mcp/waf_intelligence/`
|
||||
- New script: `observatory/waf-intel.py`
|
||||
- Follow same pattern (tool.py + optional cli.py)
|
||||
|
||||
---
|
||||
|
||||
## Sign-Off
|
||||
|
||||
✅ **Structure:** Clean, modular, scalable
|
||||
✅ **Safety:** Guardrails prevent common errors
|
||||
✅ **Backward Compat:** Old code still works (with deprecation warning)
|
||||
✅ **Ready for Phase 7:** New tools can follow this exact pattern
|
||||
✅ **Agent-Proof:** Explicit rules prevent chaos
|
||||
|
||||
---
|
||||
|
||||
**Version:** 1.0
|
||||
**Date:** December 8, 2025
|
||||
**Status:** 🟢 Ready for Production
|
||||
|
||||
The chaos is contained. Agents now have clear rules. Structure is clean.
|
||||
|
||||
You're ready for the next phase.
|
||||
Reference in New Issue
Block a user