docs: security incident resolution & firewall implementation

- Document network scanning incident (Dec 6 2025 - Jan 8 2026) - Add firewall rules to prevent internal network access - Deploy monitoring script for scanning attempts - Update INFRASTRUCTURE.md v2.2.0 with Security section - Update infrastructure_quick_ref.ipynb v2.1.0 - Root cause: compromised daarion-web container with crypto miner - Resolution: container removed, firewall applied, monitoring deployed Co-Authored-By: Warp <agent@warp.dev>
2026-01-08 11:52:53 -08:00
parent e3a8b7464a
commit e829fe66f2
2 changed files with 153 additions and 6 deletions
--- a/INFRASTRUCTURE.md
+++ b/INFRASTRUCTURE.md
@@ -1,9 +1,13 @@
 # 🏗️ Infrastructure Overview — DAARION & MicroDAO

-**Версія:** 2.1.0  
-**Останнє оновлення:** 2025-11-23 18:05  
+**Версія:** 2.2.0  
+**Останнє оновлення:** 2026-01-08 19:30  
 **Статус:** Production Ready (95% Multimodal Integration)  
 **Останні зміни:** 
+- 🔒 **Security Incident Resolution** (Dec 6 2025 - Jan 8 2026)
+- ✅ Compromised container removed (`daarion-web`)
+- ✅ Firewall rules implemented (egress filtering)
+- ✅ Monitoring for scanning attempts deployed
 - ✅ Router Multimodal API (v1.1.0) - images/files/audio/web-search
 - ✅ Telegram Gateway Multimodal - voice/photo/documents
 - ✅ Frontend Multimodal UI - enhanced mode
@@ -1022,3 +1026,84 @@ User → @YaromirBot (Telegram)
 **Status:** 95% Complete  
 **Production Ready:** ✅ Yes (with fallbacks)

+---
+
+## 🔒 Security & Incident Response
+
+### Incident #1: Network Scanning & Server Lockdown (Dec 6, 2025 - Jan 8, 2026)
+
+**Timeline:**
+- **Dec 6, 2025 10:56 UTC**: Automated SSH scanning detected from server
+- **Dec 6, 2025 11:00 UTC**: Hetzner locked server IP (144.76.224.179)
+- **Jan 8, 2026 18:00 UTC**: Unlock request approved, server recovered
+
+**Root Cause:**
+- Server compromised with cryptocurrency miner (`catcal`, `G4NQXBp`) via `daarion-web` container
+- Miner performed network scanning of Hetzner internal network (10.126.0.0/16)
+- ~500+ SSH connection attempts to internal IP range triggered automated block
+- High CPU load (35+) from mining process
+
+**Impact:**
+- ❌ Server unavailable for 33 days
+- ❌ All services down
+- ❌ Telegram bots offline
+- ❌ Lost production data/monitoring
+
+**Resolution:**
+1. ✅ Server recovered via rescue mode
+2. ✅ Compromised `daarion-web` container stopped and removed
+3. ✅ Cryptocurrency miner processes killed
+4. ✅ Firewall rules implemented to block internal network access
+5. ✅ Monitoring script deployed for future scanning attempts
+
+**Prevention Measures:**
+
+**Firewall Rules:**
+```bash
+# Block Hetzner internal networks
+iptables -I OUTPUT -d 10.0.0.0/8 -j DROP
+iptables -I OUTPUT -d 172.16.0.0/12 -j DROP
+
+# Allow only necessary ports
+iptables -I OUTPUT -d 10.0.0.0/8 -p tcp --dport 443 -j ACCEPT
+iptables -I OUTPUT -d 10.0.0.0/8 -p tcp --dport 80 -j ACCEPT
+
+# Log blocked attempts
+iptables -I OUTPUT -d 10.0.0.0/8 -j LOG --log-prefix "BLOCKED_INTERNAL_SCAN: "
+
+# Save rules
+iptables-save > /etc/iptables/rules.v4
+```
+
+**Monitoring:**
+- Script: `/root/monitor_scanning.sh`
+- Runs every 15 minutes via cron
+- Logs to `/var/log/scan_attempts.log`
+- Checks for:
+  - Suspicious network activity in Docker logs
+  - iptables blocked connection attempts
+  - Keywords: `10.126`, `172.16`, `scan`, `probe`
+
+**Security Checklist:**
+- [ ] Review all Docker images for vulnerabilities
+- [ ] Implement container security scanning (Trivy/Clair)
+- [ ] Enable Docker Content Trust
+- [ ] Set up intrusion detection (fail2ban)
+- [ ] Regular security audits
+- [ ] Container resource limits (CPU/memory)
+- [ ] Network segmentation for containers
+
+**References:**
+- Hetzner Incident ID: `L00280548`
+- Guideline: https://docs.hetzner.com/robot/dedicated-server/troubleshooting/guideline-in-case-of-server-locking/
+- Recovery Scripts: `/root/prevent_scanning.sh`, `/root/monitor_scanning.sh`
+
+**Lessons Learned:**
+1. 🔴 **Never expose containers without security scanning**
+2. 🟡 **Implement egress firewall rules from day 1**
+3. 🟢 **Monitor outgoing connections, not just incoming**
+4. 🔵 **Have disaster recovery plan documented**
+5. 🟣 **Regular security audits are critical**
+
+---
+
--- a/docs/infrastructure_quick_ref.ipynb
+++ b/docs/infrastructure_quick_ref.ipynb
@@ -6,12 +6,16 @@
   "source": [
    "# 🚀 Infrastructure Quick Reference — DAARION & MicroDAO\n",
    "\n",
-    "**Версія:** 2.0.0  \n",
-    "**Останнє оновлення:** 2025-11-23  \n",
+    "Версія:** 2.1.0  \n",
+    "Останнє оновлення:** 2026-01-08  \n",
    "\n",
    "Цей notebook містить швидкий довідник по серверах, репозиторіях та endpoints для DAGI Stack.\n",
    "\n",
-    "**NEW (v2.0.0):** \n",
+    "**NEW (v2.1.0):** \n",
+    "- 🔒 **Security Incident Resolved** (Dec 2025 - Jan 2026)\n",
+    "- ✅ Firewall rules + monitoring deployed\n",
+    "\n",
+    "**v2.0.0:** \n",
    "- ✅ Мультимодальні сервіси (STT, OCR, Web Search, Vector DB) на НОДА2\n",
    "- ✅ Router Multimodal Support (інтеграція в процесі)\n",
    "- ✅ Telegram Gateway Enhanced (STT + Vision)\n",
@@ -465,6 +469,64 @@
    "pd.DataFrame(multimodal_capabilities).T\n"
   ]
  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 🔒 Security & Incident Response\n",
+    "\n",
+    "### Incident #1: Network Scanning & Lockdown (Dec 6, 2025 - Jan 8, 2026)\n",
+    "\n",
+    "**Root Cause:** Compromised `daarion-web` container with cryptocurrency miner\n",
+    "**Impact:** Server locked by Hetzner for 33 days due to internal network scanning\n",
+    "**Resolution:** Container removed, firewall rules implemented, monitoring deployed\n",
+    "\n",
+    "### Security Measures\n",
+    "\n",
+    "1. **Egress Firewall Rules** (блокування внутрішніх мереж Hetzner)\n",
+    "2. **Monitoring Script** (`/root/monitor_scanning.sh`, runs every 15 min)\n",
+    "3. **Security Checklist:**\n",
+    "   - [ ] Container vulnerability scanning\n",
+    "   - [ ] Docker Content Trust\n",
+    "   - [ ] Resource limits (CPU/memory)\n",
+    "   - [ ] Network segmentation\n",
+    "   - [ ] Regular security audits\n",
+    "\n",
+    "**Full details:** See `INFRASTRUCTURE.md` → Security & Incident Response section\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Security Configuration\n",
+    "security_config = {\n",
+    "    \"Firewall Rules\": {\n",
+    "        \"script\": \"/root/prevent_scanning.sh\",\n",
+    "        \"status\": \"✅ Active\",\n",
+    "        \"blocks\": [\"10.0.0.0/8\", \"172.16.0.0/12\"],\n",
+    "        \"allows\": [\"80/tcp\", \"443/tcp\"]\n",
+    "    },\n",
+    "    \"Monitoring\": {\n",
+    "        \"script\": \"/root/monitor_scanning.sh\",\n",
+    "        \"status\": \"✅ Active\",\n",
+    "        \"interval\": \"15 minutes\",\n",
+    "        \"log\": \"/var/log/scan_attempts.log\"\n",
+    "    },\n",
+    "    \"Incident Response\": {\n",
+    "        \"last_incident\": \"2025-12-06\",\n",
+    "        \"recovery_time\": \"33 days\",\n",
+    "        \"status\": \"✅ Resolved\",\n",
+    "        \"prevention\": \"Firewall + Monitoring\"\n",
+    "    }\n",
+    "}\n",
+    "\n",
+    "import pandas as pd\n",
+    "pd.DataFrame(security_config).T\n"
+   ]
+  },
  {
   "cell_type": "markdown",
   "metadata": {},
@@ -499,7 +561,7 @@
    "\n",
    "---\n",
    "\n",
-    "**Last Updated:** 2025-11-23 by Auto AI  \n",
+    "**Last Updated:** 2026-01-08 (Security incident resolution & firewall implementation)  \n",
    "**Maintained by:** Ivan Tytar & DAARION Team"
   ]
  }