From e829fe66f2fdd1b40f22ecb243ef95638fd718e6 Mon Sep 17 00:00:00 2001 From: Apple Date: Thu, 8 Jan 2026 11:52:53 -0800 Subject: [PATCH] docs: security incident resolution & firewall implementation - Document network scanning incident (Dec 6 2025 - Jan 8 2026) - Add firewall rules to prevent internal network access - Deploy monitoring script for scanning attempts - Update INFRASTRUCTURE.md v2.2.0 with Security section - Update infrastructure_quick_ref.ipynb v2.1.0 - Root cause: compromised daarion-web container with crypto miner - Resolution: container removed, firewall applied, monitoring deployed Co-Authored-By: Warp --- INFRASTRUCTURE.md | 89 ++++++++++++++++++++++++++++- docs/infrastructure_quick_ref.ipynb | 70 +++++++++++++++++++++-- 2 files changed, 153 insertions(+), 6 deletions(-) diff --git a/INFRASTRUCTURE.md b/INFRASTRUCTURE.md index a4e615b0..f75eac6f 100644 --- a/INFRASTRUCTURE.md +++ b/INFRASTRUCTURE.md @@ -1,9 +1,13 @@ # 🏗️ Infrastructure Overview — DAARION & MicroDAO -**Версія:** 2.1.0 -**Останнє оновлення:** 2025-11-23 18:05 +**Версія:** 2.2.0 +**Останнє оновлення:** 2026-01-08 19:30 **Статус:** Production Ready (95% Multimodal Integration) **Останні зміни:** +- 🔒 **Security Incident Resolution** (Dec 6 2025 - Jan 8 2026) +- ✅ Compromised container removed (`daarion-web`) +- ✅ Firewall rules implemented (egress filtering) +- ✅ Monitoring for scanning attempts deployed - ✅ Router Multimodal API (v1.1.0) - images/files/audio/web-search - ✅ Telegram Gateway Multimodal - voice/photo/documents - ✅ Frontend Multimodal UI - enhanced mode @@ -1022,3 +1026,84 @@ User → @YaromirBot (Telegram) **Status:** 95% Complete **Production Ready:** ✅ Yes (with fallbacks) +--- + +## 🔒 Security & Incident Response + +### Incident #1: Network Scanning & Server Lockdown (Dec 6, 2025 - Jan 8, 2026) + +**Timeline:** +- **Dec 6, 2025 10:56 UTC**: Automated SSH scanning detected from server +- **Dec 6, 2025 11:00 UTC**: Hetzner locked server IP (144.76.224.179) +- **Jan 8, 2026 18:00 UTC**: Unlock request approved, server recovered + +**Root Cause:** +- Server compromised with cryptocurrency miner (`catcal`, `G4NQXBp`) via `daarion-web` container +- Miner performed network scanning of Hetzner internal network (10.126.0.0/16) +- ~500+ SSH connection attempts to internal IP range triggered automated block +- High CPU load (35+) from mining process + +**Impact:** +- ❌ Server unavailable for 33 days +- ❌ All services down +- ❌ Telegram bots offline +- ❌ Lost production data/monitoring + +**Resolution:** +1. ✅ Server recovered via rescue mode +2. ✅ Compromised `daarion-web` container stopped and removed +3. ✅ Cryptocurrency miner processes killed +4. ✅ Firewall rules implemented to block internal network access +5. ✅ Monitoring script deployed for future scanning attempts + +**Prevention Measures:** + +**Firewall Rules:** +```bash +# Block Hetzner internal networks +iptables -I OUTPUT -d 10.0.0.0/8 -j DROP +iptables -I OUTPUT -d 172.16.0.0/12 -j DROP + +# Allow only necessary ports +iptables -I OUTPUT -d 10.0.0.0/8 -p tcp --dport 443 -j ACCEPT +iptables -I OUTPUT -d 10.0.0.0/8 -p tcp --dport 80 -j ACCEPT + +# Log blocked attempts +iptables -I OUTPUT -d 10.0.0.0/8 -j LOG --log-prefix "BLOCKED_INTERNAL_SCAN: " + +# Save rules +iptables-save > /etc/iptables/rules.v4 +``` + +**Monitoring:** +- Script: `/root/monitor_scanning.sh` +- Runs every 15 minutes via cron +- Logs to `/var/log/scan_attempts.log` +- Checks for: + - Suspicious network activity in Docker logs + - iptables blocked connection attempts + - Keywords: `10.126`, `172.16`, `scan`, `probe` + +**Security Checklist:** +- [ ] Review all Docker images for vulnerabilities +- [ ] Implement container security scanning (Trivy/Clair) +- [ ] Enable Docker Content Trust +- [ ] Set up intrusion detection (fail2ban) +- [ ] Regular security audits +- [ ] Container resource limits (CPU/memory) +- [ ] Network segmentation for containers + +**References:** +- Hetzner Incident ID: `L00280548` +- Guideline: https://docs.hetzner.com/robot/dedicated-server/troubleshooting/guideline-in-case-of-server-locking/ +- Recovery Scripts: `/root/prevent_scanning.sh`, `/root/monitor_scanning.sh` + +**Lessons Learned:** +1. 🔴 **Never expose containers without security scanning** +2. 🟡 **Implement egress firewall rules from day 1** +3. 🟢 **Monitor outgoing connections, not just incoming** +4. 🔵 **Have disaster recovery plan documented** +5. 🟣 **Regular security audits are critical** + +--- + diff --git a/docs/infrastructure_quick_ref.ipynb b/docs/infrastructure_quick_ref.ipynb index 5f98bbb6..37e2a297 100644 --- a/docs/infrastructure_quick_ref.ipynb +++ b/docs/infrastructure_quick_ref.ipynb @@ -6,12 +6,16 @@ "source": [ "# 🚀 Infrastructure Quick Reference — DAARION & MicroDAO\n", "\n", - "**Версія:** 2.0.0 \n", - "**Останнє оновлення:** 2025-11-23 \n", + "Версія:** 2.1.0 \n", + "Останнє оновлення:** 2026-01-08 \n", "\n", "Цей notebook містить швидкий довідник по серверах, репозиторіях та endpoints для DAGI Stack.\n", "\n", - "**NEW (v2.0.0):** \n", + "**NEW (v2.1.0):** \n", + "- 🔒 **Security Incident Resolved** (Dec 2025 - Jan 2026)\n", + "- ✅ Firewall rules + monitoring deployed\n", + "\n", + "**v2.0.0:** \n", "- ✅ Мультимодальні сервіси (STT, OCR, Web Search, Vector DB) на НОДА2\n", "- ✅ Router Multimodal Support (інтеграція в процесі)\n", "- ✅ Telegram Gateway Enhanced (STT + Vision)\n", @@ -465,6 +469,64 @@ "pd.DataFrame(multimodal_capabilities).T\n" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 🔒 Security & Incident Response\n", + "\n", + "### Incident #1: Network Scanning & Lockdown (Dec 6, 2025 - Jan 8, 2026)\n", + "\n", + "**Root Cause:** Compromised `daarion-web` container with cryptocurrency miner\n", + "**Impact:** Server locked by Hetzner for 33 days due to internal network scanning\n", + "**Resolution:** Container removed, firewall rules implemented, monitoring deployed\n", + "\n", + "### Security Measures\n", + "\n", + "1. **Egress Firewall Rules** (блокування внутрішніх мереж Hetzner)\n", + "2. **Monitoring Script** (`/root/monitor_scanning.sh`, runs every 15 min)\n", + "3. **Security Checklist:**\n", + " - [ ] Container vulnerability scanning\n", + " - [ ] Docker Content Trust\n", + " - [ ] Resource limits (CPU/memory)\n", + " - [ ] Network segmentation\n", + " - [ ] Regular security audits\n", + "\n", + "**Full details:** See `INFRASTRUCTURE.md` → Security & Incident Response section\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Security Configuration\n", + "security_config = {\n", + " \"Firewall Rules\": {\n", + " \"script\": \"/root/prevent_scanning.sh\",\n", + " \"status\": \"✅ Active\",\n", + " \"blocks\": [\"10.0.0.0/8\", \"172.16.0.0/12\"],\n", + " \"allows\": [\"80/tcp\", \"443/tcp\"]\n", + " },\n", + " \"Monitoring\": {\n", + " \"script\": \"/root/monitor_scanning.sh\",\n", + " \"status\": \"✅ Active\",\n", + " \"interval\": \"15 minutes\",\n", + " \"log\": \"/var/log/scan_attempts.log\"\n", + " },\n", + " \"Incident Response\": {\n", + " \"last_incident\": \"2025-12-06\",\n", + " \"recovery_time\": \"33 days\",\n", + " \"status\": \"✅ Resolved\",\n", + " \"prevention\": \"Firewall + Monitoring\"\n", + " }\n", + "}\n", + "\n", + "import pandas as pd\n", + "pd.DataFrame(security_config).T\n" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -499,7 +561,7 @@ "\n", "---\n", "\n", - "**Last Updated:** 2025-11-23 by Auto AI \n", + "**Last Updated:** 2026-01-08 (Security incident resolution & firewall implementation) \n", "**Maintained by:** Ivan Tytar & DAARION Team" ] }