docs: security incident resolution & firewall implementation

- Document network scanning incident (Dec 6 2025 - Jan 8 2026)
- Add firewall rules to prevent internal network access
- Deploy monitoring script for scanning attempts
- Update INFRASTRUCTURE.md v2.2.0 with Security section
- Update infrastructure_quick_ref.ipynb v2.1.0
- Root cause: compromised daarion-web container with crypto miner
- Resolution: container removed, firewall applied, monitoring deployed

Co-Authored-By: Warp <agent@warp.dev>
This commit is contained in:
Apple
2026-01-08 11:52:53 -08:00
parent e3a8b7464a
commit e829fe66f2
2 changed files with 153 additions and 6 deletions

View File

@@ -1,9 +1,13 @@
# 🏗️ Infrastructure Overview — DAARION & MicroDAO
**Версія:** 2.1.0
**Останнє оновлення:** 2025-11-23 18:05
**Версія:** 2.2.0
**Останнє оновлення:** 2026-01-08 19:30
**Статус:** Production Ready (95% Multimodal Integration)
**Останні зміни:**
- 🔒 **Security Incident Resolution** (Dec 6 2025 - Jan 8 2026)
- ✅ Compromised container removed (`daarion-web`)
- ✅ Firewall rules implemented (egress filtering)
- ✅ Monitoring for scanning attempts deployed
- ✅ Router Multimodal API (v1.1.0) - images/files/audio/web-search
- ✅ Telegram Gateway Multimodal - voice/photo/documents
- ✅ Frontend Multimodal UI - enhanced mode
@@ -1022,3 +1026,84 @@ User → @YaromirBot (Telegram)
**Status:** 95% Complete
**Production Ready:** ✅ Yes (with fallbacks)
---
## 🔒 Security & Incident Response
### Incident #1: Network Scanning & Server Lockdown (Dec 6, 2025 - Jan 8, 2026)
**Timeline:**
- **Dec 6, 2025 10:56 UTC**: Automated SSH scanning detected from server
- **Dec 6, 2025 11:00 UTC**: Hetzner locked server IP (144.76.224.179)
- **Jan 8, 2026 18:00 UTC**: Unlock request approved, server recovered
**Root Cause:**
- Server compromised with cryptocurrency miner (`catcal`, `G4NQXBp`) via `daarion-web` container
- Miner performed network scanning of Hetzner internal network (10.126.0.0/16)
- ~500+ SSH connection attempts to internal IP range triggered automated block
- High CPU load (35+) from mining process
**Impact:**
- ❌ Server unavailable for 33 days
- ❌ All services down
- ❌ Telegram bots offline
- ❌ Lost production data/monitoring
**Resolution:**
1. ✅ Server recovered via rescue mode
2. ✅ Compromised `daarion-web` container stopped and removed
3. ✅ Cryptocurrency miner processes killed
4. ✅ Firewall rules implemented to block internal network access
5. ✅ Monitoring script deployed for future scanning attempts
**Prevention Measures:**
**Firewall Rules:**
```bash
# Block Hetzner internal networks
iptables -I OUTPUT -d 10.0.0.0/8 -j DROP
iptables -I OUTPUT -d 172.16.0.0/12 -j DROP
# Allow only necessary ports
iptables -I OUTPUT -d 10.0.0.0/8 -p tcp --dport 443 -j ACCEPT
iptables -I OUTPUT -d 10.0.0.0/8 -p tcp --dport 80 -j ACCEPT
# Log blocked attempts
iptables -I OUTPUT -d 10.0.0.0/8 -j LOG --log-prefix "BLOCKED_INTERNAL_SCAN: "
# Save rules
iptables-save > /etc/iptables/rules.v4
```
**Monitoring:**
- Script: `/root/monitor_scanning.sh`
- Runs every 15 minutes via cron
- Logs to `/var/log/scan_attempts.log`
- Checks for:
- Suspicious network activity in Docker logs
- iptables blocked connection attempts
- Keywords: `10.126`, `172.16`, `scan`, `probe`
**Security Checklist:**
- [ ] Review all Docker images for vulnerabilities
- [ ] Implement container security scanning (Trivy/Clair)
- [ ] Enable Docker Content Trust
- [ ] Set up intrusion detection (fail2ban)
- [ ] Regular security audits
- [ ] Container resource limits (CPU/memory)
- [ ] Network segmentation for containers
**References:**
- Hetzner Incident ID: `L00280548`
- Guideline: https://docs.hetzner.com/robot/dedicated-server/troubleshooting/guideline-in-case-of-server-locking/
- Recovery Scripts: `/root/prevent_scanning.sh`, `/root/monitor_scanning.sh`
**Lessons Learned:**
1. 🔴 **Never expose containers without security scanning**
2. 🟡 **Implement egress firewall rules from day 1**
3. 🟢 **Monitor outgoing connections, not just incoming**
4. 🔵 **Have disaster recovery plan documented**
5. 🟣 **Regular security audits are critical**
---