docs: document Security Incident #2 - recurring container compromise
Security Incident #2 Emergency Response (Jan 9, 2026): - Documented second compromise with NEW crypto miners (softirq, vrarhpb) - Root cause: Docker image auto-restarted after server reboot - Emergency mitigation completed (processes killed, container/images removed, load normalized) - Created comprehensive rebuild task document: TASK_REBUILD_DAARION_WEB.md - Updated INFRASTRUCTURE.md v2.3.0 with Incident #2 timeline and lessons learned - Updated infrastructure_quick_ref.ipynb v2.2.0 with security status Critical Changes: - daarion-web container permanently disabled until secure rebuild - Docker images DELETED (not just container stopped) - Enhanced firewall rules (SSH rate limiting, port scan blocking) - Retry test registered with Hetzner - System load normalized: 30+ → 4.19 - Zombie processes cleaned: 1499 → 5 Files Created/Updated: 1. TASK_REBUILD_DAARION_WEB.md - Detailed rebuild instructions for Cursor agent 2. INFRASTRUCTURE.md - Added Incident #2 to Security section 3. docs/infrastructure_quick_ref.ipynb - Updated security status and version Lessons Learned: - ALWAYS delete Docker images, not just containers - Auto-restart policies are dangerous for compromised containers - Complete removal = container + image + restart policy change Status: Emergency mitigation complete, statement submission pending (deadline: 2026-01-09 12:54 UTC) Hetzner Incident ID: 10F3971:2A (AbuseID) Co-Authored-By: Warp <agent@warp.dev>
This commit is contained in:
Apple
@@ -6,13 +6,18 @@
|
||||
"source": [
|
||||
"# 🚀 Infrastructure Quick Reference — DAARION & MicroDAO\n",
|
||||
"\n",
|
||||
"Версія:** 2.1.0 \n",
|
||||
"Останнє оновлення:** 2026-01-08 \n",
|
||||
"Версія:** 2.2.0 \n",
|
||||
"Останнє оновлення:** 2026-01-09 \n",
|
||||
"\n",
|
||||
"Цей notebook містить швидкий довідник по серверах, репозиторіях та endpoints для DAGI Stack.\n",
|
||||
"\n",
|
||||
"**NEW (v2.1.0):** \n",
|
||||
"- 🔒 **Security Incident Resolved** (Dec 2025 - Jan 2026)\n",
|
||||
"**NEW (v2.2.0):** \n",
|
||||
"- 🔒 **Security Incident #2** (Jan 9, 2026) - Emergency mitigation completed\n",
|
||||
"- ⚠️ **daarion-web permanently disabled** until secure rebuild\n",
|
||||
"- ✅ Enhanced firewall rules + retry test registered with Hetzner\n",
|
||||
"\n",
|
||||
"**v2.1.0:** \n",
|
||||
"- 🔒 **Security Incident #1 Resolved** (Dec 2025 - Jan 2026)\n",
|
||||
"- ✅ Firewall rules + monitoring deployed\n",
|
||||
"\n",
|
||||
"**v2.0.0:** \n",
|
||||
@@ -546,10 +551,53 @@
|
||||
"\n",
|
||||
"### Incident #1: Network Scanning & Lockdown (Dec 6, 2025 - Jan 8, 2026)\n",
|
||||
"\n",
|
||||
"**Root Cause:** Compromised `daarion-web` container with cryptocurrency miner\n",
|
||||
"**Root Cause:** Compromised `daarion-web` container with cryptocurrency miner (`catcal`, `G4NQXBp`)\n",
|
||||
"**Impact:** Server locked by Hetzner for 33 days due to internal network scanning\n",
|
||||
"**Resolution:** Container removed, firewall rules implemented, monitoring deployed\n",
|
||||
"\n",
|
||||
"### Incident #2: Recurring Compromise (Jan 9, 2026) 🔴 ACTIVE\n",
|
||||
"\n",
|
||||
"**Root Cause:** Compromised Docker image auto-restarted after server reboot \n",
|
||||
"**Malware:** NEW crypto miners (`softirq`, `vrarhpb`) - different from Incident #1 \n",
|
||||
"**Impact:** \n",
|
||||
"- ❌ Second abuse report (AbuseID: 10F3971:2A)\n",
|
||||
"- ❌ Critical CPU load: 25-35 (normal: 1-5)\n",
|
||||
"- ❌ 1499 zombie processes\n",
|
||||
"- ⚠️ Deadline: 2026-01-09 12:54 UTC (~3.5 hours remaining)\n",
|
||||
"\n",
|
||||
"**Resolution (COMPLETED):** \n",
|
||||
"1. ✅ Killed all malicious processes (softirq, vrarhpb)\n",
|
||||
"2. ✅ Stopped and removed `daarion-web` container\n",
|
||||
"3. ✅ **DELETED Docker images** (78e22c0ee972, 608e203fb5ac) - critical step\n",
|
||||
"4. ✅ Cleaned 1499 zombie processes → 5 (normal)\n",
|
||||
"5. ✅ System load normalized: 30+ → 4.19\n",
|
||||
"6. ✅ Enhanced firewall (SSH rate limiting, port scan blocking)\n",
|
||||
"7. ✅ Registered retry test with Hetzner\n",
|
||||
"8. ⏳ **PENDING:** User statement submission (URGENT)\n",
|
||||
"\n",
|
||||
"**Why Incident #2 Occurred:** \n",
|
||||
"- Incident #1 removed container but LEFT Docker image intact\n",
|
||||
"- Container had `restart: unless-stopped` in docker-compose.yml\n",
|
||||
"- Server rebooted → docker-compose auto-restarted from compromised image\n",
|
||||
"- NEW malware variant installed (different miners than Incident #1)\n",
|
||||
"\n",
|
||||
"**What is daarion-web?** \n",
|
||||
"- Next.js frontend (port 3000) - NOT critical for core functionality\n",
|
||||
"- ✅ Router, Gateway, Telegram bots, API - ALL WORKING\n",
|
||||
"- Status: DISABLED until secure rebuild completed\n",
|
||||
"\n",
|
||||
"**Lessons Learned (Critical):** \n",
|
||||
"1. 🔴 **ALWAYS delete Docker images, not just containers**\n",
|
||||
"2. 🟡 **Auto-restart policies are dangerous for compromised containers**\n",
|
||||
"3. 🟢 **Compromised images can survive container removal**\n",
|
||||
"4. 🔵 **Complete removal = container + image + restart policy change**\n",
|
||||
"\n",
|
||||
"**Next Steps:** \n",
|
||||
"1. 🔴 **URGENT:** Submit statement to Hetzner before deadline\n",
|
||||
"2. 🟡 Monitor server for 24 hours post-statement\n",
|
||||
"3. 🟢 Secure rebuild of daarion-web (see `TASK_REBUILD_DAARION_WEB.md`)\n",
|
||||
"4. 🔵 Security audit all remaining containers\n",
|
||||
"\n",
|
||||
"### Security Measures\n",
|
||||
"\n",
|
||||
"1. **Egress Firewall Rules** (блокування внутрішніх мереж Hetzner)\n",
|
||||
@@ -570,13 +618,14 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Security Configuration\n",
|
||||
"# Security Configuration (UPDATED with Incident #2)\n",
|
||||
"security_config = {\n",
|
||||
" \"Firewall Rules\": {\n",
|
||||
" \"script\": \"/root/prevent_scanning.sh\",\n",
|
||||
" \"status\": \"✅ Active\",\n",
|
||||
" \"scripts\": [\"/root/prevent_scanning.sh\", \"/root/block_ssh_scanning.sh\"],\n",
|
||||
" \"status\": \"✅ Enhanced\",\n",
|
||||
" \"blocks\": [\"10.0.0.0/8\", \"172.16.0.0/12\"],\n",
|
||||
" \"allows\": [\"80/tcp\", \"443/tcp\"]\n",
|
||||
" \"allows\": [\"80/tcp\", \"443/tcp\"],\n",
|
||||
" \"features\": [\"SSH rate limiting\", \"Port scan blocking\", \"Enhanced logging\"]\n",
|
||||
" },\n",
|
||||
" \"Monitoring\": {\n",
|
||||
" \"script\": \"/root/monitor_scanning.sh\",\n",
|
||||
@@ -584,15 +633,25 @@
|
||||
" \"interval\": \"15 minutes\",\n",
|
||||
" \"log\": \"/var/log/scan_attempts.log\"\n",
|
||||
" },\n",
|
||||
" \"Incident Response\": {\n",
|
||||
" \"last_incident\": \"2025-12-06\",\n",
|
||||
" \"Incident #1\": {\n",
|
||||
" \"date\": \"2025-12-06\",\n",
|
||||
" \"malware\": \"catcal, G4NQXBp\",\n",
|
||||
" \"recovery_time\": \"33 days\",\n",
|
||||
" \"status\": \"✅ Resolved\",\n",
|
||||
" \"prevention\": \"Firewall + Monitoring\"\n",
|
||||
" \"status\": \"✅ Resolved\"\n",
|
||||
" },\n",
|
||||
" \"Incident #2\": {\n",
|
||||
" \"date\": \"2026-01-09\",\n",
|
||||
" \"malware\": \"softirq, vrarhpb\",\n",
|
||||
" \"mitigation_time\": \"30 minutes\",\n",
|
||||
" \"status\": \"⏳ Statement Pending\",\n",
|
||||
" \"deadline\": \"2026-01-09 12:54 UTC\",\n",
|
||||
" \"actions\": [\"Container removed\", \"Images DELETED\", \"Load normalized\", \"Retry test registered\"]\n",
|
||||
" }\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"print(\"🔒 Security Configuration:\")\n",
|
||||
"print(\"=\" * 80)\n",
|
||||
"pd.DataFrame(security_config).T\n"
|
||||
]
|
||||
},
|
||||
@@ -630,8 +689,17 @@
|
||||
"\n",
|
||||
"---\n",
|
||||
"\n",
|
||||
"**Last Updated:** 2026-01-08 (Security incident resolution & firewall implementation) \n",
|
||||
"**Maintained by:** Ivan Tytar & DAARION Team"
|
||||
"**Last Updated:** 2026-01-09 (Security Incident #2 - Emergency mitigation completed) \n",
|
||||
"**Maintained by:** Ivan Tytar & DAARION Team \n",
|
||||
"\n",
|
||||
"---\n",
|
||||
"\n",
|
||||
"### 🚨 CRITICAL: Active Security Incident\n",
|
||||
"- **Incident ID:** 10F3971:2A (Hetzner AbuseID)\n",
|
||||
"- **Status:** Mitigation completed, statement submission pending\n",
|
||||
"- **Deadline:** 2026-01-09 12:54:00 UTC (~3.5 hours remaining)\n",
|
||||
"- **Action Required:** User MUST submit statement at https://statement-abuse.hetzner.com/statements/?token=28b2c7e67a409659f6c823e863887\n",
|
||||
"- **Task Document:** `/Users/apple/github-projects/microdao-daarion/TASK_REBUILD_DAARION_WEB.md`"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
Reference in New Issue
Block a user