## New Security Documentation Structure
/security/
├── README.md # Security overview & contacts
├── forensics-checklist.md # Incident investigation guide
├── persistence-scan.sh # Quick persistence detector
├── runtime-detector.sh # Mining/suspicious process detector
└── hardening/
├── docker.md # Docker security baseline
├── kubernetes.md # K8s policies (future reference)
└── cloud.md # Hetzner-specific hardening
## Key Components
### Forensics Checklist
- Process analysis commands
- Persistence mechanism detection
- Network connection analysis
- File system inspection
- Authentication audit
- Decision matrix for threat response
### Scripts
- persistence-scan.sh: Cron, systemd, executables, SSH keys
- runtime-detector.sh: Mining process detection with --kill option
### Hardening Guides
- Docker: Secure compose template, Dockerfile best practices
- Kubernetes: NetworkPolicy, PodSecurityStandard, Falco rules
- Cloud: Egress firewall, SSH hardening, fail2ban, monitoring
## Post-Incident Documentation
Based on lessons learned from Incidents #1 and #2 (Jan 2026)
Co-authored-by: Cursor Agent <agent@cursor.sh>
5.9 KiB
5.9 KiB
🔍 Forensics Checklist — Incident Investigation
Мета: Відповісти на 3 критичні питання:
- Як саме зайшли (initial access vector)
- Чи є persistence (чи повернеться знову)
- Чи можна довіряти системі далі (чи потрібен rebuild)
📋 Швидкий чекліст
A. Process-level Analysis
# Всі процеси з деревом
ps auxf
# Top CPU consumers
ps -eo pid,ppid,user,cmd,%cpu,%mem --sort=-%cpu | head -20
# Процеси конкретного користувача (напр. container user 1001)
ps aux | grep "1001"
# Zombie процеси
ps aux | grep defunct | wc -l
🔴 Red flags:
- Дивні назви:
softirq,.syslog,catcal,G4NQXBp,vrarhpb - Процеси без батьків (orphans)
- user ≠ expected
- CPU > 50% на невідомому процесі
B. Persistence Mechanisms
# Cron jobs
crontab -l
cat /etc/crontab
ls -la /etc/cron.d/
ls -la /etc/cron.daily/
ls -la /etc/cron.hourly/
# Systemd services
systemctl list-unit-files --state=enabled
ls -la /etc/systemd/system/
ls -la /usr/lib/systemd/system/
# Init scripts
ls -la /etc/init.d/
ls -la /etc/rc.local
# Docker auto-restart
docker ps --filter "restart=always"
docker ps --filter "restart=unless-stopped"
🔴 Red flags:
- Незнайомі cron jobs
- Нові systemd services
- Контейнери з
restart: unless-stopped+ compromised
C. Network Analysis
# Listening ports
ss -tulpn
netstat -tulpn
# Active connections
ss -antp
netstat -antp
# Firewall rules
iptables -L -n -v
iptables -L -n -v -t nat
# DNS queries (if available)
cat /var/log/syslog | grep -i dns
🔴 Red flags:
- Outbound до mining pools (порти 3333, 5555, 7777, 14433)
- Нові listening ports
- З'єднання до unknown IP
Known mining pool patterns:
*pool*
*xmr*
*monero*
*crypto*
*.ru:*
*.cn:*
D. File System Analysis
# Executable files in temp directories
find /tmp /var/tmp /dev/shm -type f -executable 2>/dev/null
# Recently modified binaries
find /usr/bin /usr/local/bin /usr/sbin -mtime -3 2>/dev/null
# Hidden files in home directories
find /root /home -name ".*" -type f 2>/dev/null
# Large files in unexpected places
find /tmp /var/tmp -size +10M 2>/dev/null
# SUID/SGID binaries
find / -perm -4000 -type f 2>/dev/null
find / -perm -2000 -type f 2>/dev/null
🔴 Red flags:
- Executables в /tmp, /dev/shm
- Нещодавно змінені системні бінарники
- Hidden files з executable permissions
E. Authentication & Access
# Login history
last
lastlog
who
# SSH keys
grep -R "ssh-rsa" /root/.ssh /home 2>/dev/null
cat /root/.ssh/authorized_keys
ls -la /root/.ssh/
# Failed logins
grep "Failed" /var/log/auth.log | tail -50
grep "Accepted" /var/log/auth.log | tail -50
# Sudo usage
grep "sudo" /var/log/auth.log | tail -50
🔴 Red flags:
- Незнайомі SSH ключі
- Логіни з unknown IP
- Нові користувачі
F. Docker-specific
# All containers (including stopped)
docker ps -a
# Container processes
docker top <container_name>
# Container logs
docker logs --tail 100 <container_name>
# Docker images
docker images
# Docker networks
docker network ls
docker network inspect <network>
# Container inspect (look for mounts, env vars)
docker inspect <container_name>
🔴 Red flags:
- Контейнери з
--privileged - Mounted host directories (особливо /)
- Unknown images
📊 Decision Matrix
| Знахідка | Рівень загрози | Дія |
|---|---|---|
| Підозрілий процес, CPU > 50% | 🔴 Critical | Kill + investigate |
| Cron job до unknown binary | 🔴 Critical | Remove + investigate |
| New SSH key | 🔴 Critical | Remove + rotate all |
| Outbound to mining pool | 🔴 Critical | Block + kill |
| Modified system binary | 🔴 Critical | Full rebuild |
| Container with persistence | 🟡 High | Remove container + image |
| Unknown listening port | 🟡 High | Investigate + block |
| Failed SSH attempts | 🟢 Low | Monitor + fail2ban |
🔧 Post-Investigation Actions
If compromised (any 🔴 finding):
-
Contain:
# Stop affected services docker stop <container> # Block outbound (emergency) iptables -I OUTPUT -d 0.0.0.0/0 -p tcp --dport 22 -j DROP -
Preserve evidence:
# Save process list ps auxf > /root/evidence/ps_$(date +%Y%m%d_%H%M%S).txt # Save network connections ss -antp > /root/evidence/ss_$(date +%Y%m%d_%H%M%S).txt # Save Docker state docker ps -a > /root/evidence/docker_$(date +%Y%m%d_%H%M%S).txt -
Eradicate:
# Kill processes kill -9 <pid> # Remove persistence crontab -r systemctl disable <service> # Remove Docker artifacts docker stop <container> docker rm <container> docker rmi <image> # CRITICAL! -
Recover:
- Rebuild from clean source
- Apply hardening
- Monitor for recurrence
-
Document:
- Update INFRASTRUCTURE.md
- Create incident report
- Update hardening procedures
📝 Incident Report Template
## Incident Report: [Title]
**Date:** YYYY-MM-DD HH:MM UTC
**Severity:** Critical/High/Medium/Low
**Status:** Resolved/Ongoing
### Timeline
- HH:MM — Detection
- HH:MM — Containment
- HH:MM — Eradication
- HH:MM — Recovery
### Root Cause
[Description of how the attack occurred]
### Impact
- Services affected
- Data affected
- Downtime
### Indicators of Compromise (IOCs)
- Process names
- File paths
- IP addresses
- Domains
### Remediation
- Actions taken
- Hardening applied
### Lessons Learned
- What worked
- What to improve
- Prevention measures