cba2ff47f3
## New Security Documentation Structure
/security/
├── README.md # Security overview & contacts
├── forensics-checklist.md # Incident investigation guide
├── persistence-scan.sh # Quick persistence detector
├── runtime-detector.sh # Mining/suspicious process detector
└── hardening/
├── docker.md # Docker security baseline
├── kubernetes.md # K8s policies (future reference)
└── cloud.md # Hetzner-specific hardening
## Key Components
### Forensics Checklist
- Process analysis commands
- Persistence mechanism detection
- Network connection analysis
- File system inspection
- Authentication audit
- Decision matrix for threat response
### Scripts
- persistence-scan.sh: Cron, systemd, executables, SSH keys
- runtime-detector.sh: Mining process detection with --kill option
### Hardening Guides
- Docker: Secure compose template, Dockerfile best practices
- Kubernetes: NetworkPolicy, PodSecurityStandard, Falco rules
- Cloud: Egress firewall, SSH hardening, fail2ban, monitoring
## Post-Incident Documentation
Based on lessons learned from Incidents #1 and #2 (Jan 2026)
Co-authored-by: Cursor Agent <agent@cursor.sh>
294 lines
5.9 KiB
Markdown
294 lines
5.9 KiB
Markdown
# 🔍 Forensics Checklist — Incident Investigation
|
||
|
||
**Мета:** Відповісти на 3 критичні питання:
|
||
|
||
1. **Як саме зайшли** (initial access vector)
|
||
2. **Чи є persistence** (чи повернеться знову)
|
||
3. **Чи можна довіряти системі далі** (чи потрібен rebuild)
|
||
|
||
---
|
||
|
||
## 📋 Швидкий чекліст
|
||
|
||
### A. Process-level Analysis
|
||
|
||
```bash
|
||
# Всі процеси з деревом
|
||
ps auxf
|
||
|
||
# Top CPU consumers
|
||
ps -eo pid,ppid,user,cmd,%cpu,%mem --sort=-%cpu | head -20
|
||
|
||
# Процеси конкретного користувача (напр. container user 1001)
|
||
ps aux | grep "1001"
|
||
|
||
# Zombie процеси
|
||
ps aux | grep defunct | wc -l
|
||
```
|
||
|
||
**🔴 Red flags:**
|
||
- Дивні назви: `softirq`, `.syslog`, `catcal`, `G4NQXBp`, `vrarhpb`
|
||
- Процеси без батьків (orphans)
|
||
- user ≠ expected
|
||
- CPU > 50% на невідомому процесі
|
||
|
||
---
|
||
|
||
### B. Persistence Mechanisms
|
||
|
||
```bash
|
||
# Cron jobs
|
||
crontab -l
|
||
cat /etc/crontab
|
||
ls -la /etc/cron.d/
|
||
ls -la /etc/cron.daily/
|
||
ls -la /etc/cron.hourly/
|
||
|
||
# Systemd services
|
||
systemctl list-unit-files --state=enabled
|
||
ls -la /etc/systemd/system/
|
||
ls -la /usr/lib/systemd/system/
|
||
|
||
# Init scripts
|
||
ls -la /etc/init.d/
|
||
ls -la /etc/rc.local
|
||
|
||
# Docker auto-restart
|
||
docker ps --filter "restart=always"
|
||
docker ps --filter "restart=unless-stopped"
|
||
```
|
||
|
||
**🔴 Red flags:**
|
||
- Незнайомі cron jobs
|
||
- Нові systemd services
|
||
- Контейнери з `restart: unless-stopped` + compromised
|
||
|
||
---
|
||
|
||
### C. Network Analysis
|
||
|
||
```bash
|
||
# Listening ports
|
||
ss -tulpn
|
||
netstat -tulpn
|
||
|
||
# Active connections
|
||
ss -antp
|
||
netstat -antp
|
||
|
||
# Firewall rules
|
||
iptables -L -n -v
|
||
iptables -L -n -v -t nat
|
||
|
||
# DNS queries (if available)
|
||
cat /var/log/syslog | grep -i dns
|
||
```
|
||
|
||
**🔴 Red flags:**
|
||
- Outbound до mining pools (порти 3333, 5555, 7777, 14433)
|
||
- Нові listening ports
|
||
- З'єднання до unknown IP
|
||
|
||
**Known mining pool patterns:**
|
||
```
|
||
*pool*
|
||
*xmr*
|
||
*monero*
|
||
*crypto*
|
||
*.ru:*
|
||
*.cn:*
|
||
```
|
||
|
||
---
|
||
|
||
### D. File System Analysis
|
||
|
||
```bash
|
||
# Executable files in temp directories
|
||
find /tmp /var/tmp /dev/shm -type f -executable 2>/dev/null
|
||
|
||
# Recently modified binaries
|
||
find /usr/bin /usr/local/bin /usr/sbin -mtime -3 2>/dev/null
|
||
|
||
# Hidden files in home directories
|
||
find /root /home -name ".*" -type f 2>/dev/null
|
||
|
||
# Large files in unexpected places
|
||
find /tmp /var/tmp -size +10M 2>/dev/null
|
||
|
||
# SUID/SGID binaries
|
||
find / -perm -4000 -type f 2>/dev/null
|
||
find / -perm -2000 -type f 2>/dev/null
|
||
```
|
||
|
||
**🔴 Red flags:**
|
||
- Executables в /tmp, /dev/shm
|
||
- Нещодавно змінені системні бінарники
|
||
- Hidden files з executable permissions
|
||
|
||
---
|
||
|
||
### E. Authentication & Access
|
||
|
||
```bash
|
||
# Login history
|
||
last
|
||
lastlog
|
||
who
|
||
|
||
# SSH keys
|
||
grep -R "ssh-rsa" /root/.ssh /home 2>/dev/null
|
||
cat /root/.ssh/authorized_keys
|
||
ls -la /root/.ssh/
|
||
|
||
# Failed logins
|
||
grep "Failed" /var/log/auth.log | tail -50
|
||
grep "Accepted" /var/log/auth.log | tail -50
|
||
|
||
# Sudo usage
|
||
grep "sudo" /var/log/auth.log | tail -50
|
||
```
|
||
|
||
**🔴 Red flags:**
|
||
- Незнайомі SSH ключі
|
||
- Логіни з unknown IP
|
||
- Нові користувачі
|
||
|
||
---
|
||
|
||
### F. Docker-specific
|
||
|
||
```bash
|
||
# All containers (including stopped)
|
||
docker ps -a
|
||
|
||
# Container processes
|
||
docker top <container_name>
|
||
|
||
# Container logs
|
||
docker logs --tail 100 <container_name>
|
||
|
||
# Docker images
|
||
docker images
|
||
|
||
# Docker networks
|
||
docker network ls
|
||
docker network inspect <network>
|
||
|
||
# Container inspect (look for mounts, env vars)
|
||
docker inspect <container_name>
|
||
```
|
||
|
||
**🔴 Red flags:**
|
||
- Контейнери з `--privileged`
|
||
- Mounted host directories (особливо /)
|
||
- Unknown images
|
||
|
||
---
|
||
|
||
## 📊 Decision Matrix
|
||
|
||
| Знахідка | Рівень загрози | Дія |
|
||
|----------|----------------|-----|
|
||
| Підозрілий процес, CPU > 50% | 🔴 Critical | Kill + investigate |
|
||
| Cron job до unknown binary | 🔴 Critical | Remove + investigate |
|
||
| New SSH key | 🔴 Critical | Remove + rotate all |
|
||
| Outbound to mining pool | 🔴 Critical | Block + kill |
|
||
| Modified system binary | 🔴 Critical | Full rebuild |
|
||
| Container with persistence | 🟡 High | Remove container + image |
|
||
| Unknown listening port | 🟡 High | Investigate + block |
|
||
| Failed SSH attempts | 🟢 Low | Monitor + fail2ban |
|
||
|
||
---
|
||
|
||
## 🔧 Post-Investigation Actions
|
||
|
||
### If compromised (any 🔴 finding):
|
||
|
||
1. **Contain:**
|
||
```bash
|
||
# Stop affected services
|
||
docker stop <container>
|
||
|
||
# Block outbound (emergency)
|
||
iptables -I OUTPUT -d 0.0.0.0/0 -p tcp --dport 22 -j DROP
|
||
```
|
||
|
||
2. **Preserve evidence:**
|
||
```bash
|
||
# Save process list
|
||
ps auxf > /root/evidence/ps_$(date +%Y%m%d_%H%M%S).txt
|
||
|
||
# Save network connections
|
||
ss -antp > /root/evidence/ss_$(date +%Y%m%d_%H%M%S).txt
|
||
|
||
# Save Docker state
|
||
docker ps -a > /root/evidence/docker_$(date +%Y%m%d_%H%M%S).txt
|
||
```
|
||
|
||
3. **Eradicate:**
|
||
```bash
|
||
# Kill processes
|
||
kill -9 <pid>
|
||
|
||
# Remove persistence
|
||
crontab -r
|
||
systemctl disable <service>
|
||
|
||
# Remove Docker artifacts
|
||
docker stop <container>
|
||
docker rm <container>
|
||
docker rmi <image> # CRITICAL!
|
||
```
|
||
|
||
4. **Recover:**
|
||
- Rebuild from clean source
|
||
- Apply hardening
|
||
- Monitor for recurrence
|
||
|
||
5. **Document:**
|
||
- Update INFRASTRUCTURE.md
|
||
- Create incident report
|
||
- Update hardening procedures
|
||
|
||
---
|
||
|
||
## 📝 Incident Report Template
|
||
|
||
```markdown
|
||
## Incident Report: [Title]
|
||
|
||
**Date:** YYYY-MM-DD HH:MM UTC
|
||
**Severity:** Critical/High/Medium/Low
|
||
**Status:** Resolved/Ongoing
|
||
|
||
### Timeline
|
||
- HH:MM — Detection
|
||
- HH:MM — Containment
|
||
- HH:MM — Eradication
|
||
- HH:MM — Recovery
|
||
|
||
### Root Cause
|
||
[Description of how the attack occurred]
|
||
|
||
### Impact
|
||
- Services affected
|
||
- Data affected
|
||
- Downtime
|
||
|
||
### Indicators of Compromise (IOCs)
|
||
- Process names
|
||
- File paths
|
||
- IP addresses
|
||
- Domains
|
||
|
||
### Remediation
|
||
- Actions taken
|
||
- Hardening applied
|
||
|
||
### Lessons Learned
|
||
- What worked
|
||
- What to improve
|
||
- Prevention measures
|
||
```
|