📝 Update context docs with session logging system
Some checks failed
Build and Deploy Docs / build-and-deploy (push) Has been cancelled
Some checks failed
Build and Deploy Docs / build-and-deploy (push) Has been cancelled
- Added Session Logging System section to INFRASTRUCTURE.md - Added Git Multi-Remote configuration (GitHub + Gitea + GitLab) - Updated version to 2.5.0 - Added logging commands reference - Updated infrastructure_quick_ref.ipynb with new features - Added SSH tunnel instructions for GitLab access
This commit is contained in:
@@ -1,11 +1,15 @@
|
|||||||
# 🏗️ Infrastructure Overview — DAARION & MicroDAO
|
# 🏗️ Infrastructure Overview — DAARION & MicroDAO
|
||||||
|
|
||||||
**Версія:** 2.4.0
|
**Версія:** 2.5.0
|
||||||
**Останнє оновлення:** 2026-01-09 13:50
|
**Останнє оновлення:** 2026-01-10 14:55
|
||||||
**Статус:** Production Ready (95% Multimodal Integration)
|
**Статус:** Production Ready (95% Multimodal Integration)
|
||||||
**Останні зміни:**
|
**Останні зміни:**
|
||||||
- 🔒 **Security Incident Resolution** (Dec 6 2025 - Jan 8 2026)
|
- 📝 **Session Logging System** (Jan 10, 2026) — автоматичне логування всіх дій
|
||||||
- ✅ Compromised container removed (`daarion-web`)
|
- 🔄 **Git Multi-Remote** — GitHub + Gitea + GitLab синхронізація
|
||||||
|
- 🏗️ **NODE1 Rebuild** — чиста Ubuntu 24.04 + Docker
|
||||||
|
- 🐳 **GitLab on NODE3** — додаткове дзеркало репозиторіїв
|
||||||
|
- 🔒 **Security Incident Resolution** (Dec 6 2025 - Jan 10 2026)
|
||||||
|
- ✅ NODE1 повністю перевстановлено (host compromise)
|
||||||
- ✅ Firewall rules implemented (egress filtering)
|
- ✅ Firewall rules implemented (egress filtering)
|
||||||
- ✅ Monitoring for scanning attempts deployed
|
- ✅ Monitoring for scanning attempts deployed
|
||||||
- ✅ Router Multimodal API (v1.1.0) - images/files/audio/web-search
|
- ✅ Router Multimodal API (v1.1.0) - images/files/audio/web-search
|
||||||
@@ -95,7 +99,44 @@
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🐙 GitHub Repositories
|
## 🐙 Git Repositories (Multi-Remote)
|
||||||
|
|
||||||
|
### Налаштовані Remote (3 дзеркала)
|
||||||
|
|
||||||
|
| Remote | URL | Призначення |
|
||||||
|
|--------|-----|-------------|
|
||||||
|
| **origin** | `git@github.com:IvanTytar/microdao-daarion.git` | GitHub (основний) |
|
||||||
|
| **gitea** | `http://localhost:3000/daarion-admin/microdao-daarion.git` | Gitea (локальний) |
|
||||||
|
| **gitlab** | `http://localhost:8929/root/microdao-daarion.git` | GitLab (NODE3, через tunnel) |
|
||||||
|
|
||||||
|
### Push на всі репозиторії
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Скрипт синхронізації
|
||||||
|
./scripts/git-sync-all.sh
|
||||||
|
|
||||||
|
# Або вручну
|
||||||
|
git push origin main
|
||||||
|
git push gitea main
|
||||||
|
git push gitlab main # потрібен SSH tunnel
|
||||||
|
```
|
||||||
|
|
||||||
|
### SSH Tunnel до GitLab (NODE3)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Запустити tunnel (якщо не активний)
|
||||||
|
ssh -p 33147 -L 8929:localhost:8929 -N zevs@80.77.35.151 &
|
||||||
|
|
||||||
|
# Перевірити
|
||||||
|
nc -z localhost 8929 && echo "Tunnel OK"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Credentials
|
||||||
|
|
||||||
|
| Сервіс | Логін | Пароль/Токен |
|
||||||
|
|--------|-------|--------------|
|
||||||
|
| **Gitea** | `daarion-admin` | `DaarionGit2026!` |
|
||||||
|
| **GitLab** | `root` | `glpat-daarion-gitlab-2026` |
|
||||||
|
|
||||||
### 1. MicroDAO (Current Project)
|
### 1. MicroDAO (Current Project)
|
||||||
- **Repository:** `git@github.com:IvanTytar/microdao-daarion`
|
- **Repository:** `git@github.com:IvanTytar/microdao-daarion`
|
||||||
@@ -661,6 +702,8 @@ dig gateway.daarion.city +short
|
|||||||
- **Crawl4AI Service:** `CRAWL4AI-STATUS.md`
|
- **Crawl4AI Service:** `CRAWL4AI-STATUS.md`
|
||||||
- **Architecture:** `docs/cursor/README.md`
|
- **Architecture:** `docs/cursor/README.md`
|
||||||
- **API Reference:** `docs/api.md`
|
- **API Reference:** `docs/api.md`
|
||||||
|
- **Session Logs:** `logs/sessions/` — щоденні логи сесій
|
||||||
|
- **Changelog:** `logs/CHANGELOG.md` — журнал змін
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -1227,6 +1270,91 @@ User → @YaromirBot (Telegram)
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## 📝 Session Logging System (Автоматичне логування)
|
||||||
|
|
||||||
|
### Огляд
|
||||||
|
|
||||||
|
Система автоматичного логування всіх дій при роботі над проєктом.
|
||||||
|
|
||||||
|
### Структура логів
|
||||||
|
|
||||||
|
```
|
||||||
|
logs/
|
||||||
|
├── README.md # Документація системи логування
|
||||||
|
├── CHANGELOG.md # Головний журнал змін
|
||||||
|
├── sessions/ # Щоденні логи сесій
|
||||||
|
│ └── YYYY-MM-DD.md # Лог конкретного дня
|
||||||
|
├── operations/ # Операційні логи (деплої, міграції)
|
||||||
|
├── incidents/ # Логи інцидентів безпеки
|
||||||
|
└── daily/ # Автоматичні щоденні звіти
|
||||||
|
```
|
||||||
|
|
||||||
|
### Автоматичне логування
|
||||||
|
|
||||||
|
**Git Hooks (встановлені):**
|
||||||
|
- `post-commit` — автоматично логує кожен commit
|
||||||
|
- `pre-push` — автоматично логує кожен push
|
||||||
|
|
||||||
|
**Shell Integration (опціонально):**
|
||||||
|
```bash
|
||||||
|
# Додайте до ~/.zshrc:
|
||||||
|
source /Users/apple/github-projects/microdao-daarion/scripts/logging/shell-integration.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### Команди
|
||||||
|
|
||||||
|
| Команда | Опис |
|
||||||
|
|---------|------|
|
||||||
|
| `session-start "опис"` | Почати нову сесію |
|
||||||
|
| `session-log "дія"` | Додати запис до сесії |
|
||||||
|
| `session-end` | Завершити сесію (commit + push на всі remote) |
|
||||||
|
| `daarion-note "нотатка"` | Швидка нотатка |
|
||||||
|
| `git-sync` | Push на всі репозиторії (GitHub + Gitea + GitLab) |
|
||||||
|
|
||||||
|
### Що логується автоматично
|
||||||
|
|
||||||
|
✅ **Автоматично (Git hooks):**
|
||||||
|
- Кожен commit (хеш, повідомлення, кількість файлів)
|
||||||
|
- Кожен push (remote name)
|
||||||
|
- Час кожної дії
|
||||||
|
|
||||||
|
✅ **Вручну (через команди):**
|
||||||
|
- Початок/кінець сесії
|
||||||
|
- Важливі дії та рішення
|
||||||
|
- Нотатки та TODO
|
||||||
|
|
||||||
|
### Приклад сесії
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
# 📅 Session Log: 2026-01-10
|
||||||
|
|
||||||
|
## 📋 Хронологія дій
|
||||||
|
|
||||||
|
- **10:00** — 📦 Commit `a1b2c3d`: Fix authentication bug (3 files)
|
||||||
|
- **10:15** — 🚀 Push to `origin`
|
||||||
|
- **10:30** — 📝 Deployed new version to NODE1
|
||||||
|
```
|
||||||
|
|
||||||
|
### Встановлення
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Встановити Git hooks
|
||||||
|
./scripts/logging/install-hooks.sh
|
||||||
|
|
||||||
|
# 2. Додати shell integration (опціонально)
|
||||||
|
echo 'source /Users/apple/github-projects/microdao-daarion/scripts/logging/shell-integration.sh' >> ~/.zshrc
|
||||||
|
source ~/.zshrc
|
||||||
|
```
|
||||||
|
|
||||||
|
### Синхронізація логів
|
||||||
|
|
||||||
|
Логи автоматично синхронізуються на всі 3 репозиторії при:
|
||||||
|
- `session-end` — завершення сесії
|
||||||
|
- `git-sync` — ручна синхронізація
|
||||||
|
- Звичайний `git push` (якщо логи в коміті)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## 🔒 Security & Incident Response
|
## 🔒 Security & Incident Response
|
||||||
|
|
||||||
### Incident #1: Network Scanning & Server Lockdown (Dec 6, 2025 - Jan 8, 2026)
|
### Incident #1: Network Scanning & Server Lockdown (Dec 6, 2025 - Jan 8, 2026)
|
||||||
@@ -1607,17 +1735,16 @@ ps aux | awk '$3 > 50'
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
### Incident #4: ALL PostgreSQL Images Compromised — NODE1 Host Suspected (Jan 10, 2026)
|
||||||
### Incident #4: ALL PostgreSQL Images Show Malware — NODE1 Host Compromise Suspected (Jan 10, 2026)
|
|
||||||
|
|
||||||
**Timeline:**
|
**Timeline:**
|
||||||
- **Jan 10, 2026**: Testing postgres:16-alpine — malware artifacts found
|
- **Jan 10, 2026 ~XX:XX UTC**: Testing postgres:16-alpine — COMPROMISED
|
||||||
- **Jan 10, 2026**: Testing postgres:14 (non-alpine) — malware artifacts found
|
- **Jan 10, 2026 ~XX:XX UTC**: Testing postgres:14 (non-alpine) — COMPROMISED
|
||||||
- **Jan 10, 2026**: Testing postgres:16 (Debian) — malware artifacts found
|
- **Jan 10, 2026 ~XX:XX UTC**: Testing postgres:16 (Debian) — COMPROMISED
|
||||||
|
|
||||||
**Confirmed "Compromised" Images (on NODE1):**
|
**Confirmed Compromised Images:**
|
||||||
```bash
|
```bash
|
||||||
# ALL of these show malware artifacts when run on NODE1:
|
# ALL of these show malware artifacts on NODE1:
|
||||||
❌ postgres:15-alpine # Incident #3
|
❌ postgres:15-alpine # Incident #3
|
||||||
❌ postgres:16-alpine # NEW
|
❌ postgres:16-alpine # NEW
|
||||||
❌ postgres:14 # NEW (non-alpine!)
|
❌ postgres:14 # NEW (non-alpine!)
|
||||||
@@ -1632,174 +1759,125 @@ ps aux | awk '$3 > 50'
|
|||||||
|
|
||||||
**🔴 CRITICAL ASSESSMENT:**
|
**🔴 CRITICAL ASSESSMENT:**
|
||||||
|
|
||||||
**This is NOT "all Docker Hub official images are infected".**
|
This is **NOT "all Docker Hub official images are infected"**.
|
||||||
|
|
||||||
**This is most likely NODE1 HOST COMPROMISE** (perfctl/cryptominer persistence).
|
This is most likely **NODE1 HOST COMPROMISE** (perfctl/cryptominer persistence).
|
||||||
|
|
||||||
**Evidence supporting HOST compromise (not image compromise):**
|
**Evidence supporting HOST compromise (not image compromise):**
|
||||||
|
1. `/tmp/.perf.c/` — Classic perfctl malware directory
|
||||||
| Evidence | Explanation |
|
2. `/tmp/httpd` 10MB — Typical xmrig miner size with Apache masquerade
|
||||||
|----------|-------------|
|
3. ALL postgres variants affected — Statistically impossible for Docker Hub
|
||||||
| `/tmp/.perf.c/` directory | Classic perfctl malware staging directory |
|
4. NODE1 had 3 previous incidents (#1, #2, #3) — Already compromised
|
||||||
| `/tmp/httpd` ~10MB | Typical xmrig miner with Apache masquerade |
|
5. `tmpfs noexec` didn't help — Malware runs from HOST, not container
|
||||||
| ALL postgres variants affected | Statistically impossible for Docker Hub |
|
|
||||||
| NODE1 had 3 previous incidents | Already compromised (Incidents #1, #2, #3) |
|
|
||||||
| `tmpfs noexec` didn't help | Malware runs from HOST, not container |
|
|
||||||
| Same IOCs across different images | Infection happens post-pull, not in image |
|
|
||||||
|
|
||||||
**Probable Attack Vector (perfctl family):**
|
**Probable Attack Vector (perfctl family):**
|
||||||
- Initial compromise via Incident #1 or #2 (daarion-web container)
|
- Initial compromise via Incident #1 or #2 (daarion-web)
|
||||||
- Persistence mechanism survived container/image cleanup
|
- Persistence mechanism survived container cleanup
|
||||||
- Malware hooks into Docker daemon or uses cron/systemd
|
- Malware infects ANY new container on startup
|
||||||
- Infects ANY new container on startup via:
|
- Uses techniques: cron, systemd, kernel modules, LD_PRELOAD
|
||||||
- Modified docker daemon
|
|
||||||
- LD_PRELOAD injection
|
|
||||||
- Kernel module
|
|
||||||
- Cron job that monitors new containers
|
|
||||||
|
|
||||||
**🔬 VERIFICATION PROCEDURE (REQUIRED):**
|
|
||||||
|
|
||||||
|
**Verification Procedure (REQUIRED):**
|
||||||
```bash
|
```bash
|
||||||
# ═══════════════════════════════════════════════════════════════
|
# Step 1: Get image digest from NODE1
|
||||||
# STEP 1: Get image digest from NODE1
|
docker inspect --format='{{index .RepoDigests 0}}' postgres:16
|
||||||
# ═══════════════════════════════════════════════════════════════
|
|
||||||
ssh root@144.76.224.179 "docker inspect --format='{{index .RepoDigests 0}}' postgres:16"
|
|
||||||
# Example output: postgres@sha256:abc123...
|
|
||||||
|
|
||||||
# ═══════════════════════════════════════════════════════════════
|
# Step 2: On CLEAN host (NOT NODE1!), pull same digest
|
||||||
# STEP 2: On CLEAN host (MacBook/NODE2), pull SAME digest
|
|
||||||
# ═══════════════════════════════════════════════════════════════
|
|
||||||
# On your MacBook (NOT NODE1!):
|
|
||||||
docker pull postgres:16@sha256:<digest_from_step1>
|
docker pull postgres:16@sha256:<digest_from_step1>
|
||||||
|
|
||||||
# ═══════════════════════════════════════════════════════════════
|
# Step 3: Run on clean host
|
||||||
# STEP 3: Run on clean host and check /tmp
|
docker run --rm -it postgres:16@sha256:<digest> ls -la /tmp/
|
||||||
# ═══════════════════════════════════════════════════════════════
|
# If /tmp is empty → IMAGE IS CLEAN → NODE1 IS COMPROMISED
|
||||||
docker run --rm -it postgres:16@sha256:<digest> sh -c "ls -la /tmp/ && find /tmp -type f"
|
|
||||||
|
|
||||||
# EXPECTED RESULTS:
|
# Step 4: Check NODE1 host for persistence
|
||||||
# - If /tmp is EMPTY on clean host → IMAGE IS CLEAN → NODE1 IS COMPROMISED
|
|
||||||
# - If /tmp has httpd/.perf.c on clean host → IMAGE IS COMPROMISED → Report to Docker
|
|
||||||
|
|
||||||
# ═══════════════════════════════════════════════════════════════
|
|
||||||
# STEP 4: Check NODE1 host for persistence mechanisms
|
|
||||||
# ═══════════════════════════════════════════════════════════════
|
|
||||||
ssh root@144.76.224.179 << 'REMOTE_CHECK'
|
|
||||||
echo "=== CRON ==="
|
|
||||||
crontab -l 2>/dev/null
|
|
||||||
cat /etc/crontab
|
cat /etc/crontab
|
||||||
ls -la /etc/cron.d/
|
ls -la /etc/cron.d/
|
||||||
|
systemctl list-units --type=service | grep -i "perf\|miner\|http"
|
||||||
echo "=== SYSTEMD ==="
|
cat /etc/ld.so.preload
|
||||||
systemctl list-units --type=service | grep -iE "perf|miner|http|crypto"
|
lsmod | grep -v "^Module"
|
||||||
|
|
||||||
echo "=== LD_PRELOAD ==="
|
|
||||||
cat /etc/ld.so.preload 2>/dev/null
|
|
||||||
echo $LD_PRELOAD
|
|
||||||
|
|
||||||
echo "=== KERNEL MODULES ==="
|
|
||||||
lsmod | head -20
|
|
||||||
|
|
||||||
echo "=== SUSPICIOUS PROCESSES ==="
|
|
||||||
ps aux | grep -E "(httpd|xmrig|kdevtmp|kinsing|perfctl|\.perf)" | grep -v grep
|
|
||||||
|
|
||||||
echo "=== NETWORK TO MINING POOLS ==="
|
|
||||||
ss -anp | grep -E "(3333|4444|5555|8080|8888)" | head -10
|
|
||||||
|
|
||||||
echo "=== SSH AUTHORIZED KEYS ==="
|
|
||||||
cat /root/.ssh/authorized_keys
|
|
||||||
|
|
||||||
echo "=== DOCKER DAEMON CONFIG ==="
|
|
||||||
cat /etc/docker/daemon.json 2>/dev/null
|
|
||||||
REMOTE_CHECK
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**🔴 DECISION MATRIX:**
|
**NODE1 Compromise Indicators to Check:**
|
||||||
|
|
||||||
| Verification Result | Conclusion | Action |
|
|
||||||
|---------------------|------------|--------|
|
|
||||||
| Clean host: no malware | **NODE1 COMPROMISED** | Full rebuild of NODE1 |
|
|
||||||
| Clean host: same malware | **Docker Hub compromised** | Report to Docker Security |
|
|
||||||
|
|
||||||
**If NODE1 Confirmed Compromised (most likely):**
|
|
||||||
|
|
||||||
1. 🔴 **STOP using NODE1 immediately** for any workloads
|
|
||||||
2. 🔴 **Rotate ALL secrets** that NODE1 ever accessed:
|
|
||||||
```
|
|
||||||
- SSH keys (generate new on clean machine)
|
|
||||||
- Telegram bot tokens (regenerate via @BotFather)
|
|
||||||
- PostgreSQL passwords
|
|
||||||
- All API keys in .env
|
|
||||||
- JWT secrets
|
|
||||||
- Neo4j credentials
|
|
||||||
- Redis password (if any)
|
|
||||||
```
|
|
||||||
3. 🔴 **Full OS reinstall** (not cleanup!):
|
|
||||||
- Request fresh install from Hetzner Robot
|
|
||||||
- Or use rescue mode + full disk wipe
|
|
||||||
- New SSH keys generated on clean machine
|
|
||||||
4. 🟡 **Verify images on clean host BEFORE deploying to new NODE1**
|
|
||||||
5. 🟢 **Implement proper security controls** (see Prevention below)
|
|
||||||
|
|
||||||
**Alternative PostgreSQL Sources (if Docker Hub suspected):**
|
|
||||||
```bash
|
```bash
|
||||||
# GitHub Container Registry (GHCR)
|
# Processes
|
||||||
docker pull ghcr.io/docker-library/postgres:16-alpine
|
ps aux | grep -E "(httpd|xmrig|kdevtmp|kinsing|perfctl|\.perf)"
|
||||||
|
|
||||||
# Quay.io (Red Hat operated)
|
# Network connections to mining pools
|
||||||
|
ss -anp | grep -E "(3333|4444|5555|8080|8888)"
|
||||||
|
|
||||||
|
# Suspicious files
|
||||||
|
find /tmp -type f -executable 2>/dev/null
|
||||||
|
find /var/tmp -type f -executable 2>/dev/null
|
||||||
|
find /dev/shm -type f -executable 2>/dev/null
|
||||||
|
|
||||||
|
# Cron persistence
|
||||||
|
crontab -l
|
||||||
|
cat /etc/crontab
|
||||||
|
ls -la /etc/cron.*
|
||||||
|
|
||||||
|
# Systemd persistence
|
||||||
|
systemctl list-units --type=service --all | grep -v "loaded active"
|
||||||
|
|
||||||
|
# SSH keys (attacker backdoor)
|
||||||
|
cat /root/.ssh/authorized_keys
|
||||||
|
cat /home/*/.ssh/authorized_keys
|
||||||
|
|
||||||
|
# LD_PRELOAD rootkit
|
||||||
|
cat /etc/ld.so.preload
|
||||||
|
ldd /bin/ls | grep -v "linux-vdso\|ld-linux"
|
||||||
|
|
||||||
|
# Kernel modules
|
||||||
|
lsmod
|
||||||
|
cat /proc/modules | grep -v "^Module"
|
||||||
|
```
|
||||||
|
|
||||||
|
**🔴 DECISION: NODE1 STATUS**
|
||||||
|
|
||||||
|
| If verification shows... | Then... |
|
||||||
|
|--------------------------|---------|
|
||||||
|
| Clean host pulls same digest → no malware | **NODE1 IS COMPROMISED** → Full rebuild required |
|
||||||
|
| Clean host also shows malware | **Docker Hub compromised** → Report to Docker Security |
|
||||||
|
|
||||||
|
**If NODE1 Confirmed Compromised:**
|
||||||
|
1. 🔴 **DO NOT use NODE1 for any workloads**
|
||||||
|
2. 🔴 **Rotate ALL secrets** that NODE1 ever accessed:
|
||||||
|
- SSH keys
|
||||||
|
- Telegram bot tokens
|
||||||
|
- Database passwords
|
||||||
|
- API keys
|
||||||
|
- JWT secrets
|
||||||
|
3. 🔴 **Full rebuild from scratch** (not cleanup!)
|
||||||
|
- Fresh OS install
|
||||||
|
- New SSH keys
|
||||||
|
- Pull images on clean host first, verify, then transfer
|
||||||
|
4. 🟡 **Forensics** (optional but recommended):
|
||||||
|
- Image the disk before rebuild
|
||||||
|
- Analyze persistence mechanisms
|
||||||
|
- Report to Hetzner with findings
|
||||||
|
|
||||||
|
**Alternative Registries (if Docker Hub suspected):**
|
||||||
|
```bash
|
||||||
|
# GitHub Container Registry
|
||||||
|
docker pull ghcr.io/postgres/postgres:16-alpine
|
||||||
|
|
||||||
|
# Quay.io (Red Hat)
|
||||||
docker pull quay.io/fedora/postgresql-16
|
docker pull quay.io/fedora/postgresql-16
|
||||||
|
|
||||||
# Build from official Dockerfile (most secure)
|
# Build from source (most secure)
|
||||||
git clone https://github.com/docker-library/postgres.git
|
git clone https://github.com/docker-library/postgres.git
|
||||||
cd postgres/16/alpine
|
cd postgres/16/alpine
|
||||||
docker build -t postgres:16-alpine-verified .
|
docker build -t postgres:16-alpine-local .
|
||||||
# Then scan with Trivy before use
|
|
||||||
trivy image postgres:16-alpine-verified
|
|
||||||
```
|
|
||||||
|
|
||||||
**NODE1 Persistence Locations to Check:**
|
|
||||||
```bash
|
|
||||||
# File-based persistence
|
|
||||||
/etc/cron.d/*
|
|
||||||
/etc/crontab
|
|
||||||
/var/spool/cron/*
|
|
||||||
/etc/systemd/system/*.service
|
|
||||||
/etc/init.d/*
|
|
||||||
/etc/rc.local
|
|
||||||
/root/.bashrc
|
|
||||||
/root/.profile
|
|
||||||
/etc/ld.so.preload
|
|
||||||
|
|
||||||
# Memory/process persistence
|
|
||||||
/dev/shm/*
|
|
||||||
/run/*
|
|
||||||
/var/run/*
|
|
||||||
|
|
||||||
# Docker-specific
|
|
||||||
/var/lib/docker/
|
|
||||||
/etc/docker/daemon.json
|
|
||||||
~/.docker/config.json
|
|
||||||
|
|
||||||
# Kernel-level (advanced)
|
|
||||||
/lib/modules/*/
|
|
||||||
/proc/modules
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**References:**
|
**References:**
|
||||||
- perfctl malware: https://blog.exatrack.com/Perfctl-using-portainer-and-new-persistences/
|
- perfctl malware analysis: https://blog.exatrack.com/Perfctl-using-portainer-and-new-persistences/
|
||||||
- Similar reports: https://github.com/docker-library/postgres/issues/1307
|
- Docker Hub malware reports: https://github.com/docker-library/postgres/issues/1307
|
||||||
- Docker Hub attacks: https://jfrog.com/blog/attacks-on-docker-with-millions-of-malicious-repositories-spread-malware-and-phishing-scams/
|
- Similar incidents: https://jfrog.com/blog/attacks-on-docker-with-millions-of-malicious-repositories-spread-malware-and-phishing-scams/
|
||||||
|
|
||||||
**Lessons Learned (Incident #4 Specific):**
|
**Lessons Learned (Incident #4 Specific):**
|
||||||
1. 🔴 **Host compromise masquerades as image compromise** — Always verify on clean host
|
1. 🔴 **Host compromise can masquerade as image compromise**
|
||||||
2. 🟡 **Previous incidents leave persistence** — Cleanup is not enough, rebuild required
|
2. 🟡 **Previous incidents may leave persistence** — Full rebuild needed
|
||||||
3. 🟢 **perfctl family is sophisticated** — Survives container restarts, image deletions
|
3. 🟢 **Always verify on CLEAN host** before blaming upstream
|
||||||
4. 🔵 **Multiple images "infected" = host problem** — Statistical impossibility otherwise
|
4. 🔵 **perfctl family is sophisticated** — Survives container restarts
|
||||||
5. 🟣 **NODE1 is UNTRUSTED** — Do not use until full rebuild + verification
|
5. 🟣 **NODE1 should be considered UNTRUSTED** until rebuilt
|
||||||
|
|
||||||
**Current Status:**
|
|
||||||
- ⏳ **Verification pending** — Need to test same digest on clean host
|
|
||||||
- 🔴 **NODE1 unsafe** — Do not deploy PostgreSQL or any new containers
|
|
||||||
- 🟡 **Secrets rotation needed** — Assume all NODE1 secrets compromised
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|||||||
@@ -6,17 +6,44 @@
|
|||||||
"source": [
|
"source": [
|
||||||
"# 🚀 Infrastructure Quick Reference — DAARION & MicroDAO\n",
|
"# 🚀 Infrastructure Quick Reference — DAARION & MicroDAO\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Версія:** 2.4.0 \n",
|
"**Версія:** 2.5.0 \n",
|
||||||
"Останнє оновлення:** 2026-01-10 XX:XX \n",
|
"**Останнє оновлення:** 2026-01-10 14:55 \n",
|
||||||
"\n",
|
"\n",
|
||||||
"Цей notebook містить швидкий довідник по серверах, репозиторіях та endpoints для DAGI Stack.\n",
|
"Цей notebook містить швидкий довідник по серверах, репозиторіях та endpoints для DAGI Stack.\n",
|
||||||
"\n",
|
"\n",
|
||||||
|
"---\n",
|
||||||
|
"\n",
|
||||||
|
"## 🆕 What's New (v2.5.0) - Jan 10, 2026\n",
|
||||||
|
"\n",
|
||||||
|
"### 📝 Session Logging System\n",
|
||||||
|
"- ✅ **Автоматичне логування** всіх дій (Git hooks)\n",
|
||||||
|
"- ✅ **Shell integration** — команди `session-start`, `session-log`, `session-end`\n",
|
||||||
|
"- ✅ **Структура логів**: `logs/sessions/`, `logs/CHANGELOG.md`\n",
|
||||||
|
"- 📋 **Документація**: `logs/README.md`\n",
|
||||||
|
"\n",
|
||||||
|
"### 🔄 Git Multi-Remote (3 дзеркала)\n",
|
||||||
|
"- ✅ **GitHub** (origin) — основний репозиторій\n",
|
||||||
|
"- ✅ **Gitea** (localhost:3000) — локальне дзеркало\n",
|
||||||
|
"- ✅ **GitLab** (NODE3:8929) — додаткове дзеркало\n",
|
||||||
|
"- 📋 **Скрипт синхронізації**: `./scripts/git-sync-all.sh`\n",
|
||||||
|
"\n",
|
||||||
|
"### 🏗️ NODE1 Rebuild (Security)\n",
|
||||||
|
"- ✅ **Повний rebuild** — чиста Ubuntu 24.04 LTS\n",
|
||||||
|
"- ✅ **Docker 29.1.4** встановлено\n",
|
||||||
|
"- ✅ **Базове hardening** — UFW, fail2ban\n",
|
||||||
|
"- ⚠️ **Сервіси ще не задеплоєні**\n",
|
||||||
|
"\n",
|
||||||
|
"### 🐳 GitLab on NODE3\n",
|
||||||
|
"- ✅ **GitLab CE** встановлено (порт 8929)\n",
|
||||||
|
"- ✅ **Доступ через SSH tunnel**\n",
|
||||||
|
"- 📋 **Команда**: `ssh -p 33147 -L 8929:localhost:8929 zevs@80.77.35.151`\n",
|
||||||
|
"\n",
|
||||||
|
"---\n",
|
||||||
|
"\n",
|
||||||
"**🔴 CRITICAL (v2.4.0) - Jan 10, 2026:**\n",
|
"**🔴 CRITICAL (v2.4.0) - Jan 10, 2026:**\n",
|
||||||
"- 🔴 **Incident #4: NODE1 Host Compromise Suspected**\n",
|
"- 🔴 **Incident #4: NODE1 Host Compromise** — RESOLVED via full rebuild\n",
|
||||||
"- ❌ ALL PostgreSQL images show malware on NODE1 (15-alpine, 16-alpine, 14, 16)\n",
|
"- ✅ NODE1 перевстановлено з нуля\n",
|
||||||
"- ⚠️ **NODE1 UNSAFE** - Do not deploy any containers until verified\n",
|
"- ⚠️ **Secrets rotation needed** — див. `SECRETS-ROTATION-CHECKLIST.md`\n",
|
||||||
"- 📋 **Triage script added**: `scripts/security/triage-postgres-compromise.sh`\n",
|
|
||||||
"- 🔬 **Verification required**: Test same image digest on clean host\n",
|
|
||||||
"\n",
|
"\n",
|
||||||
"**v2.3.0:** \n",
|
"**v2.3.0:** \n",
|
||||||
"- 🖥️ **NODE3 added** - Threadripper PRO 5975WX + RTX 3090 24GB\n",
|
"- 🖥️ **NODE3 added** - Threadripper PRO 5975WX + RTX 3090 24GB\n",
|
||||||
@@ -698,6 +725,14 @@
|
|||||||
"source": [
|
"source": [
|
||||||
"## 📝 Notes & Updates\n",
|
"## 📝 Notes & Updates\n",
|
||||||
"\n",
|
"\n",
|
||||||
|
"### Recent Changes (2026-01-10)\n",
|
||||||
|
"- 📝 **Session Logging System** — автоматичне логування всіх дій\n",
|
||||||
|
"- 🔄 **Git Multi-Remote** — GitHub + Gitea + GitLab синхронізація\n",
|
||||||
|
"- 🏗️ **NODE1 Rebuild** — чиста Ubuntu 24.04 + Docker 29.1.4\n",
|
||||||
|
"- 🐳 **GitLab on NODE3** — додаткове дзеркало (порт 8929)\n",
|
||||||
|
"- ✅ **Git hooks** — автологування commits/pushes\n",
|
||||||
|
"- ✅ **Shell integration** — команди session-start/log/end\n",
|
||||||
|
"\n",
|
||||||
"### Recent Changes (2025-11-23)\n",
|
"### Recent Changes (2025-11-23)\n",
|
||||||
"- ✅ **Swapper Service інтеграція** в кабінети НОД (тільки в `/nodes/node-1`, `/nodes/node-2`)\n",
|
"- ✅ **Swapper Service інтеграція** в кабінети НОД (тільки в `/nodes/node-1`, `/nodes/node-2`)\n",
|
||||||
"- ✅ **Оновлення в реальному часі** (кожні 30 секунд) для Swapper Service\n",
|
"- ✅ **Оновлення в реальному часі** (кожні 30 секунд) для Swapper Service\n",
|
||||||
@@ -708,9 +743,9 @@
|
|||||||
"- ✅ **Agent Cabinet Service** (port 8898) для метрик агентів\n",
|
"- ✅ **Agent Cabinet Service** (port 8898) для метрик агентів\n",
|
||||||
"\n",
|
"\n",
|
||||||
"### Network Architecture\n",
|
"### Network Architecture\n",
|
||||||
"- **Nodes:** 2 (1 production + 1 development)\n",
|
"- **Nodes:** 3 (NODE1 production + NODE2 development + NODE3 AI/ML)\n",
|
||||||
"- **Total Services:** 19 (додано Frontend + Agent Cabinet)\n",
|
"- **Total Services:** 19 (додано Frontend + Agent Cabinet)\n",
|
||||||
"- **Swapper Service:** Тільки в кабінетах НОД, оновлення в реальному часі\n",
|
"- **Git Remotes:** 3 (GitHub + Gitea + GitLab)\n",
|
||||||
"- **MicroDAO Cabinets:** 3 (DAARION, GREENFOOD, ENERGY UNION)\n",
|
"- **MicroDAO Cabinets:** 3 (DAARION, GREENFOOD, ENERGY UNION)\n",
|
||||||
"- **Node Cabinets:** 2 (НОДА1, НОДА2)\n",
|
"- **Node Cabinets:** 2 (НОДА1, НОДА2)\n",
|
||||||
"\n",
|
"\n",
|
||||||
@@ -724,19 +759,22 @@
|
|||||||
"- **GREENFOOD:** `http://localhost:8899/microdao/greenfood` (оркестратор: GREENFOOD)\n",
|
"- **GREENFOOD:** `http://localhost:8899/microdao/greenfood` (оркестратор: GREENFOOD)\n",
|
||||||
"- **ENERGY UNION:** `http://localhost:8899/microdao/energy-union` (оркестратор: Helion)\n",
|
"- **ENERGY UNION:** `http://localhost:8899/microdao/energy-union` (оркестратор: Helion)\n",
|
||||||
"\n",
|
"\n",
|
||||||
|
"### Git Repositories\n",
|
||||||
|
"- **GitHub:** `git@github.com:IvanTytar/microdao-daarion.git` (origin)\n",
|
||||||
|
"- **Gitea:** `http://localhost:3000/daarion-admin/microdao-daarion.git`\n",
|
||||||
|
"- **GitLab:** `http://localhost:8929/root/microdao-daarion.git` (через SSH tunnel)\n",
|
||||||
|
"\n",
|
||||||
"---\n",
|
"---\n",
|
||||||
"\n",
|
"\n",
|
||||||
"**Last Updated:** 2026-01-09 (Security Incident #2 - Emergency mitigation completed) \n",
|
"**Last Updated:** 2026-01-10 14:55 (Session Logging System + NODE1 Rebuild) \n",
|
||||||
"**Maintained by:** Ivan Tytar & DAARION Team \n",
|
"**Maintained by:** Ivan Tytar & DAARION Team \n",
|
||||||
"\n",
|
"\n",
|
||||||
"---\n",
|
"---\n",
|
||||||
"\n",
|
"\n",
|
||||||
"### 🚨 CRITICAL: Active Security Incident\n",
|
"### ✅ Security Status\n",
|
||||||
"- **Incident ID:** 10F3971:2A (Hetzner AbuseID)\n",
|
"- **NODE1:** Rebuilt from scratch (Ubuntu 24.04 + Docker)\n",
|
||||||
"- **Status:** Mitigation completed, statement submission pending\n",
|
"- **NODE3:** Clean (verified 2026-01-09)\n",
|
||||||
"- **Deadline:** 2026-01-09 12:54:00 UTC (~3.5 hours remaining)\n",
|
"- **Secrets:** Rotation pending — див. `SECRETS-ROTATION-CHECKLIST.md`"
|
||||||
"- **Action Required:** User MUST submit statement at https://statement-abuse.hetzner.com/statements/?token=28b2c7e67a409659f6c823e863887\n",
|
|
||||||
"- **Task Document:** `/Users/apple/github-projects/microdao-daarion/TASK_REBUILD_DAARION_WEB.md`"
|
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -796,6 +834,71 @@
|
|||||||
"### Full Documentation\n",
|
"### Full Documentation\n",
|
||||||
"See `INFRASTRUCTURE.md` → Incident #4"
|
"See `INFRASTRUCTURE.md` → Incident #4"
|
||||||
]
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"## 📝 Session Logging System\n",
|
||||||
|
"\n",
|
||||||
|
"### Автоматичне логування всіх дій\n",
|
||||||
|
"\n",
|
||||||
|
"Система автоматично записує всі дії при роботі над проєктом.\n",
|
||||||
|
"\n",
|
||||||
|
"### Структура логів\n",
|
||||||
|
"```\n",
|
||||||
|
"logs/\n",
|
||||||
|
"├── README.md # Документація\n",
|
||||||
|
"├── CHANGELOG.md # Головний журнал змін\n",
|
||||||
|
"├── sessions/ # Щоденні логи сесій\n",
|
||||||
|
"│ └── YYYY-MM-DD.md # Лог конкретного дня\n",
|
||||||
|
"├── operations/ # Операційні логи\n",
|
||||||
|
"└── incidents/ # Логи інцидентів\n",
|
||||||
|
"```\n",
|
||||||
|
"\n",
|
||||||
|
"### Команди (після `source ~/.zshrc`)\n",
|
||||||
|
"\n",
|
||||||
|
"| Команда | Опис |\n",
|
||||||
|
"|---------|------|\n",
|
||||||
|
"| `session-start \"опис\"` | Почати сесію |\n",
|
||||||
|
"| `session-log \"дія\"` | Додати запис |\n",
|
||||||
|
"| `session-end` | Завершити (commit + push) |\n",
|
||||||
|
"| `daarion-note \"нотатка\"` | Швидка нотатка |\n",
|
||||||
|
"| `git-sync` | Push на всі remote |\n",
|
||||||
|
"\n",
|
||||||
|
"### Автоматичне логування (Git hooks)\n",
|
||||||
|
"- ✅ Кожен `git commit` → записується в session log\n",
|
||||||
|
"- ✅ Кожен `git push` → записується в session log\n",
|
||||||
|
"\n",
|
||||||
|
"### Встановлення\n",
|
||||||
|
"```bash\n",
|
||||||
|
"# 1. Встановити Git hooks\n",
|
||||||
|
"./scripts/logging/install-hooks.sh\n",
|
||||||
|
"\n",
|
||||||
|
"# 2. Додати shell integration\n",
|
||||||
|
"echo 'source /path/to/scripts/logging/shell-integration.sh' >> ~/.zshrc\n",
|
||||||
|
"source ~/.zshrc\n",
|
||||||
|
"```\n",
|
||||||
|
"\n",
|
||||||
|
"### Git Multi-Remote (3 дзеркала)\n",
|
||||||
|
"```bash\n",
|
||||||
|
"# Всі remote\n",
|
||||||
|
"git remote -v\n",
|
||||||
|
"# origin git@github.com:IvanTytar/microdao-daarion.git\n",
|
||||||
|
"# gitea http://localhost:3000/daarion-admin/microdao-daarion.git\n",
|
||||||
|
"# gitlab http://localhost:8929/root/microdao-daarion.git\n",
|
||||||
|
"\n",
|
||||||
|
"# Push на всі\n",
|
||||||
|
"./scripts/git-sync-all.sh\n",
|
||||||
|
"# або\n",
|
||||||
|
"git push origin && git push gitea && git push gitlab\n",
|
||||||
|
"```\n",
|
||||||
|
"\n",
|
||||||
|
"### SSH Tunnel до GitLab (NODE3)\n",
|
||||||
|
"```bash\n",
|
||||||
|
"ssh -p 33147 -L 8929:localhost:8929 -N zevs@80.77.35.151 &\n",
|
||||||
|
"```"
|
||||||
|
]
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"metadata": {
|
"metadata": {
|
||||||
|
|||||||
Reference in New Issue
Block a user