daarion-admin/microdao-daarion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
8aee29d42da07c7304a97597e3eab571b2d23558
microdao-daarion/docs/infrastructure_quick_ref.ipynb
Apple 1c247ea40c
Some checks failed
Build and Deploy Docs / build-and-deploy (push) Has been cancelled
Details
📝 Update context docs with session logging system 
- Added Session Logging System section to INFRASTRUCTURE.md
- Added Git Multi-Remote configuration (GitHub + Gitea + GitLab)
- Updated version to 2.5.0
- Added logging commands reference
- Updated infrastructure_quick_ref.ipynb with new features
- Added SSH tunnel instructions for GitLab access
2026-01-10 04:58:01 -08:00

926 lines
40 KiB
Plaintext
Raw Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 🚀 Infrastructure Quick Reference — DAARION & MicroDAO\n",
"\n",
"**Версія:** 2.5.0 \n",
"**Останнє оновлення:** 2026-01-10 14:55 \n",
"\n",
"Цей notebook містить швидкий довідник по серверах, репозиторіях та endpoints для DAGI Stack.\n",
"\n",
"---\n",
"\n",
"## 🆕 What's New (v2.5.0) - Jan 10, 2026\n",
"\n",
"### 📝 Session Logging System\n",
"- ✅ **Автоматичне логування** всіх дій (Git hooks)\n",
"- ✅ **Shell integration** — команди `session-start`, `session-log`, `session-end`\n",
"- ✅ **Структура логів**: `logs/sessions/`, `logs/CHANGELOG.md`\n",
"- 📋 **Документація**: `logs/README.md`\n",
"\n",
"### 🔄 Git Multi-Remote (3 дзеркала)\n",
"- ✅ **GitHub** (origin) — основний репозиторій\n",
"- ✅ **Gitea** (localhost:3000) — локальне дзеркало\n",
"- ✅ **GitLab** (NODE3:8929) — додаткове дзеркало\n",
"- 📋 **Скрипт синхронізації**: `./scripts/git-sync-all.sh`\n",
"\n",
"### 🏗️ NODE1 Rebuild (Security)\n",
"- ✅ **Повний rebuild** — чиста Ubuntu 24.04 LTS\n",
"- ✅ **Docker 29.1.4** встановлено\n",
"- ✅ **Базове hardening** — UFW, fail2ban\n",
"- ⚠️ **Сервіси ще не задеплоєні**\n",
"\n",
"### 🐳 GitLab on NODE3\n",
"- ✅ **GitLab CE** встановлено (порт 8929)\n",
"- ✅ **Доступ через SSH tunnel**\n",
"- 📋 **Команда**: `ssh -p 33147 -L 8929:localhost:8929 zevs@80.77.35.151`\n",
"\n",
"---\n",
"\n",
"**🔴 CRITICAL (v2.4.0) - Jan 10, 2026:**\n",
"- 🔴 **Incident #4: NODE1 Host Compromise** — RESOLVED via full rebuild\n",
"- ✅ NODE1 перевстановлено з нуля\n",
"- ⚠️ **Secrets rotation needed** — див. `SECRETS-ROTATION-CHECKLIST.md`\n",
"\n",
"**v2.3.0:** \n",
"- 🖥️ **NODE3 added** - Threadripper PRO 5975WX + RTX 3090 24GB\n",
"- 🚀 Most powerful node for AI/ML workloads (32c/64t, 128GB RAM, 4TB NVMe)\n",
"- ✅ Security verified - clean system\n",
"\n",
"**v2.2.0:** \n",
"- 🔒 **Security Incident #2** (Jan 9, 2026) - Emergency mitigation completed\n",
"- ⚠️ **daarion-web permanently disabled** until secure rebuild\n",
"- ✅ Enhanced firewall rules + retry test registered with Hetzner\n",
"\n",
"**v2.1.0:** \n",
"- 🔒 **Security Incident #1 Resolved** (Dec 2025 - Jan 2026)\n",
"- ✅ Firewall rules + monitoring deployed\n",
"\n",
"**v2.0.0:** \n",
"- ✅ Мультимодальні сервіси (STT, OCR, Web Search, Vector DB) на НОДА2\n",
"- ✅ Router Multimodal Support (інтеграція в процесі)\n",
"- ✅ Telegram Gateway Enhanced (STT + Vision)\n",
"- ✅ Swapper Service інтеграція в кабінети НОД\n",
"- ✅ Кабінети мікроДАО з оркестраторами\n",
"- ✅ Оновлення в реальному часі (кожні 30 секунд)\n",
"- ✅ Управління мікроДАО в кабінеті DAARION"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Service Configuration (UPDATED with Swapper Service + Frontend + Agent Cabinet)\n",
"SERVICES = {\n",
" \"router\": {\"port\": 9102, \"container\": \"dagi-router\", \"health\": \"http://localhost:9102/health\"},\n",
" \"gateway\": {\"port\": 9300, \"container\": \"dagi-gateway\", \"health\": \"http://localhost:9300/health\"},\n",
" \"devtools\": {\"port\": 8008, \"container\": \"dagi-devtools\", \"health\": \"http://localhost:8008/health\"},\n",
" \"crewai\": {\"port\": 9010, \"container\": \"dagi-crewai\", \"health\": \"http://localhost:9010/health\"},\n",
" \"rbac\": {\"port\": 9200, \"container\": \"dagi-rbac\", \"health\": \"http://localhost:9200/health\"},\n",
" \"rag\": {\"port\": 9500, \"container\": \"dagi-rag-service\", \"health\": \"http://localhost:9500/health\"},\n",
" \"memory\": {\"port\": 8000, \"container\": \"dagi-memory-service\", \"health\": \"http://localhost:8000/health\"},\n",
" \"parser\": {\"port\": 9400, \"container\": \"dagi-parser-service\", \"health\": \"http://localhost:9400/health\"},\n",
" \"swapper\": {\"port\": 8890, \"container\": \"swapper-service\", \"health\": \"http://localhost:8890/health\", \"node1\": \"http://144.76.224.179:8890\", \"node2\": \"http://192.168.1.244:8890\"},\n",
" \"frontend\": {\"port\": 8899, \"container\": \"frontend\", \"health\": \"http://localhost:8899\"},\n",
" \"agent_cabinet\": {\"port\": 8898, \"container\": \"agent-cabinet-service\", \"health\": \"http://localhost:8898/health\"},\n",
" \"postgres\": {\"port\": 5432, \"container\": \"dagi-postgres\", \"health\": None},\n",
" \"redis\": {\"port\": 6379, \"container\": \"redis\", \"health\": \"redis-cli PING\"},\n",
" \"neo4j\": {\"port\": 7474, \"container\": \"neo4j\", \"health\": \"http://localhost:7474\"},\n",
" \"qdrant\": {\"port\": 6333, \"container\": \"dagi-qdrant\", \"health\": \"http://localhost:6333/healthz\"},\n",
" \"grafana\": {\"port\": 3000, \"container\": \"grafana\", \"health\": \"http://localhost:3000\"},\n",
" \"prometheus\": {\"port\": 9090, \"container\": \"prometheus\", \"health\": \"http://localhost:9090\"},\n",
" \"ollama\": {\"port\": 11434, \"container\": \"ollama\", \"health\": \"http://localhost:11434/api/tags\"}\n",
"}\n",
"\n",
"print(\"Service\\t\\t\\tPort\\tContainer\\t\\t\\tHealth Endpoint\")\n",
"print(\"=\"*100)\n",
"for name, service in SERVICES.items():\n",
" health = service['health'] or \"N/A\"\n",
" gpu = \" [GPU]\" if service.get('gpu') else \"\"\n",
" print(f\"{name.upper():<20} {service['port']:<7} {service['container']:<30} {health}{gpu}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 🖥️ Network Nodes\n",
"\n",
"### Node #1: Production Server (Hetzner)\n",
"- **Node ID:** node-1-hetzner-gex44\n",
"- **IP:** 144.76.224.179\n",
"- **Role:** Production Router + Gateway + All Services (24/7)\n",
"- **Location:** Hetzner Cloud (Germany)\n",
"\n",
"### Node #2: Development Node (MacBook Pro M4 Max)\n",
"- **Node ID:** node-2-macbook-m4max\n",
"- **Local IP:** 192.168.1.244\n",
"- **Role:** Development + Testing + Backup Router\n",
"- **Specs:** M4 Max (16 cores), 64GB RAM, 2TB SSD, 40-core GPU\n",
"- **Location:** Local Network (Ivan's Office)\n",
"- **Docs:** [NODE-2-MACBOOK-SPECS.md](../NODE-2-MACBOOK-SPECS.md)\n",
"\n",
"### Node #3: AI/ML Workstation (Threadripper PRO + RTX 3090)\n",
"- **Node ID:** node-3-threadripper-rtx3090\n",
"- **Hostname:** llm80-che-1-1\n",
"- **IP:** 80.77.35.151:33147\n",
"- **Role:** AI/ML Workloads, GPU Inference, Kubernetes\n",
"- **CPU:** AMD Threadripper PRO 5975WX (32c/64t, 3.6GHz)\n",
"- **RAM:** 128GB DDR4\n",
"- **GPU:** NVIDIA RTX 3090 24GB (CUDA 13.0)\n",
"- **Storage:** Samsung 990 PRO 4TB NVMe\n",
"- **OS:** Ubuntu 24.04 LTS + MicroK8s\n",
"- **Security:** ✅ Clean (verified 2026-01-09)\n",
"\n",
"---"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Network Nodes Configuration\n",
"NODES = {\n",
" \"node-1\": {\n",
" \"name\": \"Hetzner GEX44\",\n",
" \"ip\": \"144.76.224.179\",\n",
" \"local_ip\": None,\n",
" \"role\": \"production\",\n",
" \"uptime\": \"24/7\",\n",
" \"ssh\": \"root@144.76.224.179\",\n",
" \"domain\": \"gateway.daarion.city\",\n",
" \"services\": \"All (17 services)\",\n",
" \"specs\": \"See SYSTEM-INVENTORY.md\"\n",
" },\n",
" \"node-2\": {\n",
" \"name\": \"MacBook Pro M4 Max\",\n",
" \"ip\": None,\n",
" \"local_ip\": \"192.168.1.244\",\n",
" \"role\": \"development\",\n",
" \"uptime\": \"on-demand\",\n",
" \"ssh\": \"apple@192.168.1.244\",\n",
" \"domain\": None,\n",
" \"services\": \"Core only (Router, DevTools, Memory, Ollama)\",\n",
" \"specs\": \"M4 Max, 16 cores, 64GB RAM, 2TB SSD, 40-core GPU\"\n",
" },\n",
" \"node-3\": {\n",
" \"name\": \"Threadripper PRO + RTX 3090\",\n",
" \"ip\": \"80.77.35.151\",\n",
" \"local_ip\": None,\n",
" \"role\": \"ai_ml_workstation\",\n",
" \"uptime\": \"24/7\",\n",
" \"ssh\": \"zevs@80.77.35.151 -p33147\",\n",
" \"hostname\": \"llm80-che-1-1\",\n",
" \"domain\": None,\n",
" \"services\": \"MicroK8s, Ollama (GPU), MongoDB, K8s services\",\n",
" \"specs\": \"Threadripper PRO 5975WX (32c/64t), 128GB RAM, RTX 3090 24GB, Samsung 990 PRO 4TB\",\n",
" \"gpu\": \"NVIDIA RTX 3090 24GB VRAM (CUDA 13.0)\",\n",
" \"os\": \"Ubuntu 24.04 LTS\",\n",
" \"security_status\": \"Clean (verified 2026-01-09)\"\n",
" }\n",
"}\n",
"\n",
"print(\"DAGI Stack Network Nodes:\")\n",
"print(\"=\"*80)\n",
"for node_id, node in NODES.items():\n",
" print(f\"\\n{node_id.upper()}: {node['name']}\")\n",
" print(f\" Role: {node['role']}\")\n",
" print(f\" IP: {node['ip'] or node['local_ip']}\")\n",
" print(f\" SSH: {node['ssh']}\")\n",
" print(f\" Uptime: {node['uptime']}\")\n",
" print(f\" Services: {node['services']}\")\n",
" if node['domain']:\n",
" print(f\" Domain: https://{node['domain']}\")\n",
" print(f\" Specs: {node['specs']}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 🐙 GitHub Repositories\n",
"\n",
"### 1. MicroDAO (Current Project)\n",
"- **Repository:** `git@github.com:IvanTytar/microdao-daarion`\n",
"- **HTTPS:** `https://github.com/IvanTytar/microdao-daarion`\n",
"- **Remote Name:** `origin`\n",
"- **Main Branch:** `main`\n",
"- **Purpose:** MicroDAO core code, DAGI Stack, documentation\n",
"\n",
"### 2. DAARION.city\n",
"- **Repository:** `git@github.com:DAARION-DAO/daarion-ai-city.git`\n",
"- **HTTPS:** `https://github.com/DAARION-DAO/daarion-ai-city.git`\n",
"- **Remote Name:** `daarion-city`\n",
"- **Main Branch:** `main`\n",
"- **Purpose:** Official DAARION.city website and integrations\n",
"\n",
"---\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# GitHub Repositories Configuration\n",
"REPOSITORIES = {\n",
" \"microdao-daarion\": {\n",
" \"name\": \"MicroDAO\",\n",
" \"ssh_url\": \"git@github.com:IvanTytar/microdao-daarion\",\n",
" \"https_url\": \"https://github.com/IvanTytar/microdao-daarion\",\n",
" \"remote_name\": \"origin\",\n",
" \"main_branch\": \"main\",\n",
" \"purpose\": \"MicroDAO core code, DAGI Stack, documentation\",\n",
" \"clone_cmd\": \"git clone git@github.com:IvanTytar/microdao-daarion\"\n",
" },\n",
" \"daarion-ai-city\": {\n",
" \"name\": \"DAARION.city\",\n",
" \"ssh_url\": \"git@github.com:DAARION-DAO/daarion-ai-city.git\",\n",
" \"https_url\": \"https://github.com/DAARION-DAO/daarion-ai-city.git\",\n",
" \"remote_name\": \"daarion-city\",\n",
" \"main_branch\": \"main\",\n",
" \"purpose\": \"Official DAARION.city website and integrations\",\n",
" \"clone_cmd\": \"git clone git@github.com:DAARION-DAO/daarion-ai-city.git\"\n",
" }\n",
"}\n",
"\n",
"print(\"GitHub Repositories:\")\n",
"print(\"=\"*80)\n",
"for repo_id, repo in REPOSITORIES.items():\n",
" print(f\"\\n{repo['name']} ({repo_id})\")\n",
" print(f\" SSH URL: {repo['ssh_url']}\")\n",
" print(f\" HTTPS URL: {repo['https_url']}\")\n",
" print(f\" Remote: {repo['remote_name']}\")\n",
" print(f\" Branch: {repo['main_branch']}\")\n",
" print(f\" Purpose: {repo['purpose']}\")\n",
" print(f\" Clone: {repo['clone_cmd']}\")\n",
"\n",
"print(\"\\n\" + \"=\"*80)\n",
"print(\"\\nQuick Commands:\")\n",
"print(\"\\n# Clone MicroDAO:\")\n",
"print(\"git clone git@github.com:IvanTytar/microdao-daarion.git\")\n",
"print(\"\\n# Clone DAARION.city:\")\n",
"print(\"git clone git@github.com:DAARION-DAO/daarion-ai-city.git\")\n",
"print(\"\\n# Add DAARION.city as remote to MicroDAO:\")\n",
"print(\"cd microdao-daarion\")\n",
"print(\"git remote add daarion-city git@github.com:DAARION-DAO/daarion-ai-city.git\")\n",
"print(\"git fetch daarion-city\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"🤖 Для агентів Cursor: SSH доступ до НОДА1\n",
"\n",
"### Підключення до Production Server\n",
"\n",
"**SSH команда:**\n",
"```bash\n",
"ssh root@144.76.224.179\n",
"```\n",
"\n",
"**Робоча директорія:** `/opt/microdao-daarion`\n",
"\n",
"**Важливо:**\n",
"- SSH ключ має бути налаштований локально\n",
"- Працюєте від імені `root`\n",
"- Завжди перевіряйте `hostname` і `pwd` перед виконанням команд\n",
"- Не виконуйте деструктивні команди без підтвердження\n",
"\n",
"**Повна інструкція:** див. `INFRASTRUCTURE.md` → Для агентів Cursor"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# SSH Access for Cursor Agents\n",
"NODE1_ACCESS = {\n",
" \"host\": \"144.76.224.179\",\n",
" \"user\": \"root\",\n",
" \"ssh_command\": \"ssh root@144.76.224.179\",\n",
" \"project_root\": \"/opt/microdao-daarion\",\n",
" \"auth\": \"SSH key (configured locally)\",\n",
" \"common_commands\": [\n",
" \"docker ps\",\n",
" \"docker compose ps\",\n",
" \"docker logs <container_name> --tail 50\",\n",
" \"git status\",\n",
" \"git pull origin main\",\n",
" \"systemctl status docker\"\n",
" ],\n",
" \"safety_checks\": [\n",
" \"Always verify hostname before executing commands\",\n",
" \"Never use 'rm -rf' without confirmation\",\n",
" \"Never use 'docker rm -f' on production containers\",\n",
" \"Always check current directory with 'pwd'\",\n",
" \"Document all changes in git commits\"\n",
" ]\n",
"}\n",
"\n",
"print(\"🔐 SSH Access to NODE1:\")\n",
"print(\"=\"*60)\n",
"print(f\"Host: {NODE1_ACCESS['host']}\")\n",
"print(f\"User: {NODE1_ACCESS['user']}\")\n",
"print(f\"Command: {NODE1_ACCESS['ssh_command']}\")\n",
"print(f\"Project: {NODE1_ACCESS['project_root']}\")\n",
"print(f\"Auth: {NODE1_ACCESS['auth']}\")\n",
"print(\"\\nCommon Commands:\")\n",
"for cmd in NODE1_ACCESS['common_commands']:\n",
" print(f\" - {cmd}\")\n",
"print(\"\\n⚠️ Safety Checks:\")\n",
"for check in NODE1_ACCESS['safety_checks']:\n",
" print(f\" • {check}\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"⌘ Vision Encoder Service (NEW)\n",
"\n",
"### Overview\n",
"- **Service:** Vision Encoder (OpenCLIP ViT-L/14)\n",
"- **Port:** 8001\n",
"- **GPU:** Required (NVIDIA CUDA)\n",
"- **Embedding Dimension:** 768\n",
"- **Vector DB:** Qdrant (port 6333/6334)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Vision Encoder Configuration\n",
"VISION_ENCODER = {\n",
" \"service\": \"vision-encoder\",\n",
" \"port\": 8001,\n",
" \"container\": \"dagi-vision-encoder\",\n",
" \"gpu_required\": True,\n",
" \"model\": \"ViT-L-14\",\n",
" \"pretrained\": \"openai\",\n",
" \"embedding_dim\": 768,\n",
" \"endpoints\": {\n",
" \"health\": \"http://localhost:8001/health\",\n",
" \"info\": \"http://localhost:8001/info\",\n",
" \"embed_text\": \"http://localhost:8001/embed/text\",\n",
" \"embed_image\": \"http://localhost:8001/embed/image\",\n",
" \"docs\": \"http://localhost:8001/docs\"\n",
" },\n",
" \"qdrant\": {\n",
" \"host\": \"qdrant\",\n",
" \"port\": 6333,\n",
" \"grpc_port\": 6334,\n",
" \"health\": \"http://localhost:6333/healthz\"\n",
" }\n",
"}\n",
"\n",
"print(\"Vision Encoder Service Configuration:\")\n",
"print(\"=\"*80)\n",
"print(f\"Model: {VISION_ENCODER['model']} ({VISION_ENCODER['pretrained']})\")\n",
"print(f\"Embedding Dimension: {VISION_ENCODER['embedding_dim']}\")\n",
"print(f\"GPU Required: {VISION_ENCODER['gpu_required']}\")\n",
"print(f\"\\nEndpoints:\")\n",
"for name, url in VISION_ENCODER['endpoints'].items():\n",
" print(f\" {name:15} {url}\")\n",
"print(f\"\\nQdrant Vector DB:\")\n",
"print(f\" HTTP: http://localhost:{VISION_ENCODER['qdrant']['port']}\")\n",
"print(f\" gRPC: localhost:{VISION_ENCODER['qdrant']['grpc_port']}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Vision Encoder Testing Commands\n",
"VISION_ENCODER_TESTS = {\n",
" \"Health Check\": \"curl http://localhost:8001/health\",\n",
" \"Model Info\": \"curl http://localhost:8001/info\",\n",
" \"Text Embedding\": '''curl -X POST http://localhost:8001/embed/text -H \"Content-Type: application/json\" -d '{\"text\": \"DAARION governance\", \"normalize\": true}' ''',\n",
" \"Image Embedding\": '''curl -X POST http://localhost:8001/embed/image -H \"Content-Type: application/json\" -d '{\"image_url\": \"https://example.com/image.jpg\", \"normalize\": true}' ''',\n",
" \"Via Router (Text)\": '''curl -X POST http://localhost:9102/route -H \"Content-Type: application/json\" -d '{\"mode\": \"vision_embed\", \"message\": \"embed text\", \"payload\": {\"operation\": \"embed_text\", \"text\": \"test\", \"normalize\": true}}' ''',\n",
" \"Qdrant Health\": \"curl http://localhost:6333/healthz\",\n",
" \"Run Smoke Tests\": \"./test-vision-encoder.sh\"\n",
"}\n",
"\n",
"print(\"Vision Encoder Testing Commands:\")\n",
"print(\"=\"*80)\n",
"for name, cmd in VISION_ENCODER_TESTS.items():\n",
" print(f\"\\n{name}:\")\n",
" print(f\" {cmd}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 📖 Documentation Links (UPDATED)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Documentation References (UPDATED)\n",
"DOCS = {\n",
" \"Main Guide\": \"../WARP.md\",\n",
" \"Infrastructure\": \"../INFRASTRUCTURE.md\",\n",
" \"Agents Map\": \"../docs/agents.md\",\n",
" \"RAG Ingestion Status\": \"../RAG-INGESTION-STATUS.md\",\n",
" \"HMM Memory Status\": \"../HMM-MEMORY-STATUS.md\",\n",
" \"Crawl4AI Status\": \"../CRAWL4AI-STATUS.md\",\n",
" \"Vision Encoder Status\": \"../VISION-ENCODER-STATUS.md\",\n",
" \"Vision Encoder Deployment\": \"../services/vision-encoder/README.md\",\n",
" \"Repository Management\": \"../DAARION_CITY_REPO.md\",\n",
" \"Server Setup\": \"../SERVER_SETUP_INSTRUCTIONS.md\",\n",
" \"Deployment\": \"../DEPLOY-NOW.md\",\n",
" \"Helion Status\": \"../STATUS-HELION.md\",\n",
" \"Architecture Index\": \"../docs/cursor/README.md\",\n",
" \"API Reference\": \"../docs/api.md\",\n",
" \"Node #2 Specs\": \"../NODE-2-MACBOOK-SPECS.md\"\n",
"}\n",
"\n",
"print(\"Documentation Quick Links:\")\n",
"print(\"=\"*80)\n",
"for name, path in DOCS.items():\n",
" print(f\"{name:<30} {path}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 🎤 Мультимодальні Сервіси (НОДА2)\n",
"\n",
"Нові сервіси для розширення можливостей агентів:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"\n",
"multimodal_services = {\n",
" \"STT Service\": {\n",
" \"url\": \"http://192.168.1.244:8895\",\n",
" \"technology\": \"OpenAI Whisper AI\",\n",
" \"features\": [\"Voice→Text\", \"Ukrainian/English/Russian\", \"Telegram integration\"],\n",
" \"endpoints\": [\"/api/stt\", \"/api/stt/upload\", \"/health\"],\n",
" \"status\": \"✅ Ready\"\n",
" },\n",
" \"OCR Service\": {\n",
" \"url\": \"http://192.168.1.244:8896\",\n",
" \"technology\": \"Tesseract + EasyOCR\",\n",
" \"features\": [\"Image→Text\", \"Bounding boxes\", \"6 languages\", \"Confidence scores\"],\n",
" \"endpoints\": [\"/api/ocr\", \"/api/ocr/upload\", \"/health\"],\n",
" \"status\": \"✅ Ready\"\n",
" },\n",
" \"Web Search\": {\n",
" \"url\": \"http://192.168.1.244:8897\",\n",
" \"technology\": \"DuckDuckGo + Google\",\n",
" \"features\": [\"Real-time search\", \"Region-specific\", \"10+ results\"],\n",
" \"endpoints\": [\"/api/search\", \"/health\"],\n",
" \"status\": \"✅ Ready\"\n",
" },\n",
" \"Vector DB\": {\n",
" \"url\": \"http://192.168.1.244:8898\",\n",
" \"technology\": \"ChromaDB + Sentence Transformers\",\n",
" \"features\": [\"Vector database\", \"Semantic search\", \"RAG support\"],\n",
" \"endpoints\": [\"/api/collections\", \"/api/documents\", \"/api/search\", \"/health\"],\n",
" \"status\": \"✅ Ready\"\n",
" }\n",
"}\n",
"\n",
"pd.DataFrame(multimodal_services).T\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"ейсу"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"vision_agents = {\n",
" \"Sofia\": {\n",
" \"model\": \"grok-4.1\",\n",
" \"provider\": \"xAI\",\n",
" \"supports_vision\": True,\n",
" \"supports_files\": True,\n",
" \"description\": \"Vision + Code analysis\"\n",
" },\n",
" \"Spectra\": {\n",
" \"model\": \"qwen3-vl:latest\",\n",
" \"provider\": \"Ollama\",\n",
" \"supports_vision\": True,\n",
" \"supports_files\": False,\n",
" \"description\": \"Vision + Language\"\n",
" }\n",
"}\n",
"\n",
"pd.DataFrame(vision_agents).T\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 📊 Всі порти сервісів (оновлено)\n",
"\n",
"Повний список всіх сервісів з портами:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"all_ports = {\n",
" \"Frontend\": {\"port\": 8899, \"node\": \"Local\", \"status\": \"✅ Active\"},\n",
" \"STT Service\": {\"port\": 8895, \"node\": \"НОДА2\", \"status\": \"✅ Ready\"},\n",
" \"OCR Service\": {\"port\": 8896, \"node\": \"НОДА2\", \"status\": \"✅ Ready\"},\n",
" \"Web Search\": {\"port\": 8897, \"node\": \"НОДА2\", \"status\": \"✅ Ready\"},\n",
" \"Vector DB\": {\"port\": 8898, \"node\": \"НОДА2\", \"status\": \"✅ Ready\"},\n",
" \"Router\": {\"port\": 9102, \"node\": \"NODE1\", \"status\": \"🔄 Multimodal\"},\n",
" \"Telegram Gateway\": {\"port\": 9200, \"node\": \"NODE1\", \"status\": \"🔄 Enhanced\"},\n",
" \"Swapper NODE1\": {\"port\": 8890, \"node\": \"NODE1\", \"status\": \"✅ Active\"},\n",
" \"Swapper NODE2\": {\"port\": 8890, \"node\": \"НОДА2\", \"status\": \"✅ Active\"},\n",
" \"Agent Cabinet\": {\"port\": 8898, \"node\": \"Local\", \"status\": \"✅ Active\"},\n",
" \"Memory Service\": {\"port\": 8000, \"node\": \"NODE1/2\", \"status\": \"✅ Active\"}\n",
"}\n",
"\n",
"pd.DataFrame(all_ports).T\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 🔄 Мультимодальні можливості\n",
"\n",
"Статус інтеграції різних типів контенту:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"multimodal_capabilities = {\n",
" \"Текст\": {\"frontend\": \"✅\", \"telegram\": \"✅\", \"status\": \"ПРАЦЮЄ\"},\n",
" \"Голос→Текст\": {\"frontend\": \"✅\", \"telegram\": \"🔄\", \"status\": \"ІНТЕГРАЦІЯ\"},\n",
" \"Зображення→Vision\": {\"frontend\": \"✅\", \"telegram\": \"🔄\", \"status\": \"ІНТЕГРАЦІЯ\"},\n",
" \"Зображення→OCR\": {\"frontend\": \"✅\", \"telegram\": \"🔄\", \"status\": \"ІНТЕГРАЦІЯ\"},\n",
" \"Документи\": {\"frontend\": \"✅\", \"telegram\": \"⚠️\", \"status\": \"ЧАСТКОВА\"},\n",
" \"Веб-пошук\": {\"frontend\": \"✅\", \"telegram\": \"🔄\", \"status\": \"ІНТЕГРАЦІЯ\"},\n",
" \"Knowledge Base\": {\"frontend\": \"✅\", \"telegram\": \"❌\", \"status\": \"ГОТОВИЙ\"}\n",
"}\n",
"\n",
"pd.DataFrame(multimodal_capabilities).T\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 🔒 Security & Incident Response\n",
"\n",
"### Incident #1: Network Scanning & Lockdown (Dec 6, 2025 - Jan 8, 2026)\n",
"\n",
"**Root Cause:** Compromised `daarion-web` container with cryptocurrency miner (`catcal`, `G4NQXBp`)\n",
"**Impact:** Server locked by Hetzner for 33 days due to internal network scanning\n",
"**Resolution:** Container removed, firewall rules implemented, monitoring deployed\n",
"\n",
"### Incident #2: Recurring Compromise (Jan 9, 2026) 🔴 ACTIVE\n",
"\n",
"**Root Cause:** Compromised Docker image auto-restarted after server reboot \n",
"**Malware:** NEW crypto miners (`softirq`, `vrarhpb`) - different from Incident #1 \n",
"**Impact:** \n",
"- ❌ Second abuse report (AbuseID: 10F3971:2A)\n",
"- ❌ Critical CPU load: 25-35 (normal: 1-5)\n",
"- ❌ 1499 zombie processes\n",
"- ⚠️ Deadline: 2026-01-09 12:54 UTC (~3.5 hours remaining)\n",
"\n",
"**Resolution (COMPLETED):** \n",
"1. ✅ Killed all malicious processes (softirq, vrarhpb)\n",
"2. ✅ Stopped and removed `daarion-web` container\n",
"3. ✅ **DELETED Docker images** (78e22c0ee972, 608e203fb5ac) - critical step\n",
"4. ✅ Cleaned 1499 zombie processes → 5 (normal)\n",
"5. ✅ System load normalized: 30+ → 4.19\n",
"6. ✅ Enhanced firewall (SSH rate limiting, port scan blocking)\n",
"7. ✅ Registered retry test with Hetzner\n",
"8. ⏳ **PENDING:** User statement submission (URGENT)\n",
"\n",
"**Why Incident #2 Occurred:** \n",
"- Incident #1 removed container but LEFT Docker image intact\n",
"- Container had `restart: unless-stopped` in docker-compose.yml\n",
"- Server rebooted → docker-compose auto-restarted from compromised image\n",
"- NEW malware variant installed (different miners than Incident #1)\n",
"\n",
"**What is daarion-web?** \n",
"- Next.js frontend (port 3000) - NOT critical for core functionality\n",
"- ✅ Router, Gateway, Telegram bots, API - ALL WORKING\n",
"- Status: DISABLED until secure rebuild completed\n",
"\n",
"**Lessons Learned (Critical):** \n",
"1. 🔴 **ALWAYS delete Docker images, not just containers**\n",
"2. 🟡 **Auto-restart policies are dangerous for compromised containers**\n",
"3. 🟢 **Compromised images can survive container removal**\n",
"4. 🔵 **Complete removal = container + image + restart policy change**\n",
"\n",
"**Next Steps:** \n",
"1. 🔴 **URGENT:** Submit statement to Hetzner before deadline\n",
"2. 🟡 Monitor server for 24 hours post-statement\n",
"3. 🟢 Secure rebuild of daarion-web (see `TASK_REBUILD_DAARION_WEB.md`)\n",
"4. 🔵 Security audit all remaining containers\n",
"\n",
"### Security Measures\n",
"\n",
"1. **Egress Firewall Rules** (блокування внутрішніх мереж Hetzner)\n",
"2. **Monitoring Script** (`/root/monitor_scanning.sh`, runs every 15 min)\n",
"3. **Security Checklist:**\n",
" - [ ] Container vulnerability scanning\n",
" - [ ] Docker Content Trust\n",
" - [ ] Resource limits (CPU/memory)\n",
" - [ ] Network segmentation\n",
" - [ ] Regular security audits\n",
"\n",
"**Full details:** See `INFRASTRUCTURE.md` → Security & Incident Response section\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Security Configuration (UPDATED with Incident #2)\n",
"security_config = {\n",
" \"Firewall Rules\": {\n",
" \"scripts\": [\"/root/prevent_scanning.sh\", \"/root/block_ssh_scanning.sh\"],\n",
" \"status\": \"✅ Enhanced\",\n",
" \"blocks\": [\"10.0.0.0/8\", \"172.16.0.0/12\"],\n",
" \"allows\": [\"80/tcp\", \"443/tcp\"],\n",
" \"features\": [\"SSH rate limiting\", \"Port scan blocking\", \"Enhanced logging\"]\n",
" },\n",
" \"Monitoring\": {\n",
" \"script\": \"/root/monitor_scanning.sh\",\n",
" \"status\": \"✅ Active\",\n",
" \"interval\": \"15 minutes\",\n",
" \"log\": \"/var/log/scan_attempts.log\"\n",
" },\n",
" \"Incident #1\": {\n",
" \"date\": \"2025-12-06\",\n",
" \"malware\": \"catcal, G4NQXBp\",\n",
" \"recovery_time\": \"33 days\",\n",
" \"status\": \"✅ Resolved\"\n",
" },\n",
" \"Incident #2\": {\n",
" \"date\": \"2026-01-09\",\n",
" \"malware\": \"softirq, vrarhpb\",\n",
" \"mitigation_time\": \"30 minutes\",\n",
" \"status\": \"⏳ Statement Pending\",\n",
" \"deadline\": \"2026-01-09 12:54 UTC\",\n",
" \"actions\": [\"Container removed\", \"Images DELETED\", \"Load normalized\", \"Retry test registered\"]\n",
" }\n",
"}\n",
"\n",
"import pandas as pd\n",
"print(\"🔒 Security Configuration:\")\n",
"print(\"=\" * 80)\n",
"pd.DataFrame(security_config).T\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 📝 Notes & Updates\n",
"\n",
"### Recent Changes (2026-01-10)\n",
"- 📝 **Session Logging System** — автоматичне логування всіх дій\n",
"- 🔄 **Git Multi-Remote** — GitHub + Gitea + GitLab синхронізація\n",
"- 🏗️ **NODE1 Rebuild** — чиста Ubuntu 24.04 + Docker 29.1.4\n",
"- 🐳 **GitLab on NODE3** — додаткове дзеркало (порт 8929)\n",
"- ✅ **Git hooks** — автологування commits/pushes\n",
"- ✅ **Shell integration** — команди session-start/log/end\n",
"\n",
"### Recent Changes (2025-11-23)\n",
"- ✅ **Swapper Service інтеграція** в кабінети НОД (тільки в `/nodes/node-1`, `/nodes/node-2`)\n",
"- ✅ **Оновлення в реальному часі** (кожні 30 секунд) для Swapper Service\n",
"- ✅ **Кабінети мікроДАО** з оркестраторами (DAARION, GREENFOOD, ENERGY UNION)\n",
"- ✅ **Управління мікроДАО** в кабінеті DAARION (панель управління всіма мікроДАО)\n",
"- ✅ **Детальні метрики Swapper Service** (моделі, спеціалісти, конфігурація)\n",
"- ✅ **Frontend** (port 8899) з кабінетами НОД та мікроДАО\n",
"- ✅ **Agent Cabinet Service** (port 8898) для метрик агентів\n",
"\n",
"### Network Architecture\n",
"- **Nodes:** 3 (NODE1 production + NODE2 development + NODE3 AI/ML)\n",
"- **Total Services:** 19 (додано Frontend + Agent Cabinet)\n",
"- **Git Remotes:** 3 (GitHub + Gitea + GitLab)\n",
"- **MicroDAO Cabinets:** 3 (DAARION, GREENFOOD, ENERGY UNION)\n",
"- **Node Cabinets:** 2 (НОДА1, НОДА2)\n",
"\n",
"### Кабінети НОД\n",
"- **НОДА1:** `http://localhost:8899/nodes/node-1`\n",
"- **НОДА2:** `http://localhost:8899/nodes/node-2`\n",
"- **Swapper Service:** Відображається тільки тут, оновлення кожні 30 секунд\n",
"\n",
"### Кабінети МікроДАО\n",
"- **DAARION:** `http://localhost:8899/microdao/daarion` (оркестратор: DAARWIZZ)\n",
"- **GREENFOOD:** `http://localhost:8899/microdao/greenfood` (оркестратор: GREENFOOD)\n",
"- **ENERGY UNION:** `http://localhost:8899/microdao/energy-union` (оркестратор: Helion)\n",
"\n",
"### Git Repositories\n",
"- **GitHub:** `git@github.com:IvanTytar/microdao-daarion.git` (origin)\n",
"- **Gitea:** `http://localhost:3000/daarion-admin/microdao-daarion.git`\n",
"- **GitLab:** `http://localhost:8929/root/microdao-daarion.git` (через SSH tunnel)\n",
"\n",
"---\n",
"\n",
"**Last Updated:** 2026-01-10 14:55 (Session Logging System + NODE1 Rebuild) \n",
"**Maintained by:** Ivan Tytar & DAARION Team \n",
"\n",
"---\n",
"\n",
"### ✅ Security Status\n",
"- **NODE1:** Rebuilt from scratch (Ubuntu 24.04 + Docker)\n",
"- **NODE3:** Clean (verified 2026-01-09)\n",
"- **Secrets:** Rotation pending — див. `SECRETS-ROTATION-CHECKLIST.md`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 🔴 Incident #4: NODE1 Host Compromise (Jan 10, 2026)\n",
"\n",
"### Summary\n",
"ALL PostgreSQL official images show malware artifacts when run on NODE1.\n",
"This is **NOT** \"Docker Hub compromised\" — this is **NODE1 host compromise**.\n",
"\n",
"### Indicators of Compromise (IOC)\n",
"```\n",
"/tmp/httpd # ~10MB crypto miner (xmrig variant)\n",
"/tmp/.perf.c/ # perfctl malware staging directory\n",
"/tmp/mysql # Another miner variant\n",
"/tmp/cpioshuf # perfctl payload\n",
"/tmp/ipcalc* # perfctl payload\n",
"```\n",
"\n",
"### Affected Images (on NODE1)\n",
"- ❌ postgres:15-alpine\n",
"- ❌ postgres:16-alpine\n",
"- ❌ postgres:14\n",
"- ❌ postgres:16 (Debian)\n",
"\n",
"### Why This is HOST Compromise (not image)\n",
"1. ALL different image variants show same IOC\n",
"2. Previous incidents (#1, #2, #3) already compromised NODE1\n",
"3. `/tmp/.perf.c/` is classic perfctl malware directory\n",
"4. `tmpfs noexec` didn't prevent infection\n",
"\n",
"### Verification Procedure\n",
"```bash\n",
"# Run triage script from MacBook (NOT NODE1!)\n",
"cd /Users/apple/github-projects/microdao-daarion\n",
"./scripts/security/triage-postgres-compromise.sh compare\n",
"\n",
"# Or manually:\n",
"# 1. Get digest from NODE1\n",
"ssh root@144.76.224.179 \"docker inspect --format='{{index .RepoDigests 0}}' postgres:16\"\n",
"\n",
"# 2. Pull same digest on MacBook\n",
"docker pull postgres:16@sha256:<digest>\n",
"\n",
"# 3. Check if clean\n",
"docker run --rm postgres:16@sha256:<digest> ls -la /tmp/\n",
"# If empty → NODE1 compromised, image is clean\n",
"```\n",
"\n",
"### Current Status\n",
"- ⏳ **Verification pending** — Need to test on clean host\n",
"- 🔴 **NODE1 UNSAFE** — Do not deploy PostgreSQL\n",
"- 🟡 **Secrets rotation needed** — Assume all compromised\n",
"\n",
"### Full Documentation\n",
"See `INFRASTRUCTURE.md` → Incident #4"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 📝 Session Logging System\n",
"\n",
"### Автоматичне логування всіх дій\n",
"\n",
"Система автоматично записує всі дії при роботі над проєктом.\n",
"\n",
"### Структура логів\n",
"```\n",
"logs/\n",
"├── README.md # Документація\n",
"├── CHANGELOG.md # Головний журнал змін\n",
"├── sessions/ # Щоденні логи сесій\n",
"│ └── YYYY-MM-DD.md # Лог конкретного дня\n",
"├── operations/ # Операційні логи\n",
"└── incidents/ # Логи інцидентів\n",
"```\n",
"\n",
"### Команди (після `source ~/.zshrc`)\n",
"\n",
"| Команда | Опис |\n",
"|---------|------|\n",
"| `session-start \"опис\"` | Почати сесію |\n",
"| `session-log \"дія\"` | Додати запис |\n",
"| `session-end` | Завершити (commit + push) |\n",
"| `daarion-note \"нотатка\"` | Швидка нотатка |\n",
"| `git-sync` | Push на всі remote |\n",
"\n",
"### Автоматичне логування (Git hooks)\n",
"- ✅ Кожен `git commit` → записується в session log\n",
"- ✅ Кожен `git push` → записується в session log\n",
"\n",
"### Встановлення\n",
"```bash\n",
"# 1. Встановити Git hooks\n",
"./scripts/logging/install-hooks.sh\n",
"\n",
"# 2. Додати shell integration\n",
"echo 'source /path/to/scripts/logging/shell-integration.sh' >> ~/.zshrc\n",
"source ~/.zshrc\n",
"```\n",
"\n",
"### Git Multi-Remote (3 дзеркала)\n",
"```bash\n",
"# Всі remote\n",
"git remote -v\n",
"# origin git@github.com:IvanTytar/microdao-daarion.git\n",
"# gitea http://localhost:3000/daarion-admin/microdao-daarion.git\n",
"# gitlab http://localhost:8929/root/microdao-daarion.git\n",
"\n",
"# Push на всі\n",
"./scripts/git-sync-all.sh\n",
"# або\n",
"git push origin && git push gitea && git push gitlab\n",
"```\n",
"\n",
"### SSH Tunnel до GitLab (NODE3)\n",
"```bash\n",
"ssh -p 33147 -L 8929:localhost:8929 -N zevs@80.77.35.151 &\n",
"```"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.0"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Reference in New Issue View Git Blame Copy Permalink