{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 🚀 Infrastructure Quick Reference — DAARION & MicroDAO\n", "\n", "**Версія:** 2.5.0 \n", "**Останнє оновлення:** 2026-01-10 14:55 \n", "\n", "Цей notebook містить швидкий довідник по серверах, репозиторіях та endpoints для DAGI Stack.\n", "\n", "---\n", "\n", "## 🆕 What's New (v2.5.0) - Jan 10, 2026\n", "\n", "### 📝 Session Logging System\n", "- ✅ **Автоматичне логування** всіх дій (Git hooks)\n", "- ✅ **Shell integration** — команди `session-start`, `session-log`, `session-end`\n", "- ✅ **Структура логів**: `logs/sessions/`, `logs/CHANGELOG.md`\n", "- 📋 **Документація**: `logs/README.md`\n", "\n", "### 🔄 Git Multi-Remote (3 дзеркала)\n", "- ✅ **GitHub** (origin) — основний репозиторій\n", "- ✅ **Gitea** (localhost:3000) — локальне дзеркало\n", "- ✅ **GitLab** (NODE3:8929) — додаткове дзеркало\n", "- 📋 **Скрипт синхронізації**: `./scripts/git-sync-all.sh`\n", "\n", "### 🏗️ NODE1 Rebuild (Security)\n", "- ✅ **Повний rebuild** — чиста Ubuntu 24.04 LTS\n", "- ✅ **Docker 29.1.4** встановлено\n", "- ✅ **Базове hardening** — UFW, fail2ban\n", "- ⚠️ **Сервіси ще не задеплоєні**\n", "\n", "### 🐳 GitLab on NODE3\n", "- ✅ **GitLab CE** встановлено (порт 8929)\n", "- ✅ **Доступ через SSH tunnel**\n", "- 📋 **Команда**: `ssh -p 33147 -L 8929:localhost:8929 zevs@80.77.35.151`\n", "\n", "---\n", "\n", "**🔴 CRITICAL (v2.4.0) - Jan 10, 2026:**\n", "- 🔴 **Incident #4: NODE1 Host Compromise** — RESOLVED via full rebuild\n", "- ✅ NODE1 перевстановлено з нуля\n", "- ⚠️ **Secrets rotation needed** — див. `SECRETS-ROTATION-CHECKLIST.md`\n", "\n", "**v2.3.0:** \n", "- 🖥️ **NODE3 added** - Threadripper PRO 5975WX + RTX 3090 24GB\n", "- 🚀 Most powerful node for AI/ML workloads (32c/64t, 128GB RAM, 4TB NVMe)\n", "- ✅ Security verified - clean system\n", "\n", "**v2.2.0:** \n", "- 🔒 **Security Incident #2** (Jan 9, 2026) - Emergency mitigation completed\n", "- ⚠️ **daarion-web permanently disabled** until secure rebuild\n", "- ✅ Enhanced firewall rules + retry test registered with Hetzner\n", "\n", "**v2.1.0:** \n", "- 🔒 **Security Incident #1 Resolved** (Dec 2025 - Jan 2026)\n", "- ✅ Firewall rules + monitoring deployed\n", "\n", "**v2.0.0:** \n", "- ✅ Мультимодальні сервіси (STT, OCR, Web Search, Vector DB) на НОДА2\n", "- ✅ Router Multimodal Support (інтеграція в процесі)\n", "- ✅ Telegram Gateway Enhanced (STT + Vision)\n", "- ✅ Swapper Service інтеграція в кабінети НОД\n", "- ✅ Кабінети мікроДАО з оркестраторами\n", "- ✅ Оновлення в реальному часі (кожні 30 секунд)\n", "- ✅ Управління мікроДАО в кабінеті DAARION" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Service Configuration (UPDATED with Swapper Service + Frontend + Agent Cabinet)\n", "SERVICES = {\n", " \"router\": {\"port\": 9102, \"container\": \"dagi-router\", \"health\": \"http://localhost:9102/health\"},\n", " \"gateway\": {\"port\": 9300, \"container\": \"dagi-gateway\", \"health\": \"http://localhost:9300/health\"},\n", " \"devtools\": {\"port\": 8008, \"container\": \"dagi-devtools\", \"health\": \"http://localhost:8008/health\"},\n", " \"crewai\": {\"port\": 9010, \"container\": \"dagi-crewai\", \"health\": \"http://localhost:9010/health\"},\n", " \"rbac\": {\"port\": 9200, \"container\": \"dagi-rbac\", \"health\": \"http://localhost:9200/health\"},\n", " \"rag\": {\"port\": 9500, \"container\": \"dagi-rag-service\", \"health\": \"http://localhost:9500/health\"},\n", " \"memory\": {\"port\": 8000, \"container\": \"dagi-memory-service\", \"health\": \"http://localhost:8000/health\"},\n", " \"parser\": {\"port\": 9400, \"container\": \"dagi-parser-service\", \"health\": \"http://localhost:9400/health\"},\n", " \"swapper\": {\"port\": 8890, \"container\": \"swapper-service\", \"health\": \"http://localhost:8890/health\", \"node1\": \"http://144.76.224.179:8890\", \"node2\": \"http://192.168.1.244:8890\"},\n", " \"frontend\": {\"port\": 8899, \"container\": \"frontend\", \"health\": \"http://localhost:8899\"},\n", " \"agent_cabinet\": {\"port\": 8898, \"container\": \"agent-cabinet-service\", \"health\": \"http://localhost:8898/health\"},\n", " \"postgres\": {\"port\": 5432, \"container\": \"dagi-postgres\", \"health\": None},\n", " \"redis\": {\"port\": 6379, \"container\": \"redis\", \"health\": \"redis-cli PING\"},\n", " \"neo4j\": {\"port\": 7474, \"container\": \"neo4j\", \"health\": \"http://localhost:7474\"},\n", " \"qdrant\": {\"port\": 6333, \"container\": \"dagi-qdrant\", \"health\": \"http://localhost:6333/healthz\"},\n", " \"grafana\": {\"port\": 3000, \"container\": \"grafana\", \"health\": \"http://localhost:3000\"},\n", " \"prometheus\": {\"port\": 9090, \"container\": \"prometheus\", \"health\": \"http://localhost:9090\"},\n", " \"ollama\": {\"port\": 11434, \"container\": \"ollama\", \"health\": \"http://localhost:11434/api/tags\"}\n", "}\n", "\n", "print(\"Service\\t\\t\\tPort\\tContainer\\t\\t\\tHealth Endpoint\")\n", "print(\"=\"*100)\n", "for name, service in SERVICES.items():\n", " health = service['health'] or \"N/A\"\n", " gpu = \" [GPU]\" if service.get('gpu') else \"\"\n", " print(f\"{name.upper():<20} {service['port']:<7} {service['container']:<30} {health}{gpu}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 🖥️ Network Nodes\n", "\n", "### Node #1: Production Server (Hetzner)\n", "- **Node ID:** node-1-hetzner-gex44\n", "- **IP:** 144.76.224.179\n", "- **Role:** Production Router + Gateway + All Services (24/7)\n", "- **Location:** Hetzner Cloud (Germany)\n", "\n", "### Node #2: Development Node (MacBook Pro M4 Max)\n", "- **Node ID:** node-2-macbook-m4max\n", "- **Local IP:** 192.168.1.244\n", "- **Role:** Development + Testing + Backup Router\n", "- **Specs:** M4 Max (16 cores), 64GB RAM, 2TB SSD, 40-core GPU\n", "- **Location:** Local Network (Ivan's Office)\n", "- **Docs:** [NODE-2-MACBOOK-SPECS.md](../NODE-2-MACBOOK-SPECS.md)\n", "\n", "### Node #3: AI/ML Workstation (Threadripper PRO + RTX 3090)\n", "- **Node ID:** node-3-threadripper-rtx3090\n", "- **Hostname:** llm80-che-1-1\n", "- **IP:** 80.77.35.151:33147\n", "- **Role:** AI/ML Workloads, GPU Inference, Kubernetes\n", "- **CPU:** AMD Threadripper PRO 5975WX (32c/64t, 3.6GHz)\n", "- **RAM:** 128GB DDR4\n", "- **GPU:** NVIDIA RTX 3090 24GB (CUDA 13.0)\n", "- **Storage:** Samsung 990 PRO 4TB NVMe\n", "- **OS:** Ubuntu 24.04 LTS + MicroK8s\n", "- **Security:** ✅ Clean (verified 2026-01-09)\n", "\n", "---" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Network Nodes Configuration\n", "NODES = {\n", " \"node-1\": {\n", " \"name\": \"Hetzner GEX44\",\n", " \"ip\": \"144.76.224.179\",\n", " \"local_ip\": None,\n", " \"role\": \"production\",\n", " \"uptime\": \"24/7\",\n", " \"ssh\": \"root@144.76.224.179\",\n", " \"domain\": \"gateway.daarion.city\",\n", " \"services\": \"All (17 services)\",\n", " \"specs\": \"See SYSTEM-INVENTORY.md\"\n", " },\n", " \"node-2\": {\n", " \"name\": \"MacBook Pro M4 Max\",\n", " \"ip\": None,\n", " \"local_ip\": \"192.168.1.244\",\n", " \"role\": \"development\",\n", " \"uptime\": \"on-demand\",\n", " \"ssh\": \"apple@192.168.1.244\",\n", " \"domain\": None,\n", " \"services\": \"Core only (Router, DevTools, Memory, Ollama)\",\n", " \"specs\": \"M4 Max, 16 cores, 64GB RAM, 2TB SSD, 40-core GPU\"\n", " },\n", " \"node-3\": {\n", " \"name\": \"Threadripper PRO + RTX 3090\",\n", " \"ip\": \"80.77.35.151\",\n", " \"local_ip\": None,\n", " \"role\": \"ai_ml_workstation\",\n", " \"uptime\": \"24/7\",\n", " \"ssh\": \"zevs@80.77.35.151 -p33147\",\n", " \"hostname\": \"llm80-che-1-1\",\n", " \"domain\": None,\n", " \"services\": \"MicroK8s, Ollama (GPU), MongoDB, K8s services\",\n", " \"specs\": \"Threadripper PRO 5975WX (32c/64t), 128GB RAM, RTX 3090 24GB, Samsung 990 PRO 4TB\",\n", " \"gpu\": \"NVIDIA RTX 3090 24GB VRAM (CUDA 13.0)\",\n", " \"os\": \"Ubuntu 24.04 LTS\",\n", " \"security_status\": \"Clean (verified 2026-01-09)\"\n", " }\n", "}\n", "\n", "print(\"DAGI Stack Network Nodes:\")\n", "print(\"=\"*80)\n", "for node_id, node in NODES.items():\n", " print(f\"\\n{node_id.upper()}: {node['name']}\")\n", " print(f\" Role: {node['role']}\")\n", " print(f\" IP: {node['ip'] or node['local_ip']}\")\n", " print(f\" SSH: {node['ssh']}\")\n", " print(f\" Uptime: {node['uptime']}\")\n", " print(f\" Services: {node['services']}\")\n", " if node['domain']:\n", " print(f\" Domain: https://{node['domain']}\")\n", " print(f\" Specs: {node['specs']}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 🐙 GitHub Repositories\n", "\n", "### 1. MicroDAO (Current Project)\n", "- **Repository:** `git@github.com:IvanTytar/microdao-daarion`\n", "- **HTTPS:** `https://github.com/IvanTytar/microdao-daarion`\n", "- **Remote Name:** `origin`\n", "- **Main Branch:** `main`\n", "- **Purpose:** MicroDAO core code, DAGI Stack, documentation\n", "\n", "### 2. DAARION.city\n", "- **Repository:** `git@github.com:DAARION-DAO/daarion-ai-city.git`\n", "- **HTTPS:** `https://github.com/DAARION-DAO/daarion-ai-city.git`\n", "- **Remote Name:** `daarion-city`\n", "- **Main Branch:** `main`\n", "- **Purpose:** Official DAARION.city website and integrations\n", "\n", "---\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# GitHub Repositories Configuration\n", "REPOSITORIES = {\n", " \"microdao-daarion\": {\n", " \"name\": \"MicroDAO\",\n", " \"ssh_url\": \"git@github.com:IvanTytar/microdao-daarion\",\n", " \"https_url\": \"https://github.com/IvanTytar/microdao-daarion\",\n", " \"remote_name\": \"origin\",\n", " \"main_branch\": \"main\",\n", " \"purpose\": \"MicroDAO core code, DAGI Stack, documentation\",\n", " \"clone_cmd\": \"git clone git@github.com:IvanTytar/microdao-daarion\"\n", " },\n", " \"daarion-ai-city\": {\n", " \"name\": \"DAARION.city\",\n", " \"ssh_url\": \"git@github.com:DAARION-DAO/daarion-ai-city.git\",\n", " \"https_url\": \"https://github.com/DAARION-DAO/daarion-ai-city.git\",\n", " \"remote_name\": \"daarion-city\",\n", " \"main_branch\": \"main\",\n", " \"purpose\": \"Official DAARION.city website and integrations\",\n", " \"clone_cmd\": \"git clone git@github.com:DAARION-DAO/daarion-ai-city.git\"\n", " }\n", "}\n", "\n", "print(\"GitHub Repositories:\")\n", "print(\"=\"*80)\n", "for repo_id, repo in REPOSITORIES.items():\n", " print(f\"\\n{repo['name']} ({repo_id})\")\n", " print(f\" SSH URL: {repo['ssh_url']}\")\n", " print(f\" HTTPS URL: {repo['https_url']}\")\n", " print(f\" Remote: {repo['remote_name']}\")\n", " print(f\" Branch: {repo['main_branch']}\")\n", " print(f\" Purpose: {repo['purpose']}\")\n", " print(f\" Clone: {repo['clone_cmd']}\")\n", "\n", "print(\"\\n\" + \"=\"*80)\n", "print(\"\\nQuick Commands:\")\n", "print(\"\\n# Clone MicroDAO:\")\n", "print(\"git clone git@github.com:IvanTytar/microdao-daarion.git\")\n", "print(\"\\n# Clone DAARION.city:\")\n", "print(\"git clone git@github.com:DAARION-DAO/daarion-ai-city.git\")\n", "print(\"\\n# Add DAARION.city as remote to MicroDAO:\")\n", "print(\"cd microdao-daarion\")\n", "print(\"git remote add daarion-city git@github.com:DAARION-DAO/daarion-ai-city.git\")\n", "print(\"git fetch daarion-city\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "🤖 Для агентів Cursor: SSH доступ до НОДА1\n", "\n", "### Підключення до Production Server\n", "\n", "**SSH команда:**\n", "```bash\n", "ssh root@144.76.224.179\n", "```\n", "\n", "**Робоча директорія:** `/opt/microdao-daarion`\n", "\n", "**Важливо:**\n", "- SSH ключ має бути налаштований локально\n", "- Працюєте від імені `root`\n", "- Завжди перевіряйте `hostname` і `pwd` перед виконанням команд\n", "- Не виконуйте деструктивні команди без підтвердження\n", "\n", "**Повна інструкція:** див. `INFRASTRUCTURE.md` → Для агентів Cursor" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# SSH Access for Cursor Agents\n", "NODE1_ACCESS = {\n", " \"host\": \"144.76.224.179\",\n", " \"user\": \"root\",\n", " \"ssh_command\": \"ssh root@144.76.224.179\",\n", " \"project_root\": \"/opt/microdao-daarion\",\n", " \"auth\": \"SSH key (configured locally)\",\n", " \"common_commands\": [\n", " \"docker ps\",\n", " \"docker compose ps\",\n", " \"docker logs --tail 50\",\n", " \"git status\",\n", " \"git pull origin main\",\n", " \"systemctl status docker\"\n", " ],\n", " \"safety_checks\": [\n", " \"Always verify hostname before executing commands\",\n", " \"Never use 'rm -rf' without confirmation\",\n", " \"Never use 'docker rm -f' on production containers\",\n", " \"Always check current directory with 'pwd'\",\n", " \"Document all changes in git commits\"\n", " ]\n", "}\n", "\n", "print(\"🔐 SSH Access to NODE1:\")\n", "print(\"=\"*60)\n", "print(f\"Host: {NODE1_ACCESS['host']}\")\n", "print(f\"User: {NODE1_ACCESS['user']}\")\n", "print(f\"Command: {NODE1_ACCESS['ssh_command']}\")\n", "print(f\"Project: {NODE1_ACCESS['project_root']}\")\n", "print(f\"Auth: {NODE1_ACCESS['auth']}\")\n", "print(\"\\nCommon Commands:\")\n", "for cmd in NODE1_ACCESS['common_commands']:\n", " print(f\" - {cmd}\")\n", "print(\"\\n⚠️ Safety Checks:\")\n", "for check in NODE1_ACCESS['safety_checks']:\n", " print(f\" • {check}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "⌘ Vision Encoder Service (NEW)\n", "\n", "### Overview\n", "- **Service:** Vision Encoder (OpenCLIP ViT-L/14)\n", "- **Port:** 8001\n", "- **GPU:** Required (NVIDIA CUDA)\n", "- **Embedding Dimension:** 768\n", "- **Vector DB:** Qdrant (port 6333/6334)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Vision Encoder Configuration\n", "VISION_ENCODER = {\n", " \"service\": \"vision-encoder\",\n", " \"port\": 8001,\n", " \"container\": \"dagi-vision-encoder\",\n", " \"gpu_required\": True,\n", " \"model\": \"ViT-L-14\",\n", " \"pretrained\": \"openai\",\n", " \"embedding_dim\": 768,\n", " \"endpoints\": {\n", " \"health\": \"http://localhost:8001/health\",\n", " \"info\": \"http://localhost:8001/info\",\n", " \"embed_text\": \"http://localhost:8001/embed/text\",\n", " \"embed_image\": \"http://localhost:8001/embed/image\",\n", " \"docs\": \"http://localhost:8001/docs\"\n", " },\n", " \"qdrant\": {\n", " \"host\": \"qdrant\",\n", " \"port\": 6333,\n", " \"grpc_port\": 6334,\n", " \"health\": \"http://localhost:6333/healthz\"\n", " }\n", "}\n", "\n", "print(\"Vision Encoder Service Configuration:\")\n", "print(\"=\"*80)\n", "print(f\"Model: {VISION_ENCODER['model']} ({VISION_ENCODER['pretrained']})\")\n", "print(f\"Embedding Dimension: {VISION_ENCODER['embedding_dim']}\")\n", "print(f\"GPU Required: {VISION_ENCODER['gpu_required']}\")\n", "print(f\"\\nEndpoints:\")\n", "for name, url in VISION_ENCODER['endpoints'].items():\n", " print(f\" {name:15} {url}\")\n", "print(f\"\\nQdrant Vector DB:\")\n", "print(f\" HTTP: http://localhost:{VISION_ENCODER['qdrant']['port']}\")\n", "print(f\" gRPC: localhost:{VISION_ENCODER['qdrant']['grpc_port']}\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Vision Encoder Testing Commands\n", "VISION_ENCODER_TESTS = {\n", " \"Health Check\": \"curl http://localhost:8001/health\",\n", " \"Model Info\": \"curl http://localhost:8001/info\",\n", " \"Text Embedding\": '''curl -X POST http://localhost:8001/embed/text -H \"Content-Type: application/json\" -d '{\"text\": \"DAARION governance\", \"normalize\": true}' ''',\n", " \"Image Embedding\": '''curl -X POST http://localhost:8001/embed/image -H \"Content-Type: application/json\" -d '{\"image_url\": \"https://example.com/image.jpg\", \"normalize\": true}' ''',\n", " \"Via Router (Text)\": '''curl -X POST http://localhost:9102/route -H \"Content-Type: application/json\" -d '{\"mode\": \"vision_embed\", \"message\": \"embed text\", \"payload\": {\"operation\": \"embed_text\", \"text\": \"test\", \"normalize\": true}}' ''',\n", " \"Qdrant Health\": \"curl http://localhost:6333/healthz\",\n", " \"Run Smoke Tests\": \"./test-vision-encoder.sh\"\n", "}\n", "\n", "print(\"Vision Encoder Testing Commands:\")\n", "print(\"=\"*80)\n", "for name, cmd in VISION_ENCODER_TESTS.items():\n", " print(f\"\\n{name}:\")\n", " print(f\" {cmd}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 📖 Documentation Links (UPDATED)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Documentation References (UPDATED)\n", "DOCS = {\n", " \"Main Guide\": \"../WARP.md\",\n", " \"Infrastructure\": \"../INFRASTRUCTURE.md\",\n", " \"Agents Map\": \"../docs/agents.md\",\n", " \"RAG Ingestion Status\": \"../RAG-INGESTION-STATUS.md\",\n", " \"HMM Memory Status\": \"../HMM-MEMORY-STATUS.md\",\n", " \"Crawl4AI Status\": \"../CRAWL4AI-STATUS.md\",\n", " \"Vision Encoder Status\": \"../VISION-ENCODER-STATUS.md\",\n", " \"Vision Encoder Deployment\": \"../services/vision-encoder/README.md\",\n", " \"Repository Management\": \"../DAARION_CITY_REPO.md\",\n", " \"Server Setup\": \"../SERVER_SETUP_INSTRUCTIONS.md\",\n", " \"Deployment\": \"../DEPLOY-NOW.md\",\n", " \"Helion Status\": \"../STATUS-HELION.md\",\n", " \"Architecture Index\": \"../docs/cursor/README.md\",\n", " \"API Reference\": \"../docs/api.md\",\n", " \"Node #2 Specs\": \"../NODE-2-MACBOOK-SPECS.md\"\n", "}\n", "\n", "print(\"Documentation Quick Links:\")\n", "print(\"=\"*80)\n", "for name, path in DOCS.items():\n", " print(f\"{name:<30} {path}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 🎤 Мультимодальні Сервіси (НОДА2)\n", "\n", "Нові сервіси для розширення можливостей агентів:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "\n", "multimodal_services = {\n", " \"STT Service\": {\n", " \"url\": \"http://192.168.1.244:8895\",\n", " \"technology\": \"OpenAI Whisper AI\",\n", " \"features\": [\"Voice→Text\", \"Ukrainian/English/Russian\", \"Telegram integration\"],\n", " \"endpoints\": [\"/api/stt\", \"/api/stt/upload\", \"/health\"],\n", " \"status\": \"✅ Ready\"\n", " },\n", " \"OCR Service\": {\n", " \"url\": \"http://192.168.1.244:8896\",\n", " \"technology\": \"Tesseract + EasyOCR\",\n", " \"features\": [\"Image→Text\", \"Bounding boxes\", \"6 languages\", \"Confidence scores\"],\n", " \"endpoints\": [\"/api/ocr\", \"/api/ocr/upload\", \"/health\"],\n", " \"status\": \"✅ Ready\"\n", " },\n", " \"Web Search\": {\n", " \"url\": \"http://192.168.1.244:8897\",\n", " \"technology\": \"DuckDuckGo + Google\",\n", " \"features\": [\"Real-time search\", \"Region-specific\", \"10+ results\"],\n", " \"endpoints\": [\"/api/search\", \"/health\"],\n", " \"status\": \"✅ Ready\"\n", " },\n", " \"Vector DB\": {\n", " \"url\": \"http://192.168.1.244:8898\",\n", " \"technology\": \"ChromaDB + Sentence Transformers\",\n", " \"features\": [\"Vector database\", \"Semantic search\", \"RAG support\"],\n", " \"endpoints\": [\"/api/collections\", \"/api/documents\", \"/api/search\", \"/health\"],\n", " \"status\": \"✅ Ready\"\n", " }\n", "}\n", "\n", "pd.DataFrame(multimodal_services).T\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "ейсу" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "vision_agents = {\n", " \"Sofia\": {\n", " \"model\": \"grok-4.1\",\n", " \"provider\": \"xAI\",\n", " \"supports_vision\": True,\n", " \"supports_files\": True,\n", " \"description\": \"Vision + Code analysis\"\n", " },\n", " \"Spectra\": {\n", " \"model\": \"qwen3-vl:latest\",\n", " \"provider\": \"Ollama\",\n", " \"supports_vision\": True,\n", " \"supports_files\": False,\n", " \"description\": \"Vision + Language\"\n", " }\n", "}\n", "\n", "pd.DataFrame(vision_agents).T\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 📊 Всі порти сервісів (оновлено)\n", "\n", "Повний список всіх сервісів з портами:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "all_ports = {\n", " \"Frontend\": {\"port\": 8899, \"node\": \"Local\", \"status\": \"✅ Active\"},\n", " \"STT Service\": {\"port\": 8895, \"node\": \"НОДА2\", \"status\": \"✅ Ready\"},\n", " \"OCR Service\": {\"port\": 8896, \"node\": \"НОДА2\", \"status\": \"✅ Ready\"},\n", " \"Web Search\": {\"port\": 8897, \"node\": \"НОДА2\", \"status\": \"✅ Ready\"},\n", " \"Vector DB\": {\"port\": 8898, \"node\": \"НОДА2\", \"status\": \"✅ Ready\"},\n", " \"Router\": {\"port\": 9102, \"node\": \"NODE1\", \"status\": \"🔄 Multimodal\"},\n", " \"Telegram Gateway\": {\"port\": 9200, \"node\": \"NODE1\", \"status\": \"🔄 Enhanced\"},\n", " \"Swapper NODE1\": {\"port\": 8890, \"node\": \"NODE1\", \"status\": \"✅ Active\"},\n", " \"Swapper NODE2\": {\"port\": 8890, \"node\": \"НОДА2\", \"status\": \"✅ Active\"},\n", " \"Agent Cabinet\": {\"port\": 8898, \"node\": \"Local\", \"status\": \"✅ Active\"},\n", " \"Memory Service\": {\"port\": 8000, \"node\": \"NODE1/2\", \"status\": \"✅ Active\"}\n", "}\n", "\n", "pd.DataFrame(all_ports).T\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 🔄 Мультимодальні можливості\n", "\n", "Статус інтеграції різних типів контенту:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "multimodal_capabilities = {\n", " \"Текст\": {\"frontend\": \"✅\", \"telegram\": \"✅\", \"status\": \"ПРАЦЮЄ\"},\n", " \"Голос→Текст\": {\"frontend\": \"✅\", \"telegram\": \"🔄\", \"status\": \"ІНТЕГРАЦІЯ\"},\n", " \"Зображення→Vision\": {\"frontend\": \"✅\", \"telegram\": \"🔄\", \"status\": \"ІНТЕГРАЦІЯ\"},\n", " \"Зображення→OCR\": {\"frontend\": \"✅\", \"telegram\": \"🔄\", \"status\": \"ІНТЕГРАЦІЯ\"},\n", " \"Документи\": {\"frontend\": \"✅\", \"telegram\": \"⚠️\", \"status\": \"ЧАСТКОВА\"},\n", " \"Веб-пошук\": {\"frontend\": \"✅\", \"telegram\": \"🔄\", \"status\": \"ІНТЕГРАЦІЯ\"},\n", " \"Knowledge Base\": {\"frontend\": \"✅\", \"telegram\": \"❌\", \"status\": \"ГОТОВИЙ\"}\n", "}\n", "\n", "pd.DataFrame(multimodal_capabilities).T\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 🔒 Security & Incident Response\n", "\n", "### Incident #1: Network Scanning & Lockdown (Dec 6, 2025 - Jan 8, 2026)\n", "\n", "**Root Cause:** Compromised `daarion-web` container with cryptocurrency miner (`catcal`, `G4NQXBp`)\n", "**Impact:** Server locked by Hetzner for 33 days due to internal network scanning\n", "**Resolution:** Container removed, firewall rules implemented, monitoring deployed\n", "\n", "### Incident #2: Recurring Compromise (Jan 9, 2026) 🔴 ACTIVE\n", "\n", "**Root Cause:** Compromised Docker image auto-restarted after server reboot \n", "**Malware:** NEW crypto miners (`softirq`, `vrarhpb`) - different from Incident #1 \n", "**Impact:** \n", "- ❌ Second abuse report (AbuseID: 10F3971:2A)\n", "- ❌ Critical CPU load: 25-35 (normal: 1-5)\n", "- ❌ 1499 zombie processes\n", "- ⚠️ Deadline: 2026-01-09 12:54 UTC (~3.5 hours remaining)\n", "\n", "**Resolution (COMPLETED):** \n", "1. ✅ Killed all malicious processes (softirq, vrarhpb)\n", "2. ✅ Stopped and removed `daarion-web` container\n", "3. ✅ **DELETED Docker images** (78e22c0ee972, 608e203fb5ac) - critical step\n", "4. ✅ Cleaned 1499 zombie processes → 5 (normal)\n", "5. ✅ System load normalized: 30+ → 4.19\n", "6. ✅ Enhanced firewall (SSH rate limiting, port scan blocking)\n", "7. ✅ Registered retry test with Hetzner\n", "8. ⏳ **PENDING:** User statement submission (URGENT)\n", "\n", "**Why Incident #2 Occurred:** \n", "- Incident #1 removed container but LEFT Docker image intact\n", "- Container had `restart: unless-stopped` in docker-compose.yml\n", "- Server rebooted → docker-compose auto-restarted from compromised image\n", "- NEW malware variant installed (different miners than Incident #1)\n", "\n", "**What is daarion-web?** \n", "- Next.js frontend (port 3000) - NOT critical for core functionality\n", "- ✅ Router, Gateway, Telegram bots, API - ALL WORKING\n", "- Status: DISABLED until secure rebuild completed\n", "\n", "**Lessons Learned (Critical):** \n", "1. 🔴 **ALWAYS delete Docker images, not just containers**\n", "2. 🟡 **Auto-restart policies are dangerous for compromised containers**\n", "3. 🟢 **Compromised images can survive container removal**\n", "4. 🔵 **Complete removal = container + image + restart policy change**\n", "\n", "**Next Steps:** \n", "1. 🔴 **URGENT:** Submit statement to Hetzner before deadline\n", "2. 🟡 Monitor server for 24 hours post-statement\n", "3. 🟢 Secure rebuild of daarion-web (see `TASK_REBUILD_DAARION_WEB.md`)\n", "4. 🔵 Security audit all remaining containers\n", "\n", "### Security Measures\n", "\n", "1. **Egress Firewall Rules** (блокування внутрішніх мереж Hetzner)\n", "2. **Monitoring Script** (`/root/monitor_scanning.sh`, runs every 15 min)\n", "3. **Security Checklist:**\n", " - [ ] Container vulnerability scanning\n", " - [ ] Docker Content Trust\n", " - [ ] Resource limits (CPU/memory)\n", " - [ ] Network segmentation\n", " - [ ] Regular security audits\n", "\n", "**Full details:** See `INFRASTRUCTURE.md` → Security & Incident Response section\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Security Configuration (UPDATED with Incident #2)\n", "security_config = {\n", " \"Firewall Rules\": {\n", " \"scripts\": [\"/root/prevent_scanning.sh\", \"/root/block_ssh_scanning.sh\"],\n", " \"status\": \"✅ Enhanced\",\n", " \"blocks\": [\"10.0.0.0/8\", \"172.16.0.0/12\"],\n", " \"allows\": [\"80/tcp\", \"443/tcp\"],\n", " \"features\": [\"SSH rate limiting\", \"Port scan blocking\", \"Enhanced logging\"]\n", " },\n", " \"Monitoring\": {\n", " \"script\": \"/root/monitor_scanning.sh\",\n", " \"status\": \"✅ Active\",\n", " \"interval\": \"15 minutes\",\n", " \"log\": \"/var/log/scan_attempts.log\"\n", " },\n", " \"Incident #1\": {\n", " \"date\": \"2025-12-06\",\n", " \"malware\": \"catcal, G4NQXBp\",\n", " \"recovery_time\": \"33 days\",\n", " \"status\": \"✅ Resolved\"\n", " },\n", " \"Incident #2\": {\n", " \"date\": \"2026-01-09\",\n", " \"malware\": \"softirq, vrarhpb\",\n", " \"mitigation_time\": \"30 minutes\",\n", " \"status\": \"⏳ Statement Pending\",\n", " \"deadline\": \"2026-01-09 12:54 UTC\",\n", " \"actions\": [\"Container removed\", \"Images DELETED\", \"Load normalized\", \"Retry test registered\"]\n", " }\n", "}\n", "\n", "import pandas as pd\n", "print(\"🔒 Security Configuration:\")\n", "print(\"=\" * 80)\n", "pd.DataFrame(security_config).T\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 📝 Notes & Updates\n", "\n", "### Recent Changes (2026-01-10)\n", "- 📝 **Session Logging System** — автоматичне логування всіх дій\n", "- 🔄 **Git Multi-Remote** — GitHub + Gitea + GitLab синхронізація\n", "- 🏗️ **NODE1 Rebuild** — чиста Ubuntu 24.04 + Docker 29.1.4\n", "- 🐳 **GitLab on NODE3** — додаткове дзеркало (порт 8929)\n", "- ✅ **Git hooks** — автологування commits/pushes\n", "- ✅ **Shell integration** — команди session-start/log/end\n", "\n", "### Recent Changes (2025-11-23)\n", "- ✅ **Swapper Service інтеграція** в кабінети НОД (тільки в `/nodes/node-1`, `/nodes/node-2`)\n", "- ✅ **Оновлення в реальному часі** (кожні 30 секунд) для Swapper Service\n", "- ✅ **Кабінети мікроДАО** з оркестраторами (DAARION, GREENFOOD, ENERGY UNION)\n", "- ✅ **Управління мікроДАО** в кабінеті DAARION (панель управління всіма мікроДАО)\n", "- ✅ **Детальні метрики Swapper Service** (моделі, спеціалісти, конфігурація)\n", "- ✅ **Frontend** (port 8899) з кабінетами НОД та мікроДАО\n", "- ✅ **Agent Cabinet Service** (port 8898) для метрик агентів\n", "\n", "### Network Architecture\n", "- **Nodes:** 3 (NODE1 production + NODE2 development + NODE3 AI/ML)\n", "- **Total Services:** 19 (додано Frontend + Agent Cabinet)\n", "- **Git Remotes:** 3 (GitHub + Gitea + GitLab)\n", "- **MicroDAO Cabinets:** 3 (DAARION, GREENFOOD, ENERGY UNION)\n", "- **Node Cabinets:** 2 (НОДА1, НОДА2)\n", "\n", "### Кабінети НОД\n", "- **НОДА1:** `http://localhost:8899/nodes/node-1`\n", "- **НОДА2:** `http://localhost:8899/nodes/node-2`\n", "- **Swapper Service:** Відображається тільки тут, оновлення кожні 30 секунд\n", "\n", "### Кабінети МікроДАО\n", "- **DAARION:** `http://localhost:8899/microdao/daarion` (оркестратор: DAARWIZZ)\n", "- **GREENFOOD:** `http://localhost:8899/microdao/greenfood` (оркестратор: GREENFOOD)\n", "- **ENERGY UNION:** `http://localhost:8899/microdao/energy-union` (оркестратор: Helion)\n", "\n", "### Git Repositories\n", "- **GitHub:** `git@github.com:IvanTytar/microdao-daarion.git` (origin)\n", "- **Gitea:** `http://localhost:3000/daarion-admin/microdao-daarion.git`\n", "- **GitLab:** `http://localhost:8929/root/microdao-daarion.git` (через SSH tunnel)\n", "\n", "---\n", "\n", "**Last Updated:** 2026-01-10 14:55 (Session Logging System + NODE1 Rebuild) \n", "**Maintained by:** Ivan Tytar & DAARION Team \n", "\n", "---\n", "\n", "### ✅ Security Status\n", "- **NODE1:** Rebuilt from scratch (Ubuntu 24.04 + Docker)\n", "- **NODE3:** Clean (verified 2026-01-09)\n", "- **Secrets:** Rotation pending — див. `SECRETS-ROTATION-CHECKLIST.md`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 🔴 Incident #4: NODE1 Host Compromise (Jan 10, 2026)\n", "\n", "### Summary\n", "ALL PostgreSQL official images show malware artifacts when run on NODE1.\n", "This is **NOT** \"Docker Hub compromised\" — this is **NODE1 host compromise**.\n", "\n", "### Indicators of Compromise (IOC)\n", "```\n", "/tmp/httpd # ~10MB crypto miner (xmrig variant)\n", "/tmp/.perf.c/ # perfctl malware staging directory\n", "/tmp/mysql # Another miner variant\n", "/tmp/cpioshuf # perfctl payload\n", "/tmp/ipcalc* # perfctl payload\n", "```\n", "\n", "### Affected Images (on NODE1)\n", "- ❌ postgres:15-alpine\n", "- ❌ postgres:16-alpine\n", "- ❌ postgres:14\n", "- ❌ postgres:16 (Debian)\n", "\n", "### Why This is HOST Compromise (not image)\n", "1. ALL different image variants show same IOC\n", "2. Previous incidents (#1, #2, #3) already compromised NODE1\n", "3. `/tmp/.perf.c/` is classic perfctl malware directory\n", "4. `tmpfs noexec` didn't prevent infection\n", "\n", "### Verification Procedure\n", "```bash\n", "# Run triage script from MacBook (NOT NODE1!)\n", "cd /Users/apple/github-projects/microdao-daarion\n", "./scripts/security/triage-postgres-compromise.sh compare\n", "\n", "# Or manually:\n", "# 1. Get digest from NODE1\n", "ssh root@144.76.224.179 \"docker inspect --format='{{index .RepoDigests 0}}' postgres:16\"\n", "\n", "# 2. Pull same digest on MacBook\n", "docker pull postgres:16@sha256:\n", "\n", "# 3. Check if clean\n", "docker run --rm postgres:16@sha256: ls -la /tmp/\n", "# If empty → NODE1 compromised, image is clean\n", "```\n", "\n", "### Current Status\n", "- ⏳ **Verification pending** — Need to test on clean host\n", "- 🔴 **NODE1 UNSAFE** — Do not deploy PostgreSQL\n", "- 🟡 **Secrets rotation needed** — Assume all compromised\n", "\n", "### Full Documentation\n", "See `INFRASTRUCTURE.md` → Incident #4" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 📝 Session Logging System\n", "\n", "### Автоматичне логування всіх дій\n", "\n", "Система автоматично записує всі дії при роботі над проєктом.\n", "\n", "### Структура логів\n", "```\n", "logs/\n", "├── README.md # Документація\n", "├── CHANGELOG.md # Головний журнал змін\n", "├── sessions/ # Щоденні логи сесій\n", "│ └── YYYY-MM-DD.md # Лог конкретного дня\n", "├── operations/ # Операційні логи\n", "└── incidents/ # Логи інцидентів\n", "```\n", "\n", "### Команди (після `source ~/.zshrc`)\n", "\n", "| Команда | Опис |\n", "|---------|------|\n", "| `session-start \"опис\"` | Почати сесію |\n", "| `session-log \"дія\"` | Додати запис |\n", "| `session-end` | Завершити (commit + push) |\n", "| `daarion-note \"нотатка\"` | Швидка нотатка |\n", "| `git-sync` | Push на всі remote |\n", "\n", "### Автоматичне логування (Git hooks)\n", "- ✅ Кожен `git commit` → записується в session log\n", "- ✅ Кожен `git push` → записується в session log\n", "\n", "### Встановлення\n", "```bash\n", "# 1. Встановити Git hooks\n", "./scripts/logging/install-hooks.sh\n", "\n", "# 2. Додати shell integration\n", "echo 'source /path/to/scripts/logging/shell-integration.sh' >> ~/.zshrc\n", "source ~/.zshrc\n", "```\n", "\n", "### Git Multi-Remote (3 дзеркала)\n", "```bash\n", "# Всі remote\n", "git remote -v\n", "# origin git@github.com:IvanTytar/microdao-daarion.git\n", "# gitea http://localhost:3000/daarion-admin/microdao-daarion.git\n", "# gitlab http://localhost:8929/root/microdao-daarion.git\n", "\n", "# Push на всі\n", "./scripts/git-sync-all.sh\n", "# або\n", "git push origin && git push gitea && git push gitlab\n", "```\n", "\n", "### SSH Tunnel до GitLab (NODE3)\n", "```bash\n", "ssh -p 33147 -L 8929:localhost:8929 -N zevs@80.77.35.151 &\n", "```" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.0" } }, "nbformat": 4, "nbformat_minor": 4 }