Files

Apple 3de3c8cb36 feat: Add presence heartbeat for Matrix online status

- matrix-gateway: POST /internal/matrix/presence/online endpoint
- usePresenceHeartbeat hook with activity tracking
- Auto away after 5 min inactivity
- Offline on page close/visibility change
- Integrated in MatrixChatRoom component

2025-11-27 00:19:40 -08:00

30 KiB

Raw Blame History

🏗️ Infrastructure Overview — DAARION & MicroDAO

Версія: 2.1.0
Останнє оновлення: 2025-11-23 18:05
Статус: Production Ready (95% Multimodal Integration)
Останні зміни:

✅ Router Multimodal API (v1.1.0) - images/files/audio/web-search
✅ Telegram Gateway Multimodal - voice/photo/documents
✅ Frontend Multimodal UI - enhanced mode
✅ Web Search Service (НОДА2)
⚠️ STT/OCR Services (НОДА2 Docker issues, fallback працює)

📍 Network Nodes

Node #1: Production Server (Hetzner GEX44 #2844465)

Node ID: node-1-hetzner-gex44
IP Address: 144.76.224.179
SSH Access: ssh root@144.76.224.179
Location: Hetzner Cloud (Germany)
Project Root: /opt/microdao-daarion
Docker Network: dagi-network
Role: Production Router + Gateway + All Services
Uptime: 24/7
Prometheus Tunnel: scripts/start-node1-prometheus-tunnel.sh (дефолт localhost:19090 → NODE1:9090, можна змінити LOCAL_PORT)

Domains:

gateway.daarion.city → 144.76.224.179 (Gateway + Nginx)
api.daarion.city → TBD (API Gateway)
daarion.city → TBD (Main website)

Node #2: Development Node (MacBook Pro M4 Max)

Node ID: node-2-macbook-m4max
Local IP: 192.168.1.33 (updated 2025-11-23)
SSH Access: ssh apple@192.168.1.244 (if enabled)
Location: Local Network (Ivan's Office)
Project Root: /Users/apple/github-projects/microdao-daarion
Role: Development + Testing + Backup Router
Specs: M4 Max (16 cores), 64GB RAM, 2TB SSD, 40-core GPU
Uptime: On-demand (battery-powered)

See full specs: NODE-2-MACBOOK-SPECS.md
Current state: NODE-2-CURRENT-STATE.md — What's running now

🐙 GitHub Repositories

1. MicroDAO (Current Project)

Repository: git@github.com:IvanTytar/microdao-daarion.git
HTTPS: https://github.com/IvanTytar/microdao-daarion.git
Remote Name: origin
Main Branch: main
Purpose: MicroDAO core code, DAGI Stack, documentation

Quick Clone:

git clone git@github.com:IvanTytar/microdao-daarion.git
cd microdao-daarion

2. DAARION.city

Repository: git@github.com:DAARION-DAO/daarion-ai-city.git
HTTPS: https://github.com/DAARION-DAO/daarion-ai-city.git
Remote Name: daarion-city
Main Branch: main
Purpose: Official DAARION.city website and integrations

Quick Clone:

git clone git@github.com:DAARION-DAO/daarion-ai-city.git
cd daarion-ai-city

Add as remote to MicroDAO:

cd microdao-daarion
git remote add daarion-city git@github.com:DAARION-DAO/daarion-ai-city.git
git fetch daarion-city

🚀 Services & Ports (Docker Compose)

Core Services

Service	Port	Container Name	Health Endpoint
DAGI Router	9102	`dagi-router`	`http://localhost:9102/health`
Bot Gateway	9300	`dagi-gateway`	`http://localhost:9300/health`
DevTools Backend	8008	`dagi-devtools`	`http://localhost:8008/health`
CrewAI Orchestrator	9010	`dagi-crewai`	`http://localhost:9010/health`
RBAC Service	9200	`dagi-rbac`	`http://localhost:9200/health`
RAG Service	9500	`dagi-rag-service`	`http://localhost:9500/health`
Memory Service	8000	`dagi-memory-service`	`http://localhost:8000/health`
Parser Service	9400	`dagi-parser-service`	`http://localhost:9400/health`
Swapper Service	8890-8891	`swapper-service`	`http://localhost:8890/health`
Frontend (Vite)	8899	`frontend`	`http://localhost:8899`
Agent Cabinet Service	8898	`agent-cabinet-service`	`http://localhost:8898/health`
PostgreSQL	5432	`dagi-postgres`	-
Redis	6379	`redis`	`redis-cli PING`
Neo4j	7687 (bolt), 7474 (http)	`neo4j`	`http://localhost:7474`
Qdrant	6333 (http), 6334 (grpc)	`dagi-qdrant`	`http://localhost:6333/healthz`
Grafana	3000	`grafana`	`http://localhost:3000`
Prometheus	9090	`prometheus`	`http://localhost:9090`
Neo4j Exporter	9091	`neo4j-exporter`	`http://localhost:9091/metrics`
Ollama	11434	`ollama` (external)	`http://localhost:11434/api/tags`

Multimodal Services (НОДА2)

Service	Port	Container Name	Health Endpoint
STT Service	8895	`stt-service`	`http://192.168.1.244:8895/health`
OCR Service	8896	`ocr-service`	`http://192.168.1.244:8896/health`
Web Search	8897	`web-search-service`	`http://192.168.1.244:8897/health`
Vector DB	8898	`vector-db-service`	`http://192.168.1.244:8898/health`

Note: Vision Encoder (port 8001) не запущений на Node #1. Замість нього використовується Swapper Service з vision-8b моделлю (Qwen3-VL 8B) для обробки зображень через динамічне завантаження моделей.

Swapper Service:

Порт: 8890 (HTTP), 8891 (Prometheus metrics)
URL НОДА1: http://144.76.224.179:8890
URL НОДА2: http://192.168.1.244:8890
Відображення: Тільки в кабінетах НОД (/nodes/node-1, /nodes/node-2)
Оновлення: В реальному часі (кожні 30 секунд)
Моделі: 5 моделей (qwen3:8b, qwen3-vl:8b, qwen2.5:7b-instruct, qwen2.5:3b-instruct, qwen2-math:7b)
Спеціалісти: 6 спеціалістів (vision-8b, math-7b, structured-fc-3b, rag-mini-4b, lang-gateway-4b, security-guard-7b)

HTTPS Gateway (Nginx)

Port: 443 (HTTPS), 80 (HTTP redirect)
Domain: gateway.daarion.city
SSL: Let's Encrypt (auto-renewal)
Proxy Pass:
- /telegram/webhook → http://localhost:9300/telegram/webhook
- /helion/telegram/webhook → http://localhost:9300/helion/telegram/webhook

🤖 Telegram Bots

1. DAARWIZZ Bot

Username: @DAARWIZZBot
Bot ID: 8323412397
Token: 8323412397:AAFxaru-hHRl08A3T6TC02uHLvO5wAB0m3M ✅
Webhook: https://gateway.daarion.city/telegram/webhook
Status: Active (Production)

2. Helion Bot (Energy Union AI)

Username: @HelionEnergyBot (example)
Bot ID: 8112062582
Token: 8112062582:AAGI7tPFo4gvZ6bfbkFu9miq5GdAH2_LvcM ✅
Webhook: https://gateway.daarion.city/helion/telegram/webhook
Status: Ready for deployment

🔐 Environment Variables (.env)

Essential Variables

# Bot Gateway
TELEGRAM_BOT_TOKEN=8323412397:AAFxaru-hHRl08A3T6TC02uHLvO5wAB0m3M
HELION_TELEGRAM_BOT_TOKEN=8112062582:AAGI7tPFo4gvZ6bfbkFu9miq5GdAH2_LvcM
GATEWAY_PORT=9300

# DAGI Router
ROUTER_PORT=9102
ROUTER_CONFIG_PATH=./router-config.yml

# Ollama (Local LLM)
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_MODEL=qwen3:8b

# Memory Service
MEMORY_SERVICE_URL=http://memory-service:8000
MEMORY_DATABASE_URL=postgresql://postgres:postgres@postgres:5432/daarion_memory

# PostgreSQL
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_DB=daarion_memory

# RBAC
RBAC_PORT=9200
RBAC_DATABASE_URL=sqlite:///./rbac.db

# Vision Encoder (GPU required for production)
VISION_ENCODER_URL=http://vision-encoder:8001
VISION_DEVICE=cuda
VISION_MODEL_NAME=ViT-L-14
VISION_MODEL_PRETRAINED=openai

# Qdrant Vector Database
QDRANT_HOST=qdrant
QDRANT_PORT=6333
QDRANT_ENABLED=true

# CORS
CORS_ORIGINS=http://localhost:3000,https://daarion.city

# Environment
ENVIRONMENT=production
DEBUG=false
LOG_LEVEL=INFO

🌌 SPACE API (planets, nodes, events)

Сервіс: space-service (FastAPI / Node.js)
Порти: 7001 (FastAPI), 3005 (Node.js)

GET /space/planets

Повертає DAO-планети (health, treasury, satellites, anomaly score, position).

Response Example:

[
  {
    "dao_id": "dao:3",
    "name": "Aurora Circle",
    "health": "good",
    "treasury": 513200,
    "activity": 0.84,
    "governance_temperature": 72,
    "anomaly_score": 0.04,
    "position": { "x": 120, "y": 40, "z": -300 },
    "node_count": 12,
    "satellites": [
      {
        "node_id": "node:03",
        "gpu_load": 0.66,
        "latency": 14,
        "agents": 22
      }
    ]
  }
]

GET /space/nodes

Повертає стан кожної ноди (GPU, CPU, memory, network, agents, status).

Response Example:

[
  {
    "node_id": "node:03",
    "name": "Quantum Relay",
    "microdao": "microdao:7",
    "gpu": {
      "load": 0.72,
      "vram_used": 30.1,
      "vram_total": 40.0,
      "temperature": 71
    },
    "cpu": {
      "load": 0.44,
      "temperature": 62
    },
    "memory": {
      "used": 11.2,
      "total": 32.0
    },
    "network": {
      "latency": 12,
      "bandwidth_in": 540,
      "bandwidth_out": 430,
      "packet_loss": 0.01
    },
    "agents": 14,
    "status": "healthy"
  }
]

GET /space/events

Поточні DAO/Space події (governance, treasury, anomalies, node alerts).

Query Parameters:

seconds (optional): Time window in seconds (default: 120)

Response Example:

[
  {
    "type": "dao.vote.opened",
    "dao_id": "dao:3",
    "timestamp": 1735680041,
    "severity": "info",
    "meta": {
      "proposal_id": "P-173",
      "title": "Budget Allocation 2025"
    }
  },
  {
    "type": "node.alert.overload",
    "node_id": "node:05",
    "timestamp": 1735680024,
    "severity": "warn",
    "meta": {
      "gpu_load": 0.92
    }
  }
]

Джерела даних:

Дані	Джерело	Компонент
DAO	microDAO Service / DAO-Service	PostgreSQL
Ноди	NodeMetrics Agent → NATS → Metrics Collector	Redis / Timescale
Агенти	Router → Agent Registry	Redis / SQLite
Події	NATS JetStream	JetStream Stream `events.space`

Frontend Integration:

API клієнти: src/api/space/getPlanets.ts, src/api/space/getNodes.ts, src/api/space/getSpaceEvents.ts
Використання: City Dashboard, Space Dashboard, Living Map, World Prototype

📦 Deployment Workflow

1. Local Development → GitHub

# On Mac (local)
cd /Users/apple/github-projects/microdao-daarion
git add .
git commit -m "feat: description"
git push origin main

2. GitHub → Production Server

# SSH to server
ssh root@144.76.224.179

# Navigate to project
cd /opt/microdao-daarion

# Pull latest changes
git pull origin main

# Restart services
docker-compose down
docker-compose up -d --build

# Check status
docker-compose ps
docker-compose logs -f gateway

3. HTTPS Gateway Setup

# On server (one-time setup)
sudo ./scripts/setup-nginx-gateway.sh gateway.daarion.city admin@daarion.city

4. Register Telegram Webhook

# On server
./scripts/register-agent-webhook.sh daarwizz 8323412397:AAFxaru-hHRl08A3T6TC02uHLvO5wAB0m3M gateway.daarion.city
./scripts/register-agent-webhook.sh helion 8112062582:AAGI7tPFo4gvZ6bfbkFu9miq5GdAH2_LvcM gateway.daarion.city

🧪 Testing & Monitoring

Health Checks (All Services)

# On server
curl http://localhost:9102/health  # Router
curl http://localhost:9300/health  # Gateway
curl http://localhost:8000/health  # Memory
curl http://localhost:9200/health  # RBAC
curl http://localhost:9500/health  # RAG
curl http://localhost:8001/health  # Vision Encoder
curl http://localhost:6333/healthz # Qdrant

# Public HTTPS
curl https://gateway.daarion.city/health

Smoke Tests

# On server
cd /opt/microdao-daarion
./smoke.sh

View Logs

# All services
docker-compose logs -f

# Specific service
docker-compose logs -f gateway
docker-compose logs -f router
docker-compose logs -f memory-service

# Filter by error level
docker-compose logs gateway | grep ERROR

Database Check

# PostgreSQL
docker exec -it dagi-postgres psql -U postgres -c "\l"
docker exec -it dagi-postgres psql -U postgres -d daarion_memory -c "\dt"

🌐 DNS Configuration

Current DNS Records (Cloudflare/Hetzner)

Record Type	Name	Value	TTL
A	`gateway.daarion.city`	`144.76.224.179`	300
A	`daarion.city`	TBD	300
A	`api.daarion.city`	TBD	300

Verify DNS:

dig gateway.daarion.city +short
# Should return: 144.76.224.179

📂 Key File Locations

On Server (`/opt/microdao-daarion`)

Docker Compose: docker-compose.yml
Environment: .env (never commit!)
Router Config: router-config.yml
Nginx Setup: scripts/setup-nginx-gateway.sh
Webhook Register: scripts/register-agent-webhook.sh
Logs: logs/ directory
Data: data/ directory

System Prompts

DAARWIZZ: gateway-bot/daarwizz_prompt.txt
Helion: gateway-bot/helion_prompt.txt

Documentation

Quick Start: WARP.md
Agents Map: docs/agents.md
RAG Ingestion: RAG-INGESTION-STATUS.md
HMM Memory: HMM-MEMORY-STATUS.md
Crawl4AI Service: CRAWL4AI-STATUS.md
Architecture: docs/cursor/README.md
API Reference: docs/api.md

🔄 Backup & Restore

Backup Database

# PostgreSQL dump
docker exec dagi-postgres pg_dump -U postgres daarion_memory > backup_$(date +%Y%m%d).sql

# RBAC SQLite
cp data/rbac/rbac.db backups/rbac_$(date +%Y%m%d).db

Restore Database

# PostgreSQL restore
cat backup_20250117.sql | docker exec -i dagi-postgres psql -U postgres daarion_memory

# RBAC restore
cp backups/rbac_20250117.db data/rbac/rbac.db
docker-compose restart rbac

📞 Contacts & Support

Team

Owner: Ivan Tytar
Email: admin@daarion.city
GitHub: @IvanTytar

External Services

Hetzner Support: https://www.hetzner.com/support
Cloudflare Support: https://dash.cloudflare.com
Telegram Bot Support: https://core.telegram.org/bots

🔗 Quick Reference Links

Documentation

WARP.md — Main developer guide
SYSTEM-INVENTORY.md — Complete system inventory (GPU, AI models, 17 services)
DAARION_CITY_REPO.md — Repository management
RAG-INGESTION-STATUS.md — RAG event-driven ingestion (Wave 1, 2, 3)
HMM-MEMORY-STATUS.md — Hierarchical Memory System for agents
CRAWL4AI-STATUS.md — Web crawler for document ingestion (PDF, Images, HTML)
VISION-ENCODER-STATUS.md — Vision Encoder service status (OpenCLIP multimodal embeddings)
VISION-RAG-IMPLEMENTATION.md — Vision RAG complete implementation (client, image search, routing)
services/vision-encoder/README.md — Vision Encoder deployment guide
SERVER_SETUP_INSTRUCTIONS.md — Server setup
DEPLOY-NOW.md — Deployment checklist
STATUS-HELION.md — Helion agent status

Monitoring Dashboards

Gateway Health: https://gateway.daarion.city/health
Router Providers: http://localhost:9102/providers
Routing Table: http://localhost:9102/routing
Prometheus: http://localhost:9090 (Metrics, Alerts, Targets)
Grafana Dashboard: http://localhost:3000 (Neo4j metrics, DAO/Agents/Users analytics)
Neo4j Browser: http://localhost:7474 (Graph visualization, Cypher queries)
Neo4j Exporter: http://localhost:9091/metrics (Prometheus metrics endpoint)

🚨 Troubleshooting

Service Not Starting

# Check logs
docker-compose logs service-name

# Restart service
docker-compose restart service-name

# Rebuild and restart
docker-compose up -d --build service-name

Database Connection Issues

# Check PostgreSQL
docker exec -it dagi-postgres psql -U postgres -c "SELECT 1"

# Restart PostgreSQL
docker-compose restart postgres

# Check connection from memory service
docker exec -it dagi-memory-service env | grep DATABASE

Webhook Not Working

# Check webhook status
curl "https://api.telegram.org/bot<TOKEN>/getWebhookInfo"

# Re-register webhook
./scripts/register-agent-webhook.sh <agent> <token> <domain>

# Check gateway logs
docker-compose logs -f gateway | grep webhook

SSL Certificate Issues

# Check certificate
sudo certbot certificates

# Renew certificate
sudo certbot renew --dry-run
sudo certbot renew

# Restart Nginx
sudo systemctl restart nginx

📊 Metrics & Analytics (Future)

Planned Monitoring Stack

Prometheus: Metrics collection
Grafana: Dashboards
Loki: Log aggregation
Alertmanager: Alerts

Port Reservations:

Prometheus: 9090
Grafana: 3000
Loki: 3100

🖥️ Кабінети НОД та МікроДАО

Кабінети НОД

НОДА1: http://localhost:8899/nodes/node-1
НОДА2: http://localhost:8899/nodes/node-2

Функціонал:

Огляд (метрики, статус, GPU)
Агенти (список, деплой, управління)
Сервіси (Swapper Service з детальними метриками, інші сервіси)
Метрики (CPU, RAM, Disk, Network)
Плагіни (встановлені та доступні)
Інвентаризація (повна інформація про встановлене ПЗ)

Swapper Service в кабінетах НОД:

Статус сервісу (CPU, RAM, VRAM, Uptime)
Конфігурація (режим, max concurrent, memory buffer, eviction)
Моделі (таблиця з усіма моделями, статусом, uptime, запитами)
Спеціалісти (6 спеціалістів з інформацією про моделі та використання)
Активна модель (якщо є)
Оновлення в реальному часі (кожні 30 секунд)

Кабінети МікроДАО

DAARION: http://localhost:8899/microdao/daarion
GREENFOOD: http://localhost:8899/microdao/greenfood
ENERGY UNION: http://localhost:8899/microdao/energy-union

Функціонал:

Огляд (чат з оркестратором, статистика)
Агенти (список агентів, оркестратор з НОДИ1)
Канали (список каналів)
Проєкти (майбутнє)
Управління мікроДАО (тільки для DAARION - панель управління всіма мікроДАО)
DAARION Core (тільки для DAARION)
Налаштування

Оркестратори:

DAARION → DAARWIZZ (agent-daarwizz)
GREENFOOD → GREENFOOD Assistant (agent-greenfood-assistant)
ENERGY UNION → Helion (agent-helion)

🎤 Multimodal Services Details (НОДА2)

STT Service — Speech-to-Text

URL: http://192.168.1.244:8895
Technology: OpenAI Whisper AI (base model)
Functions:
- Voice → Text transcription
- Ukrainian, English, Russian support
- Auto-transcription for Telegram bots
Endpoints:
- POST /api/stt — Transcribe base64 audio
- POST /api/stt/upload — Upload audio file
- GET /health — Health check
Status: ✅ Ready for Integration

OCR Service — Text Extraction

URL: http://192.168.1.244:8896
Technology: Tesseract + EasyOCR
Functions:
- Image → Text extraction
- Bounding boxes detection
- Multi-language support (uk, en, ru, pl, de, fr)
- Confidence scores
Endpoints:
- POST /api/ocr — Extract text from base64 image
- POST /api/ocr/upload — Upload image file
- GET /health — Health check
Status: ✅ Ready for Integration

Web Search Service

URL: http://192.168.1.244:8897
Technology: DuckDuckGo + Google Search
Functions:
- Real-time web search
- Region-specific search (ua-uk, us-en)
- JSON structured results
- Up to 10+ results per query
Endpoints:
- POST /api/search — Search with JSON body
- GET /api/search?query=... — Search with query params
- GET /health — Health check
Status: ✅ Ready for Integration

Vector DB Service — Knowledge Base

URL: http://192.168.1.244:8898
Technology: ChromaDB + Sentence Transformers
Functions:
- Vector database for documents
- Semantic search
- Document embeddings (all-MiniLM-L6-v2)
- RAG (Retrieval-Augmented Generation) support
Endpoints:
- POST /api/collections — Create collection
- GET /api/collections — List collections
- POST /api/documents — Add documents
- POST /api/search — Semantic search
- DELETE /api/documents — Delete documents
- GET /health — Health check
Status: ✅ Ready for Integration

🔄 Router Multimodal Support (NODE1)

Enhanced /route endpoint

URL: http://144.76.224.179:9102/route
New Payload Structure:

{
  "agent": "sofia",
  "message": "Analyze this image",
  "mode": "chat",
  "payload": {
    "context": {
      "system_prompt": "...",
      "images": ["data:image/png;base64,..."],
      "files": [{"name": "doc.pdf", "data": "..."}],
      "audio": "data:audio/webm;base64,..."
    }
  }
}

Vision Agents

Sofia (grok-4.1, xAI) — Vision + Code + Files
Spectra (qwen3-vl:latest, Ollama) — Vision + Language

Features:

📷 Image processing (PIL)
📎 File processing (PDF, TXT, MD)
🎤 Audio transcription (via STT Service)
🌐 Web search integration
📚 Knowledge Base / RAG

Status: 🔄 Integration in Progress

📱 Telegram Gateway Multimodal Updates

Enhanced Features:

🎤 Voice Messages → Auto-transcription via STT Service
📷 Photos → Vision analysis via Sofia/Spectra
📎 Documents → Text extraction via OCR/Parser
🌐 Web Search → Real-time search results

Workflow:

Telegram Bot → Voice/Photo/File
    ↓
Gateway → STT/OCR/Parser Service
    ↓
Router → Vision/LLM Agent
    ↓
Response → Telegram Bot

Status: 🔄 Integration in Progress

📊 All Services Port Summary

Service	Port	Node	Technology	Status
Frontend	8899	Local	React + Vite	✅
STT Service	8895	НОДА2	Whisper AI	✅ Ready
OCR Service	8896	НОДА2	Tesseract + EasyOCR	✅ Ready
Web Search	8897	НОДА2	DuckDuckGo + Google	✅ Ready
Vector DB	8898	НОДА2	ChromaDB	✅ Ready
Router	9102	NODE1	FastAPI + Ollama	🔄 Multimodal
Telegram Gateway	9200	NODE1	FastAPI + NATS	🔄 Enhanced
Swapper NODE1	8890	NODE1	LLM Manager	✅
Swapper NODE2	8890	НОДА2	LLM Manager	✅

Last Updated: 2025-11-23 by Auto AI
Maintained by: Ivan Tytar & DAARION Team
Status: ✅ Production Ready (🔄 Multimodal Integration in Progress)

🎨 Multimodal Integration (v2.1.0)

Router Multimodal API (NODE1)

Version: 1.1.0-multimodal
Endpoint: http://144.76.224.179:9102/route

Features:

{
  "features": [
    "multimodal",
    "vision",
    "stt",
    "ocr",
    "web-search"
  ]
}

Request Format:

{
  "agent": "daarwizz",
  "message": "User message",
  "mode": "chat",
  "images": ["data:image/jpeg;base64,..."],
  "files": [{"name": "doc.pdf", "content": "base64...", "type": "application/pdf"}],
  "audio": "base64_encoded_audio",
  "web_search_query": "search query",
  "language": "uk"
}

Vision Agents:

sofia - Sofia Vision Agent (qwen3-vl:8b)
spectra - Spectra Vision Agent (qwen3-vl:8b)

Обробка:

Vision agents → images передаються напряму
Звичайні agents → images конвертуються через OCR
Audio → транскрибується через STT
Files → текст витягується (PDF, TXT, MD)

Telegram Gateway Multimodal (NODE1)

Location: /opt/microdao-daarion/gateway-bot/
Handlers: gateway_multimodal_handlers.py

Supported Content Types:

🎤 Voice messages → STT → Router
📸 Photos → Vision/OCR → Router
📎 Documents → Text extraction → Router

Example Flow:

1. User sends voice to @DAARWIZZBot
2. Gateway downloads from Telegram
3. Gateway sends base64 audio to Router
4. Router transcribes via STT (or fallback)
5. Router processes with agent LLM
6. Gateway sends response back to Telegram

Telegram Bot Tokens (реальні з BOT_CONFIGS):

CLAN: $CLAN_TELEGRAM_BOT_TOKEN (@CLAN_bot)
DAARWIZZ: $DAARWIZZ_TELEGRAM_BOT_TOKEN (@DAARWIZZBot)
DRUID: $DRUID_TELEGRAM_BOT_TOKEN (@DRUIDBot)
EONARCH: $EONARCH_TELEGRAM_BOT_TOKEN (@EONARCHBot)
GREENFOOD: $GREENFOOD_TELEGRAM_BOT_TOKEN (@GREENFOODBot) - має CrewAI команду
Helion: $HELION_TELEGRAM_BOT_TOKEN (@HelionBot)
NUTRA: $NUTRA_TELEGRAM_BOT_TOKEN (@NUTRABot)
Soul: $SOUL_TELEGRAM_BOT_TOKEN (@SoulBot)
Yaromir: $YAROMIR_TELEGRAM_BOT_TOKEN (@YaromirBot) - CrewAI Orchestrator

ВСЬОГО: 9 Telegram ботів (перевірено в BOT_CONFIGS)

Webhook Pattern: https://gateway.daarion.city/{bot_id}/telegram/webhook

Multimodal Support:

✅ Всі 9 ботів підтримують voice/photo/document через universal webhook

CrewAI команди (внутрішні агенти, БЕЗ Telegram ботів):

Yaromir (Orchestrator) → делегує:
- Вождь (Strategic, qwen2.5:14b)
- Проводник (Mentor, qwen2.5:7b)
- Домір (Harmony, qwen2.5:3b)
- Создатель (Innovation, qwen2.5:14b)
GREENFOOD (Orchestrator) → має свою CrewAI команду

Примітка: Вождь, Проводник, Домір, Создатель мають промпти (*_prompt.txt) але НЕ мають Telegram токенів. Вони працюють тільки всередині CrewAI workflow.

Frontend Multimodal UI

Location: src/components/microdao/

Components:

MicroDaoOrchestratorChatEnhanced.tsx - Enhanced chat with multimodal
MultimodalInput.tsx - Input component (images/files/voice/web-search)

Features:

✅ Switch toggle для розширеного режиму
✅ Image upload (drag & drop, click)
✅ File upload (PDF, TXT, MD)
✅ Voice recording (Web Audio API)
✅ Web search integration
✅ Real-time preview

Usage:

Open http://localhost:8899/microdao/daarion
Enable "Розширений режим" (switch)
Upload images, files, or record voice
Send to agent

НОДА2 Multimodal Services

Location: MacBook M4 Max (192.168.1.33)

Service	Port	Status	Notes
STT (Whisper)	8895	⚠️ Docker issue	Fallback працює
OCR (Tesseract/EasyOCR)	8896	⚠️ Docker issue	Fallback працює
Web Search	8897	✅ HEALTHY	DuckDuckGo + Google
Vector DB (ChromaDB)	8898	✅ HEALTHY	RAG ready

Fallback Mechanism:

Router має fallback логіку для недоступних сервісів
Якщо STT недоступний → повертається помилка (graceful)
Якщо OCR недоступний → fallback на базовий text extraction

Testing Multimodal

1. Router API

# Health check
curl http://144.76.224.179:9102/health

# Basic text
curl -X POST http://144.76.224.179:9102/route \
  -H 'Content-Type: application/json' \
  -d '{"agent":"daarwizz","message":"Привіт","mode":"chat"}'

# With image (Vision)
curl -X POST http://144.76.224.179:9102/route \
  -H 'Content-Type: application/json' \
  -d '{
    "agent":"sofia",
    "message":"Опиши це зображення",
    "images":["data:image/jpeg;base64,/9j/4AAQ..."],
    "mode":"chat"
  }'

2. Telegram Bots (9 реальних ботів)

Всі боти (з BOT_CONFIGS):

@CLAN_bot, @DAARWIZZBot, @DRUIDBot, @EONARCHBot,
@GREENFOODBot, @HelionBot, @NUTRABot, @SoulBot, @YaromirBot

Тести:

Send voice message: "Привіт, як справи?"
Send photo with caption: "Що на цьому фото?"
Send document: "Проаналізуй цей документ"

CrewAI Workflow (через @YaromirBot):

User → @YaromirBot (Telegram)
         ↓
    Yaromir Orchestrator
         ↓ (CrewAI delegation)
    ┌────┴────┬────────┬─────────┐
    ↓         ↓        ↓         ↓
  Вождь   Проводник  Домир   Создатель
(Internal CrewAI agents - NO Telegram bots)
         ↓
    Yaromir → Response → Telegram

Примітка: Вождь, Проводник, Домір, Создатель НЕ є окремими Telegram ботами. Вони працюють тільки всередині CrewAI коли Yaromir делегує завдання.

3. Frontend

1. Open http://localhost:8899/microdao/daarion
2. Enable "Розширений режим"
3. Upload image
4. Upload file
5. Record voice

Implementation Files

Router (NODE1):

/app/multimodal/handlers.py - Multimodal обробники
/app/http_api.py - Updated with multimodal support

Gateway (NODE1):

/opt/microdao-daarion/gateway-bot/gateway_multimodal_handlers.py
/opt/microdao-daarion/gateway-bot/http_api.py (updated)

Frontend:

src/pages/MicroDaoCabinetPage.tsx
src/components/microdao/MicroDaoOrchestratorChatEnhanced.tsx
src/components/microdao/chat/MultimodalInput.tsx

НОДА2 Services:

services/stt-service/
services/ocr-service/
services/web-search-service/
services/vector-db-service/

Documentation

Created Files:

/tmp/MULTIMODAL-INTEGRATION-FINAL-REPORT.md
/tmp/TELEGRAM-GATEWAY-MULTIMODAL-INTEGRATION.md
/tmp/MULTIMODAL-INTEGRATION-SUCCESS.md
/tmp/COMPLETE-MULTIMODAL-ECOSYSTEM.md
ROUTER-MULTIMODAL-SUPPORT.md

Time Invested: ~6.5 hours
Status: 95% Complete
Production Ready: ✅ Yes (with fallbacks)

30 KiB Raw Blame History Unescape Escape

🏗️ Infrastructure Overview — DAARION & MicroDAO

📍 Network Nodes

Node #1: Production Server (Hetzner GEX44 #2844465)

Node #2: Development Node (MacBook Pro M4 Max)

🐙 GitHub Repositories

1. MicroDAO (Current Project)

2. DAARION.city

🚀 Services & Ports (Docker Compose)

Core Services

Multimodal Services (НОДА2)

HTTPS Gateway (Nginx)

🤖 Telegram Bots

1. DAARWIZZ Bot

2. Helion Bot (Energy Union AI)

🔐 Environment Variables (.env)

Essential Variables

🌌 SPACE API (planets, nodes, events)

GET /space/planets

GET /space/nodes

GET /space/events

Джерела даних:

📦 Deployment Workflow

1. Local Development → GitHub

2. GitHub → Production Server

3. HTTPS Gateway Setup

4. Register Telegram Webhook

🧪 Testing & Monitoring

Health Checks (All Services)

Smoke Tests

View Logs

Database Check

🌐 DNS Configuration

Current DNS Records (Cloudflare/Hetzner)

📂 Key File Locations

On Server (/opt/microdao-daarion)

System Prompts

Documentation

🔄 Backup & Restore

Backup Database

Restore Database

📞 Contacts & Support

Team

External Services

🔗 Quick Reference Links

Documentation

Monitoring Dashboards

🚨 Troubleshooting

Service Not Starting

Database Connection Issues

Webhook Not Working

SSL Certificate Issues

📊 Metrics & Analytics (Future)

Planned Monitoring Stack

🖥️ Кабінети НОД та МікроДАО

Кабінети НОД

Кабінети МікроДАО

🎤 Multimodal Services Details (НОДА2)

STT Service — Speech-to-Text

OCR Service — Text Extraction

Web Search Service

Vector DB Service — Knowledge Base

🔄 Router Multimodal Support (NODE1)

Enhanced /route endpoint

Vision Agents

Features:

📱 Telegram Gateway Multimodal Updates

Enhanced Features:

Workflow:

📊 All Services Port Summary

🎨 Multimodal Integration (v2.1.0)

Router Multimodal API (NODE1)

Telegram Gateway Multimodal (NODE1)

Frontend Multimodal UI

НОДА2 Multimodal Services

Testing Multimodal

1. Router API

2. Telegram Bots (9 реальних ботів)

3. Frontend

Implementation Files

30 KiB

Raw Blame History

On Server (`/opt/microdao-daarion`)