Apple 2a353040f6 feat: add tests and integrate dots.ocr model
G.2.5 - Tests:
- Add pytest test suite with fixtures
- test_preprocessing.py - PDF/image loading, normalization, validation
- test_postprocessing.py - chunks, QA pairs, markdown generation
- test_inference.py - dummy parser and inference functions
- test_api.py - API endpoint tests
- Add pytest.ini configuration

G.1.3 - dots.ocr Integration:
- Update model_loader.py with real model loading code
  - Support for AutoModelForVision2Seq and AutoProcessor
  - Device handling (CUDA/CPU/MPS) with fallback
  - Error handling with dummy fallback option
- Update inference.py with real model inference
  - Process images through model
  - Generate and decode outputs
  - Parse model output to blocks
- Add model_output_parser.py
  - Parse JSON or plain text model output
  - Convert to structured blocks
  - Layout detection support (placeholder)

Dependencies:
- Add pytest, pytest-asyncio, httpx for testing
2025-11-15 13:25:01 -08:00

DAGI Stack

Decentralized Agentic Gateway Infrastructure

Production-ready AI router with multi-agent orchestration, microDAO governance, and bot gateway integration.

License: MIT Docker Python 3.11+


🏗️ Architecture

┌─────────────────────────────────────────────────────────────┐
│                        Bot Gateway                          │
│                    (Telegram/Discord)                       │
│                       Port: 9300                            │
└──────────────────────┬──────────────────────────────────────┘
                       │
                       ↓
┌─────────────────────────────────────────────────────────────┐
│                     DAGI Router                             │
│              (Dynamic Rule-Based Routing)                   │
│                       Port: 9102                            │
└───┬──────────────────┬──────────────────┬──────────────────┘
    │                  │                  │
    ↓                  ↓                  ↓
┌─────────┐    ┌──────────────┐    ┌──────────────┐
│   LLM   │    │   DevTools   │    │   CrewAI     │
│Provider │    │   Backend    │    │ Orchestrator │
│         │    │  Port: 8008  │    │  Port: 9010  │
└─────────┘    └──────────────┘    └──────────────┘
    ↑
    │ RBAC Context Injection
    │
┌─────────────────────────────────────────────────────────────┐
│                      RBAC Service                           │
│              (Role-Based Access Control)                    │
│                       Port: 9200                            │
└─────────────────────────────────────────────────────────────┘

Core Components

Component Port Description
DAGI Router 9102 Main routing engine with rule-based dispatch
Bot Gateway 9300 Telegram/Discord webhook receiver
DevTools Backend 8008 File operations, test execution, notebooks
CrewAI Orchestrator 9010 Multi-agent workflow execution
RBAC Service 9200 Role resolution and access control
Ollama 11434 Local LLM (optional)

🚀 Quick Start

Prerequisites

  • Docker 20.10+
  • Docker Compose 2.0+
  • 4GB+ RAM
  • 10GB+ disk space

1. Clone Repository

git clone https://github.com/daarion/dagi-stack.git
cd dagi-stack

2. Configure Environment

cp .env.example .env
# Edit .env with your tokens and settings
nano .env

Required variables:

  • TELEGRAM_BOT_TOKEN - Get from @BotFather
  • OLLAMA_BASE_URL - Local Ollama URL (or use remote LLM)

3. Start Services

docker-compose up -d

4. Verify Health

./smoke.sh

Or manually:

curl http://localhost:9102/health  # Router
curl http://localhost:8008/health  # DevTools
curl http://localhost:9010/health  # CrewAI
curl http://localhost:9200/health  # RBAC
curl http://localhost:9300/health  # Gateway

5. Test Basic Routing

curl -X POST http://localhost:9102/route \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Hello from DAGI!",
    "mode": "chat",
    "metadata": {}
  }'

📦 Services Overview

DAGI Router

Central routing engine that dispatches requests based on configurable rules:

  • Rule-based routing: Priority-ordered rules match requests to providers
  • Multi-provider support: Ollama, DeepSeek, OpenAI, custom agents
  • Metadata enrichment: Injects context (dao_id, user_id, RBAC roles)
  • Timeout handling: Configurable timeouts with fallback strategies

Key files:

  • main_v2.py - Entry point
  • routing_engine.py - Core routing logic
  • router-config.yml - Routing rules configuration

Bot Gateway

HTTP server for bot platforms:

  • Telegram webhooks: /telegram/webhook
  • Discord webhooks: /discord/webhook
  • Chat normalization: Converts platform-specific messages to unified format
  • RBAC integration: Enriches requests with user roles before routing

Key files:

  • gateway-bot/main.py
  • gateway-bot/http_api.py
  • gateway-bot/router_client.py

DevTools Backend

Tool execution service for development tasks:

  • File operations: Read/write files in workspace
  • Test execution: Run pytest/jest/etc
  • Notebook execution: Jupyter notebook support
  • Security: Path validation, size limits

Endpoints:

  • POST /fs/read - Read file
  • POST /fs/write - Write file
  • POST /ci/run-tests - Execute tests
  • POST /notebook/execute - Run notebook

CrewAI Orchestrator

Multi-agent workflow execution:

  • 4 workflows:
    • microdao_onboarding - Welcome new members
    • code_review - Code quality analysis
    • proposal_review - Governance proposal assessment
    • task_decomposition - Break down complex tasks

Endpoints:

  • POST /workflow/run - Execute workflow
  • GET /workflow/list - List available workflows

RBAC Service

Role-based access control:

  • Roles: admin, member, contributor, guest
  • Entitlements: Granular permissions per role
  • DAO isolation: Multi-tenancy support

Endpoints:

  • POST /rbac/resolve - Resolve user role and permissions
  • GET /roles - List all roles

🔧 Configuration

Routing Rules

Edit router-config.yml to customize routing behavior:

routing_rules:
  - name: "microdao_orchestrator"
    priority: 5
    conditions:
      mode: "crew"
    use_provider: "microdao_orchestrator"
    timeout_ms: 60000

Rule fields:

  • priority - Lower = higher priority
  • conditions - Match criteria (mode, prompt patterns, metadata)
  • use_provider - Target provider name
  • timeout_ms - Request timeout

Environment Variables

See .env.example for full list. Key variables:

# LLM Configuration
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_MODEL=qwen3:8b

# Service Ports
ROUTER_PORT=9102
GATEWAY_PORT=9300

# Security
RBAC_SECRET_KEY=your-secret-here

# Logging
LOG_LEVEL=INFO
LOG_FORMAT=json

🧪 Testing

Smoke Tests

Run basic health checks:

./smoke.sh

E2E Tests

Test individual components:

./test-devtools.sh    # DevTools integration
./test-crewai.sh      # CrewAI workflows
./test-gateway.sh     # Gateway + RBAC

Manual Testing

# Test LLM routing
curl -X POST http://localhost:9102/route \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Test", "mode": "chat", "metadata": {}}'

# Test DevTools
curl -X POST http://localhost:8008/fs/read \
  -H "Content-Type: application/json" \
  -d '{"path": "README.md"}'

# Test CrewAI
curl -X POST http://localhost:9010/workflow/run \
  -H "Content-Type: application/json" \
  -d '{"workflow_name": "code_review", "inputs": {}}'

# Test RBAC
curl -X POST http://localhost:9200/rbac/resolve \
  -H "Content-Type: application/json" \
  -d '{"dao_id": "greenfood-dao", "user_id": "tg:12345"}'

📊 Monitoring & Logs

View Logs

# All services
docker-compose logs -f

# Specific service
docker-compose logs -f router
docker-compose logs -f gateway

Structured JSON Logs

All services use structured logging:

{
  "timestamp": "2024-11-15T12:00:00Z",
  "level": "INFO",
  "service": "router",
  "message": "Request routed successfully",
  "request_id": "123e4567-e89b-12d3-a456-426614174000",
  "user_id": "tg:12345",
  "dao_id": "greenfood-dao",
  "duration_ms": 125.5
}

Health Checks

All services expose /health endpoint:

curl http://localhost:9102/health

🚢 Deployment

docker-compose up -d

Kubernetes

See docs/DEPLOYMENT.md for Kubernetes manifests and Helm charts.

Systemd

For production servers without containers:

sudo cp deploy/systemd/dagi-router.service /etc/systemd/system/
sudo systemctl enable dagi-router
sudo systemctl start dagi-router

Full deployment guide: docs/DEPLOYMENT.md


🛣️ Roadmap

Phase 1: Core Router

  • Multi-provider LLM support
  • Rule-based routing engine
  • YAML configuration
  • Basic health checks

Phase 2: Orchestration

  • DevTools integration
  • CrewAI workflows
  • Bot gateway (Telegram/Discord)
  • RBAC service

Phase 3: Production (Current)

  • Docker deployment
  • Structured logging
  • Smoke test suite
  • Prometheus metrics
  • CI/CD pipelines

Phase 4: Governance (Planned)

  • On-chain voting integration
  • Token-weighted decisions
  • Proposal lifecycle management
  • Treasury operations

Phase 5: Scale (Future)

  • Horizontal scaling
  • Load balancing
  • Distributed tracing
  • Performance optimization

📚 Documentation


🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

  1. Fork the repository
  2. Create feature branch (git checkout -b feature/amazing-feature)
  3. Commit changes (git commit -m 'Add amazing feature')
  4. Push to branch (git push origin feature/amazing-feature)
  5. Open Pull Request

📄 License

This project is licensed under the MIT License - see LICENSE file for details.



💬 Support


Built with ❤️ by the DAARION Community


🎯 First Deployment

Ready to deploy? Follow our step-by-step guide:

📖 First Deployment Guide - Complete walkthrough for first live deployment

5-step process (15 minutes):

  1. Initial Setup - Configure .env, generate secrets
  2. Pre-flight Checks - Verify Docker, disk, memory
  3. Service Startup - docker-compose up -d
  4. Health Verification - Run ./smoke.sh
  5. First Real Dialog - Test Telegram bot or curl

Includes:

  • Pre-deployment checklist
  • Troubleshooting guide
  • Post-deployment verification
  • Success confirmation criteria

🧪 Golden Scenarios

After deployment, validate your stack with production scenarios:

📖 Golden Scenarios Guide - 5 end-to-end test scenarios

Scenarios:

  1. Basic Chat - Telegram → Gateway → Router → LLM (5s)
  2. microDAO Onboarding - CrewAI 3-agent workflow (60s)
  3. DevTools File Operation - Read/write files (1s)
  4. Code Review - Multi-agent code analysis (90s)
  5. RBAC Permission Check - Access control validation (100ms)

Each scenario includes:

  • Setup requirements
  • Expected flow diagram
  • Verification commands
  • Success criteria
  • Troubleshooting tips
Description
DAARION & MicroDAO - Main Project Repository
Readme 42 MiB
Languages
HTML 65.4%
Python 32.7%
Shell 1%
PLpgSQL 0.3%
Jupyter Notebook 0.2%
Other 0.2%