Skip to content

# PHASE 3 ROADMAP — Core Agent Services

After Phase 2 Agent Integration

Status: 📋 Planning → ✅ SPEC READY
Master Task: PHASE3_MASTER_TASK.md ⭐
Summary: PHASE3_READY.md
Priority: High
Estimated Time: 6-8 weeks
Dependencies: Phase 2 complete

## 🎯 Goal

Replace Phase 2 stubs with production-ready services: - Real LLM Proxy (multi-provider routing) - Real Agent Memory (RAG + vector DB) - Tool Registry (agent actions) - Agent Blueprint Management (CRUD + versioning)

## 📦 Phase 3 Components

### 1. LLM Proxy Service (2 weeks)

Purpose: Centralized LLM gateway with routing, rate limiting, cost tracking

Features: - Multi-provider support (OpenAI, Anthropic, DeepSeek, Local) - Model selection & routing - Rate limiting per agent/microDAO - Cost tracking & billing - Streaming support - Error handling & retries - Prompt sanitization

API:

    POST /internal/llm/proxy
    {
      "model": "gpt-4",
      "messages": [...],
      "stream": false,
      "max_tokens": 1000,
      "agent_id": "agent:sofia",
      "microdao_id": "microdao:daarion"
    }

    GET /internal/llm/models
    → List available models

    GET /internal/llm/usage?agent_id=agent:sofia&period=30d
    → Usage statistics

Tech Stack: - FastAPI - httpx for provider calls - Redis for rate limiting - PostgreSQL for usage tracking

Files:

    services/llm-proxy/
    ├── main.py
    ├── providers/
    │   ├── openai.py
    │   ├── anthropic.py
    │   ├── deepseek.py
    │   └── local.py
    ├── routing.py
    ├── rate_limiter.py
    ├── cost_tracker.py
    ├── models.py
    └── config.yaml

### 2. Agent Memory Service (2 weeks)

Purpose: Persistent memory + RAG for agents

Features: - Short-term memory (recent context) - Mid-term memory (session/task memory) - Long-term memory (knowledge base) - Vector search (RAG) - Memory indexing (from channel history) - Memory pruning (for cost/performance) - Per-agent & per-microDAO isolation

API:

    POST /internal/agent-memory/query
    {
      "agent_id": "agent:sofia",
      "microdao_id": "microdao:daarion",
      "query": "What did we discuss about Phase 2?",
      "k": 5,
      "memory_types": ["mid_term", "long_term"]
    }
    → Top-k relevant memories

    POST /internal/agent-memory/store
    {
      "agent_id": "agent:sofia",
      "microdao_id": "microdao:daarion",
      "memory_type": "mid_term",
      "content": {
        "user_message": "...",
        "agent_reply": "...",
        "context": {...}
      }
    }
    → Store new memory

    GET /internal/agent-memory/agents/{agent_id}/stats
    → Memory usage stats

Tech Stack: - FastAPI - PostgreSQL (structured memory) - Qdrant/Weaviate/ChromaDB (vector DB for RAG) - LangChain/LlamaIndex (RAG helpers)

Files:

    services/agent-memory/
    ├── main.py
    ├── vector_store.py
    ├── memory_manager.py
    ├── rag_engine.py
    ├── indexer.py
    ├── models.py
    └── config.yaml

### 3. Tool Registry Service (1.5 weeks)

Purpose: Centralized tool definitions & execution for agents

Features: - Tool catalog (list all available tools) - Tool execution (secure sandbox) - Tool permissions (agent → tool mapping) - Tool versioning - Execution logs & auditing

Tools (initial set): - `create_task(channel_id, title, description)` - `create_followup(user_id, message_id, reminder_text, due_date)` - `search_docs(query)` - `create_project(microdao_id, name, description)` - `summarize_channel(channel_id, period)` - `send_notification(user_id, text)`

API:

    GET /internal/tools/catalog
    → List all tools

    POST /internal/tools/execute
    {
      "tool_name": "create_task",
      "agent_id": "agent:sofia",
      "microdao_id": "microdao:daarion",
      "parameters": {
        "channel_id": "...",
        "title": "Review Phase 2",
        "description": "..."
      }
    }
    → Execute tool, return result

    GET /internal/tools/agents/{agent_id}/permissions
    → List tools agent can use

Tech Stack: - FastAPI - Dynamic tool loading (plugins) - Sandboxed execution (Docker/gVisor) - PostgreSQL (tool definitions, permissions, logs)

Files:

    services/tool-registry/
    ├── main.py
    ├── catalog.py
    ├── executor.py
    ├── sandbox.py
    ├── permissions.py
    ├── tools/
    │   ├── task_tools.py
    │   ├── project_tools.py
    │   ├── notification_tools.py
    │   └── ...
    └── config.yaml

### 4. Agent Blueprint Service (1 week)

Purpose: CRUD + versioning for agent definitions

Features: - Create/Read/Update/Delete agent blueprints - Blueprint versioning - Blueprint templates (archetypes) - Blueprint validation - Blueprint inheritance

API:

    GET /internal/agents/blueprints
    → List all blueprints

    POST /internal/agents/blueprints
    {
      "code": "sofia_prime_v2",
      "name": "Sofia Prime v2",
      "model": "gpt-4.1",
      "instructions": "...",
      "capabilities": {...},
      "tools": ["create_task", "summarize_channel"]
    }
    → Create blueprint

    GET /internal/agents/blueprints/{blueprint_id}
    → Get blueprint

    GET /internal/agents/{agent_id}/blueprint
    → Get blueprint for specific agent instance

    PUT /internal/agents/blueprints/{blueprint_id}
    → Update blueprint (creates new version)

Tech Stack: - FastAPI - PostgreSQL (blueprints, versions) - YAML/JSON schema validation

Files:

    services/agents-service/
    ├── main.py
    ├── blueprints/
    │   ├── crud.py
    │   ├── versioning.py
    │   ├── validation.py
    │   └── templates.py
    ├── models.py
    └── config.yaml

### 5. Integration Updates (1 week)

Update agent-runtime to use real services:

    # Before (Phase 2):
    blueprint = await load_agent_blueprint(agent_id)  # Mock
    memory = await query_memory(...)  # Stub
    llm_response = await generate_response(...)  # Stub

    # After (Phase 3):
    blueprint = await agents_service.get_blueprint(agent_id)  # Real
    memory = await memory_service.query(...)  # Real RAG
    llm_response = await llm_proxy.generate(...)  # Real multi-provider

    # NEW: Tool usage
    if llm_suggests_tool_use:
        tool_result = await tool_registry.execute(tool_name, parameters)
        # Add tool result to context, call LLM again

## 📅 Timeline

### Week 1-2: LLM Proxy

- Week 1: Core routing + OpenAI provider
- Week 2: Multi-provider + rate limiting + cost tracking

### Week 3-4: Agent Memory

- Week 3: Vector store setup + basic RAG
- Week 4: Memory management + indexing

### Week 5-6: Tool Registry

- Week 5: Catalog + basic tools (task, followup)
- Week 6: Executor + permissions + sandboxing

### Week 7: Agent Blueprint Service

- CRUD + versioning + validation

### Week 8: Integration & Testing

- Update agent-runtime
- E2E testing
- Performance optimization
- Documentation

## 🧪 Testing Strategy

### LLM Proxy Testing:

- Unit: Each provider (OpenAI, Anthropic, etc.)
- Integration: Rate limiting, cost tracking
- Load: 100 concurrent requests
- Failover: Provider unavailable scenarios

### Agent Memory Testing:

- RAG accuracy: Retrieve relevant memories
- Memory indexing: Auto-index from channels
- Vector search performance: < 500ms
- Memory pruning: Clean old memories

### Tool Registry Testing:

- Tool execution: All tools work
- Permissions: Agent cannot use unauthorized tools
- Sandboxing: Tools cannot escape sandbox
- Audit logs: All executions logged

### E2E Testing:

- User asks agent to create task → Task created
- User asks agent to summarize → Summary posted
- Agent uses memory correctly in replies
- Multiple providers work (switch between OpenAI/DeepSeek)

## 🎯 Acceptance Criteria

### Phase 3 Complete When:

- ✅ LLM Proxy supports 3+ providers
- ✅ Agent Memory RAG works (< 500ms queries)
- ✅ Tool Registry has 5+ working tools
- ✅ Agent Blueprint CRUD works
- ✅ agent-runtime integrated with all services
- ✅ E2E: User → Agent (with tool use) → Result
- ✅ Cost tracking shows LLM usage per agent
- ✅ Memory usage shows per agent/microDAO
- ✅ All services pass health checks
- ✅ Documentation complete

## 📊 Success Metrics

Metric | Target
--- | ---
LLM response time | < 2s (non-streaming)
Memory query time | < 500ms
Tool execution time | < 3s
E2E agent reply | < 5s (with tool use)
LLM cost per request | < $0.05
System uptime | > 99.5%

## 🔗 Dependencies

### External Services:

- OpenAI API (for GPT-4)
- Anthropic API (for Claude, optional)
- DeepSeek API (optional)
- Qdrant/Weaviate (for vector DB)

### Internal Services:

- PostgreSQL (for all structured data)
- Redis (for rate limiting, caching)
- NATS (for events)

## 💡 Optional Enhancements (Phase 3.5)

### LLM Proxy:

- Streaming SSE support
- Local model support (Ollama, vLLM)
- Prompt caching
- A/B testing for prompts

### Agent Memory:

- Hierarchical memory (microDAO → team → agent)
- Memory sharing between agents
- Memory snapshots (save/restore agent state)
- Memory analytics dashboard

### Tool Registry:

- Tool marketplace (community tools)
- Tool composition (chain tools)
- Visual tool builder
- Tool usage analytics

## 🚀 Quick Start (After Phase 2)

### To prepare for Phase 3:

    # 1. Review Phase 3 roadmap
    cat docs/tasks/PHASE3_ROADMAP.md

    # 2. Set up external services
    # - Get OpenAI API key
    # - Set up Qdrant (Docker or cloud)
    # - Set up Redis

    # 3. Start with LLM Proxy
    mkdir -p services/llm-proxy
    cd services/llm-proxy
    # Follow PHASE3_LLM_PROXY_TASK.md (to be created)

## 📝 Task Files (To Be Created)

After Phase 2 complete, create detailed tasks:

1. TASK_PHASE3_LLM_PROXY.md (2 weeks)
2. TASK_PHASE3_AGENT_MEMORY.md (2 weeks)
3. TASK_PHASE3_TOOL_REGISTRY.md (1.5 weeks)
4. TASK_PHASE3_BLUEPRINT_SERVICE.md (1 week)
5. TASK_PHASE3_INTEGRATION.md (1 week)

## 🎓 Architecture Evolution

### Phase 1 (Complete):

    User → Frontend → messaging-service → Matrix → Frontend

### Phase 2 (Current):

    User → Messenger → agent_filter → Router → agent-runtime (stub) → Reply

### Phase 3 (Target):

    User → Messenger
        ↓
    agent_filter → Router → agent-runtime
        ↓
    ├─ LLM Proxy → [OpenAI | Anthropic | DeepSeek]
    ├─ Agent Memory → [Vector DB | PostgreSQL]
    ├─ Tool Registry → [Task | Project | Notification tools]
    └─ Agent Blueprint → [Definitions | Versions]
        ↓
    Reply with tool results

## ✅ Current Status

- ✅ Phase 1: Messenger Core (Complete)
- 📋 Phase 2: Agent Integration (In Progress)
- 📋 Phase 3: Core Services (This Roadmap)
- 🔜 Phase 4: Advanced Features (TBD)

Ready for Phase 3?

First complete Phase 2, then return to this roadmap for detailed implementation tasks.

Version: 1.0.0
Date: 2025-11-24
Status: Planning