Skip to content

# Task: Unified RAG-Gateway service (Milvus + Neo4j) for all agents

## Goal

Design and implement a single RAG-gateway service that sits between agents and storage backends (Milvus, Neo4j, etc.), so that:

- Agents never talk directly to Milvus or Neo4j.
- All retrieval, graph queries and hybrid RAG behavior go through one service with a clear API.
- Security, multi-tenancy, logging, and optimization are centralized.

This task is about architecture and API first (code layout, endpoints, data contracts). A later task can cover concrete implementation details if needed.

> This spec is intentionally high-level but should be detailed enough for Cursor to scaffold the service, HTTP API, and integration points with DAGI Router.

## Context

- Project root: `microdao-daarion/`.
- There are (or will be) multiple agents:
- DAARWIZZ (system orchestrator)
- Helion (Energy Union)
- Team/Project/Messenger/Co-Memory agents, etc.
- Agents already have access to:
- DAGI Router (LLM routing, tools, orchestrator).
- Memory service (short/long-term chat memory).
- Parser-service (OCR and document parsing).

We now want a RAG layer that can:

- Perform semantic document search across all DAO documents / messages / files.
- Use a vector DB (Milvus) and graph DB (Neo4j) together.
- Provide a clean tool-like API to agents.

The RAG layer should be exposed as a standalone service:

- Working name: `rag-gateway` or `knowledge-service`.
- Internally can use Haystack (or similar) for pipelines.

## High-level architecture

### 1. RAG-Gateway service

Create a new service (later we can place it under `services/rag-gateway/`), with HTTP API, which will:

- Accept tool-style requests from DAGI Router / agents.
- Internally talk to:
- Milvus (vector search, embeddings).
- Neo4j (graph queries, traversals).
- Return structured JSON for agents to consume.

Core API endpoints (first iteration):

- `POST /rag/search_docs` — semantic/hybrid document search.
- `POST /rag/enrich_answer` — enrich an existing answer with sources.
- `POST /graph/query` — run a graph query (Cypher or intent-based).
- `POST /graph/explain_path` — return graph-based explanation / path between entities.

Agents will see these as tools (e.g. `rag.search_docs`, `graph.query_context`) configured in router config.

### 2. Haystack as internal orchestrator

Within the RAG-gateway, use Haystack components (or analogous) to organize:

- `MilvusDocumentStore` as the main vector store.
- Retrievers:
- Dense retriever over Milvus.
- Optional BM25/keyword retriever (for hybrid search).
- Pipelines:
- `indexing_pipeline` — ingest DAO documents/messages/files into Milvus.
- `query_pipeline` — answer agent queries using retrieved documents.
- `graph_rag_pipeline` — combine Neo4j graph queries with Milvus retrieval.

The key idea: agents never talk to Haystack directly, only to RAG-gateway HTTP API.

## Data model & schema

### 1. Milvus document schema

Define a standard metadata schema for all documents/chunks stored in Milvus. Required fields:

- `team_id` / `dao_id` — which DAO / team this data belongs to.
- `project_id` — optional project-level grouping.
- `channel_id` — optional chat/channel ID (Telegram, internal channel, etc.).
- `agent_id` — which agent produced/owns this piece.
- `visibility` — one of `"public" | "confidential"`.
- `doc_type` — one of `"message" | "doc" | "file" | "wiki" | "rwa" | "transaction"` (extensible).
- `tags` — list of tags (topics, domains, etc.).
- `created_at` — timestamp.

These should be part of Milvus metadata, so that RAG-gateway can apply filters (by DAO, project, visibility, etc.).

### 2. Neo4j graph schema

Design a minimal default graph model with node labels:

- `User`, `Agent`, `MicroDAO`, `Project`, `Channel`
- `Topic`, `Resource`, `File`, `RWAObject` (e.g. energy asset, food batch, water object).

Key relationships (examples):

- `(:User)-[:MEMBER_OF]->(:MicroDAO)`
- `(:Agent)-[:SERVES]->(:MicroDAO|:Project)`
- `(:Doc)-[:MENTIONS]->(:Topic)`
- `(:Project)-[:USES]->(:Resource)`

Every node/relationship should also carry:

- `team_id` / `dao_id`
- `visibility` or similar privacy flag

This allows RAG-gateway to enforce access control at query time.

## RAG tools API for agents

Define 2–3 canonical tools that DAGI Router can call. These map to RAG-gateway endpoints.

### 1. `rag.search_docs`

Main tool for most knowledge queries.

Request JSON example:

    {
      "agent_id": "ag_daarwizz",
      "team_id": "dao_greenfood",
      "query": "які проєкти у нас вже використовують Milvus?",
      "top_k": 5,
      "filters": {
        "project_id": "prj_x",
        "doc_type": ["doc", "wiki"],
        "visibility": "public"
      }
    }

Response JSON example:

    {
      "matches": [
        {
          "score": 0.82,
          "title": "Spec microdao RAG stack",
          "snippet": "...",
          "source_ref": {
            "type": "doc",
            "id": "doc_123",
            "url": "https://...",
            "team_id": "dao_greenfood",
            "doc_type": "doc"
          }
        }
      ]
    }

### 2. `graph.query_context`

For relationship/structural questions ("хто з ким повʼязаний", "які проєкти використовують X" etc.).

Two options (can support both):

1. Low-level Cypher:

`json { "team_id": "dao_energy", "cypher": "MATCH (p:Project)-[:USES]->(r:Resource {name:$name}) RETURN p LIMIT 10", "params": {"name": "Milvus"} }`

1. High-level intent:

`json { "team_id": "dao_energy", "intent": "FIND_PROJECTS_BY_TECH", "args": {"tech": "Milvus"} }`

RAG-gateway then maps intent → Cypher internally.

### 3. `rag.enrich_answer`

Given a draft answer from an agent, RAG-gateway retrieves supporting documents and returns enriched answer + citations.

Request example:

    {
      "team_id": "dao_greenfood",
      "question": "Поясни коротко архітектуру RAG шару в нашому місті.",
      "draft_answer": "Архітектура складається з ...",
      "max_docs": 3
    }

Response example:

    {
      "enriched_answer": "Архітектура складається з ... (з врахуванням джерел)",
      "sources": [
        {"id": "doc_1", "title": "RAG spec", "url": "https://..."},
        {"id": "doc_2", "title": "Milvus setup", "url": "https://..."}
      ]
    }

## Multi-tenancy & security

Add a small authorization layer inside RAG-gateway:

- Each request includes:
- `user_id`, `team_id` (DAO), optional `roles`.
- `mode` / `visibility` (e.g. `"public"` or `"confidential"`).
- Before querying Milvus/Neo4j, RAG-gateway applies filters:
- `team_id = ...`
- `visibility` within allowed scope.
- Optional role-based constraints (Owner/Guardian/Member) affecting what doc_types can be seen.

Implementation hints:

- Start with a simple `AccessContext` object built from request, used by all pipelines.
- Later integrate with existing PDP/RBAC if available.

## Ingestion & pipelines

Define an ingestion plan and API.

### 1. Ingest service / worker

Create a separate ingestion component (can be part of RAG-gateway or standalone worker) that:

- Listens to events like:
- `message.created`
- `doc.upsert`
- `file.uploaded`
- For each event:
- Builds text chunks.
- Computes embeddings.
- Writes chunks into Milvus with proper metadata.
- Updates Neo4j graph (nodes/edges) where appropriate.

Requirements:

- Pipelines must be idempotent — re-indexing same document does not break anything.
- Create an API / job for `reindex(team_id)` to reindex a full DAO if needed.
- Store embedding model version in metadata (e.g. `embed_model: "bge-m3@v1"`) to ease future migrations.

### 2. Event contracts

Align ingestion with the existing Event Catalog (if present in `docs/cursor`):

- Document which event types lead to RAG ingestion.
- For each event, define mapping → Milvus doc, Neo4j nodes/edges.

## Optimization for agents

Add support for:

1. Semantic cache per agent

2. Cache `query → RAG-result` for N minutes per (`agent_id`, `team_id`).

3. Useful for frequently repeated queries.

4. RAG behavior profiles per agent

5. In agent config (probably in router config), define:

   - `rag_mode: off | light | strict`
   - `max_context_tokens`
   - `max_docs_per_query`
6. RAG-gateway can read these via metadata from Router, or Router can decide when to call RAG at all.

## Files to create/modify (suggested)

> NOTE: This is a suggestion; adjust exact paths/names to fit the existing project structure.

- New service directory: `services/rag-gateway/`:
- `main.py` — FastAPI (or similar) entrypoint.
- `api.py` — defines `/rag/search_docs`, `/rag/enrich_answer`, `/graph/query`, `/graph/explain_path`.
- `core/pipelines.py` — Haystack pipelines (indexing, query, graph-rag).
- `core/schema.py` — Pydantic models for request/response, data schema.
- `core/access.py` — access control context + checks.
- `core/backends/milvus_client.py` — wrapper for Milvus.

- `core/backends/neo4j_client.py` — wrapper for Neo4j.

- Integration with DAGI Router:

- Update `router-config.yml` to define RAG tools:
  - `rag.search_docs`
  - `graph.query_context`
  - `rag.enrich_answer`

- Configure providers for RAG-gateway base URL.

- Docs:

- `docs/cursor/rag_gateway_api_spec.md` — optional detailed API spec for RAG tools.

## Acceptance criteria

1. Service skeleton

2. A new RAG-gateway service exists under `services/` with:

   - A FastAPI (or similar) app.
   - Endpoints:
   - `POST /rag/search_docs`
   - `POST /rag/enrich_answer`
   - `POST /graph/query`
   - `POST /graph/explain_path`
   - Pydantic models for requests/responses.

3. Data contracts

4. Milvus document metadata schema is defined (and used in code).

5. Neo4j node/edge labels and key relationships are documented and referenced in code.

6. Security & multi-tenancy

7. All RAG/graph endpoints accept `user_id`, `team_id`, and enforce at least basic filtering by `team_id` and `visibility`.

8. Agent tool contracts

9. JSON contracts for tools `rag.search_docs`, `graph.query_context`, and `rag.enrich_answer` are documented and used by RAG-gateway.

10. DAGI Router integration is sketched (even if not fully wired): provider entry + basic routing rule examples.

11. Ingestion design

12. Ingestion pipeline is outlined in code (or stubs) with clear TODOs:

    - where to hook event consumption,
    - how to map events to Milvus/Neo4j.

13. Idempotency and `reindex(team_id)` strategy described in code/docs.

14. Documentation

15. This file (`docs/cursor/rag_gateway_task.md`) plus, optionally, a more detailed API spec file for RAG-gateway.

## How to run this task with Cursor

From repo root (`microdao-daarion`):

    cursor task < docs/cursor/rag_gateway_task.md

Cursor should then:

- Scaffold the RAG-gateway service structure.
- Implement request/response models and basic endpoints.
- Sketch out Milvus/Neo4j client wrappers and pipelines.
- Optionally, add TODOs where deeper implementation is needed.