feat: implement RAG Service MVP with PARSER + Memory integration

RAG Service Implementation:
- Create rag-service/ with full structure (config, document_store, embedding, pipelines)
- Document Store: PostgreSQL + pgvector via Haystack
- Embedding: BAAI/bge-m3 (multilingual, 1024 dim)
- Ingest Pipeline: Convert ParsedDocument to Haystack Documents, embed, index
- Query Pipeline: Retrieve documents, generate answers via DAGI Router
- FastAPI endpoints: /ingest, /query, /health

Tests:
- Unit tests for ingest and query pipelines
- E2E test with example parsed JSON
- Test fixtures with real PARSER output example

Router Integration:
- Add mode='rag_query' routing rule in router-config.yml
- Priority 7, uses local_qwen3_8b for RAG queries

Docker:
- Add rag-service to docker-compose.yml
- Configure dependencies (router, city-db)
- Add model cache volume

Documentation:
- Complete README with API examples
- Integration guides for PARSER and Router
This commit is contained in:
Apple
2025-11-16 04:41:53 -08:00
parent d3c701f3ff
commit 9b86f9a694
19 changed files with 1275 additions and 97 deletions

View File

@@ -0,0 +1,52 @@
"""
Embedding service for RAG
Uses SentenceTransformers via Haystack
"""
import logging
from typing import Optional
from haystack.components.embedders import SentenceTransformersTextEmbedder
from app.core.config import settings
logger = logging.getLogger(__name__)
# Global embedder instance
_text_embedder: Optional[SentenceTransformersTextEmbedder] = None
def get_text_embedder() -> SentenceTransformersTextEmbedder:
"""
Get or create SentenceTransformersTextEmbedder instance
Returns:
SentenceTransformersTextEmbedder configured with embedding model
"""
global _text_embedder
if _text_embedder is not None:
return _text_embedder
logger.info(f"Loading embedding model: {settings.EMBED_MODEL_NAME}")
logger.info(f"Device: {settings.EMBED_DEVICE}")
try:
_text_embedder = SentenceTransformersTextEmbedder(
model=settings.EMBED_MODEL_NAME,
device=settings.EMBED_DEVICE,
)
logger.info("Text embedder initialized successfully")
return _text_embedder
except Exception as e:
logger.error(f"Failed to initialize TextEmbedder: {e}", exc_info=True)
raise RuntimeError(f"TextEmbedder initialization failed: {e}") from e
def reset_embedder():
"""Reset global embedder instance (for testing)"""
global _text_embedder
_text_embedder = None