feat: complete RAG pipeline integration (ingest + query + Memory)
Parser Service: - Add /ocr/ingest endpoint (PARSER → RAG in one call) - Add RAG_BASE_URL and RAG_TIMEOUT to config - Add OcrIngestResponse schema - Create file_converter utility for PDF/image → PNG bytes - Endpoint accepts file, dao_id, doc_id, user_id - Automatically parses with dots.ocr and sends to RAG Service Router Integration: - Add _handle_rag_query() method in RouterApp - Combines Memory + RAG → LLM pipeline - Get Memory context (facts, events, summaries) - Query RAG Service for documents - Build prompt with Memory + RAG documents - Call LLM provider with combined context - Return answer with citations Clients: - Create rag_client.py for Router (query RAG Service) - Create memory_client.py for Router (get Memory context) E2E Tests: - Create e2e_rag_pipeline.sh script for full pipeline test - Test ingest → query → router query flow - Add E2E_RAG_README.md with usage examples Docker: - Add RAG_SERVICE_URL and MEMORY_SERVICE_URL to router environment
This commit is contained in:
59
services/parser-service/app/utils/file_converter.py
Normal file
59
services/parser-service/app/utils/file_converter.py
Normal file
@@ -0,0 +1,59 @@
|
||||
"""
|
||||
Helper functions for file conversion (PDF/image → PNG bytes)
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Tuple, Optional
|
||||
from io import BytesIO
|
||||
|
||||
from PIL import Image
|
||||
from app.runtime.preprocessing import convert_pdf_to_images, load_image, detect_file_type
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def pdf_or_image_to_png_bytes(
|
||||
filename: Optional[str],
|
||||
file_bytes: bytes
|
||||
) -> Tuple[bytes, int]:
|
||||
"""
|
||||
Convert PDF or image file to PNG bytes
|
||||
|
||||
Args:
|
||||
filename: Original filename (for type detection)
|
||||
file_bytes: File content as bytes
|
||||
|
||||
Returns:
|
||||
Tuple of (PNG bytes, number of pages)
|
||||
|
||||
Raises:
|
||||
ValueError: If file type is not supported or conversion fails
|
||||
"""
|
||||
# Detect file type
|
||||
doc_type = detect_file_type(file_bytes, filename)
|
||||
|
||||
if doc_type == "pdf":
|
||||
# Convert PDF to images
|
||||
images = convert_pdf_to_images(file_bytes)
|
||||
|
||||
if not images:
|
||||
raise ValueError("PDF conversion produced no images")
|
||||
|
||||
# Convert first page to PNG bytes (for single-page processing)
|
||||
# For multi-page, we'll process all pages separately
|
||||
first_image = images[0]
|
||||
buf = BytesIO()
|
||||
first_image.convert("RGB").save(buf, format="PNG")
|
||||
png_bytes = buf.getvalue()
|
||||
|
||||
return png_bytes, len(images)
|
||||
|
||||
else:
|
||||
# Load image and convert to PNG
|
||||
image = load_image(file_bytes)
|
||||
buf = BytesIO()
|
||||
image.convert("RGB").save(buf, format="PNG")
|
||||
png_bytes = buf.getvalue()
|
||||
|
||||
return png_bytes, 1
|
||||
|
||||
Reference in New Issue
Block a user