microdao-daarion/docs/sofiia_ui_vnext_audit.md

Sofiia UI vNext — Audit Report

Generated: 2026-02-26 | Scope: file uploads, document DB, session memory, dialog map

---

1. Existing Infrastructure (What We Reuse)

Document Processing — gateway-bot/services/doc_service.py

Fully working channel-agnostic document service:

• parse_document() → Swapper /document endpoint → markdown/text
• ingest_document() → Router POST /v1/documents/ingest → Qdrant chunks
• ask_about_document() → RAG query via Router
• extract_summary_from_bytes() — local extraction for XLSX/CSV/PDF

Supported formats (from gateway-bot/http_api.py): .pdf .doc .docx .rtf .odt .txt .md .csv .tsv .xls .xlsx .xlsm .ods

Plan: sofiia-console proxies uploads to Router /v1/documents/ingest (same path as Telegram).

Storage on NODA2 (docker-compose.memory-node2.yml)

Storage	Container	Port	Notes
PostgreSQL 16	dagi-postgres-node2	5433	DB: daarion_memory, tables: sofiia_messages etc.
Qdrant 1.12.4	dagi-qdrant-node2	6333	Collections: memories, sofiia_messages, sofiia_summaries
Neo4j 5.15	dagi-neo4j-node2	7687	Available for Phase 2 dialog graph

Memory Service Endpoints (Reusable)

• POST /agents/{agent_id}/memory — save chat turn → Postgres + Qdrant + Neo4j
• GET /agents/{agent_id}/memory — retrieve recent events
• POST /threads / GET /threads/{id} — conversation threads
• POST /memories — long-term memory with semantic search
• POST /retrieve — vector search across memories
• POST /facts/upsert / GET /facts/{key} — key-value store

sofiia-console (What Already Exists)

• _do_save_memory() — auto-saves every chat turn to Memory Service
• GET /api/memory/context — retrieves context for session
• POST /api/voice/stt — file upload (multipart) → memory-service STT
• session_id, project_id, user_id — already in request model

---

2. What Is Missing (What We Build)

Component	Status	Plan
sofiia-console DATABASE_URL	❌ MISSING	Add to docker-compose + SQLite fallback
POST /api/files/upload	❌ MISSING	Build in sofiia-console BFF
projects table	❌ MISSING	SQLite (Phase 1), Postgres (Phase 2)
documents table	❌ MISSING	SQLite + metadata
sessions table	❌ MISSING	SQLite + started_at, last_active
messages table	❌ MISSING	SQLite + parent_msg_id for branching
GET /api/chat/history	❌ MISSING	Load messages from SQLite
Projects sidebar UI	❌ MISSING	Left panel in index.html
Dialog Map (tree)	❌ MISSING	Collapsible tree + branching
Upload UI button	❌ MISSING	Paperclip icon in chat bar

---

3. Architecture Decision: SQLite First

Rationale: sofiia-console currently has no DB. Adding a new Postgres connection requires network config changes and service dependency. SQLite:

• Zero infra changes (just a volume mount)
• Works immediately in Docker
• Can migrate to Postgres later via aiosqlite → asyncpg
• Sufficient for 1 user (operator) console workload

Phase 2: DATABASE_URL=postgresql://... env override → same schema via asyncpg.

---

4. Storage Schema (Phase 1)

-- projects
CREATE TABLE projects (
    project_id TEXT PRIMARY KEY,
    name       TEXT NOT NULL,
    description TEXT DEFAULT '',
    created_at  TEXT NOT NULL,  -- ISO8601
    updated_at  TEXT NOT NULL
);

-- documents
CREATE TABLE documents (
    doc_id       TEXT PRIMARY KEY,
    project_id   TEXT NOT NULL REFERENCES projects(project_id),
    file_id      TEXT NOT NULL,
    sha256       TEXT NOT NULL,
    mime         TEXT NOT NULL,
    size_bytes   INTEGER NOT NULL,
    filename     TEXT NOT NULL,
    title        TEXT DEFAULT '',
    tags         TEXT DEFAULT '[]',   -- JSON array
    created_at   TEXT NOT NULL,
    extracted_text TEXT DEFAULT ''    -- first 4KB preview
);

-- sessions
CREATE TABLE sessions (
    session_id  TEXT PRIMARY KEY,
    project_id  TEXT NOT NULL REFERENCES projects(project_id),
    title       TEXT DEFAULT '',
    started_at  TEXT NOT NULL,
    last_active TEXT NOT NULL,
    turn_count  INTEGER DEFAULT 0
);

-- messages (with branching via parent_msg_id)
CREATE TABLE messages (
    msg_id       TEXT PRIMARY KEY,
    session_id   TEXT NOT NULL REFERENCES sessions(session_id),
    role         TEXT NOT NULL,   -- "user" | "assistant"
    content      TEXT NOT NULL,
    ts           TEXT NOT NULL,   -- ISO8601
    parent_msg_id TEXT,           -- NULL for first message; enables branching
    branch_label TEXT DEFAULT ''  -- "main" | "branch-1" | etc.
);

---

5. File Upload Architecture

Browser → POST /api/files/upload (multipart)
              ↓
          BFF: validate mime + size
              ↓
          Save to ./data/uploads/{sha256[:2]}/{sha256}_{filename}
              ↓
          Extract text (pdf/docx/txt/md via python libs or Router OCR)
              ↓
          Store metadata in documents table
              ↓
          POST /v1/documents/ingest → Qdrant (async, best-effort)
              ↓
          Return: {file_id, sha256, mime, size, preview_text, doc_id}

Size limits (env-configurable):

Type	Env	Default
Images	UPLOAD_MAX_IMAGE_MB	10 MB
Videos	UPLOAD_MAX_VIDEO_MB	200 MB
Docs	UPLOAD_MAX_DOC_MB	50 MB

---

6. Session Persistence Strategy

Current: session_id generated on each /api/chat/send → not persisted between page loads.

Phase 1 Fix:

1. Browser stores session_id in localStorage
2. BFF GET /api/sessions/{session_id} checks if session exists → load last N messages
3. New /api/chat/send saves messages to SQLite messages table
4. GET /api/chat/history?session_id=...&limit=50 returns ordered messages

---

7. Dialog Map (Phase 1: Tree View)

Not a full graph canvas — collapsible tree in UI:

• Each session = root node
• Each assistant turn = child node
• "Fork from message" creates a new branch (new session_id with parent_msg_id)
• UI renders as nested <details> tree, no canvas required
• GET /api/sessions/{session_id}/map returns {nodes, edges} JSON

Phase 2: Upgrade to D3.js force-directed graph or Cytoscape.js when Neo4j available.

---

8. Integration Hooks (Phase 2 Flags)

USE_FABRIC_OCR = os.getenv("USE_FABRIC_OCR", "false").lower() == "true"
USE_EMBEDDINGS = os.getenv("USE_EMBEDDINGS", "false").lower() == "true"

• USE_FABRIC_OCR=true → images/PDFs go through Router /v1/capability/ocr
• USE_EMBEDDINGS=true → extracted text indexed in Qdrant via Memory Service

---

9. Constraints

• Access: localhost-only by default (Docker port binding 127.0.0.1:8002:8002)
• Secrets: never stored in upload files or exposed in API responses
• Filename sanitization: secure_filename() + sha256 as storage key (no path traversal)
• Content-type: validated server-side via python-magic or file header bytes (not just extension)