0c8bef82f4
## Agents Added - Alateya: R&D, biotech, innovations - Clan (Spirit): Community spirit agent - Eonarch: Consciousness evolution agent ## Changes - docker-compose.node1.yml: Added tokens for all 3 new agents - gateway-bot/http_api.py: Added configs and webhook endpoints - gateway-bot/clan_prompt.txt: New prompt file - gateway-bot/eonarch_prompt.txt: New prompt file ## Fixes - Fixed ROUTER_URL from :9102 to :8000 (internal container port) - All 9 Telegram agents now working ## Documentation - Created PROJECT-MASTER-INDEX.md - single entry point - Added various status documents and scripts Tokens configured: - Helion, NUTRA, Agromatrix (existing) - Alateya, Clan, Eonarch (new) - Druid, GreenFood, DAARWIZZ (configured)
184 lines
4.4 KiB
Markdown
184 lines
4.4 KiB
Markdown
# Qdrant Canonical Migration - Cutover Checklist
|
|
|
|
**Status:** GO
|
|
**Date:** 2026-01-26
|
|
**Risk Level:** Operational (security invariants verified)
|
|
|
|
---
|
|
|
|
## Pre-Cutover Verification
|
|
|
|
### Security Invariants (VERIFIED ✅)
|
|
|
|
| Invariant | Status |
|
|
|-----------|--------|
|
|
| `tenant_id` always required | ✅ |
|
|
| `agent_ids ⊆ allowed_agent_ids` (or admin) | ✅ |
|
|
| Admin default: no private | ✅ |
|
|
| Empty `should` → error | ✅ |
|
|
| Private only via owner | ✅ |
|
|
| Qdrant `match.any` format | ✅ |
|
|
|
|
---
|
|
|
|
## Cutover Steps
|
|
|
|
### 1. Deploy
|
|
|
|
```bash
|
|
# Copy code to NODE1
|
|
scp -r services/memory/qdrant/ root@NODE1:/opt/microdao-daarion/services/memory/
|
|
scp docs/memory/canonical_collections.yaml root@NODE1:/opt/microdao-daarion/docs/memory/
|
|
scp scripts/qdrant_*.py root@NODE1:/opt/microdao-daarion/scripts/
|
|
|
|
# IMPORTANT: Verify dim/metric in canonical_collections.yaml matches live embedding
|
|
# Current: dim=1024, metric=cosine
|
|
|
|
# Restart service that owns Qdrant reads/writes
|
|
docker compose restart memory-service
|
|
# OR
|
|
systemctl restart memory-service
|
|
```
|
|
|
|
### 2. Migration
|
|
|
|
```bash
|
|
# Dry run - MUST pass before real migration
|
|
python3 scripts/qdrant_migrate_to_canonical.py --all --dry-run 2>&1 | tee migration_dry_run.log
|
|
|
|
# Verify dry run output:
|
|
# - Target collection name(s) shown
|
|
# - Per-collection counts listed
|
|
# - Zero dim/metric mismatches (unless --skip-dim-check used)
|
|
|
|
# Real migration
|
|
python3 scripts/qdrant_migrate_to_canonical.py --all --continue-on-error 2>&1 | tee migration_$(date +%Y%m%d_%H%M%S).log
|
|
|
|
# Review summary:
|
|
# - Collections processed: X/Y
|
|
# - Points migrated: N
|
|
# - Errors: should be 0 or minimal
|
|
```
|
|
|
|
### 3. Parity Check
|
|
|
|
```bash
|
|
python3 scripts/qdrant_parity_check.py --agents helion,nutra,druid 2>&1 | tee parity_check.log
|
|
|
|
# Requirements:
|
|
# - Count parity within tolerance
|
|
# - topK overlap threshold passes
|
|
# - Schema validation passes
|
|
```
|
|
|
|
### 4. Dual-Read Window
|
|
|
|
```bash
|
|
# Enable dual-read for validation
|
|
export DUAL_READ_OLD=true
|
|
|
|
# Restart service to pick up env change
|
|
docker compose restart memory-service
|
|
```
|
|
|
|
**Validation queries (must pass):**
|
|
|
|
| Query Type | Expected Result |
|
|
|------------|-----------------|
|
|
| Agent-only (Helion) | Returns own docs, no other agents |
|
|
| Multi-agent (DAARWIZZ) | Returns from allowed agents only |
|
|
| Private visibility | Only owner sees private |
|
|
|
|
```bash
|
|
# Run smoke test
|
|
python3 scripts/qdrant_smoke_test.py --host localhost
|
|
```
|
|
|
|
### 5. Cutover
|
|
|
|
```bash
|
|
# Disable dual-read
|
|
export DUAL_READ_OLD=false
|
|
|
|
# Ensure no legacy writes
|
|
export DUAL_WRITE_OLD=false
|
|
# OR remove these env vars entirely
|
|
|
|
# Restart service
|
|
docker compose restart memory-service
|
|
|
|
# Verify service is healthy
|
|
curl -s http://localhost:8000/health
|
|
```
|
|
|
|
### 6. Post-Cutover Guard
|
|
|
|
```bash
|
|
# Keep legacy collections for rollback window (recommended: 7 days)
|
|
# DO NOT delete legacy collections yet
|
|
|
|
# After rollback window (7 days):
|
|
# 1. Run one more parity check
|
|
python3 scripts/qdrant_parity_check.py --all
|
|
|
|
# 2. If parity passes, delete legacy collections
|
|
# WARNING: This is irreversible
|
|
# python3 -c "
|
|
# from qdrant_client import QdrantClient
|
|
# client = QdrantClient(host='localhost', port=6333)
|
|
# legacy = ['helion_docs', 'nutra_messages', ...] # list all legacy
|
|
# for col in legacy:
|
|
# client.delete_collection(col)
|
|
# print(f'Deleted: {col}')
|
|
# "
|
|
```
|
|
|
|
---
|
|
|
|
## Rollback Procedure
|
|
|
|
If issues arise after cutover:
|
|
|
|
```bash
|
|
# 1. Re-enable dual-read from legacy
|
|
export DUAL_READ_OLD=true
|
|
export DUAL_WRITE_OLD=true # if needed
|
|
|
|
# 2. Restart service
|
|
docker compose restart memory-service
|
|
|
|
# 3. Investigate issues
|
|
# - Check migration logs
|
|
# - Check parity results
|
|
# - Review error messages
|
|
|
|
# 4. If canonical data is corrupted, switch to legacy-only mode:
|
|
# (requires code change to bypass canonical reads)
|
|
```
|
|
|
|
---
|
|
|
|
## Files Reference
|
|
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| `services/memory/qdrant/` | Canonical Qdrant module |
|
|
| `docs/memory/canonical_collections.yaml` | Collection config |
|
|
| `docs/memory/cm_payload_v1.md` | Payload schema docs |
|
|
| `scripts/qdrant_migrate_to_canonical.py` | Migration tool |
|
|
| `scripts/qdrant_parity_check.py` | Parity verification |
|
|
| `scripts/qdrant_smoke_test.py` | Security smoke test |
|
|
|
|
---
|
|
|
|
## Sign-off
|
|
|
|
- [ ] Dry run passed
|
|
- [ ] Migration completed
|
|
- [ ] Parity check passed
|
|
- [ ] Dual-read validation passed
|
|
- [ ] Cutover completed
|
|
- [ ] Post-cutover health verified
|
|
- [ ] Rollback window started (7 days)
|
|
- [ ] Legacy collections deleted (after rollback window)
|