feat: add is_archived flag, assign all agents to MicroDAOs, filter archived in API

docs/internal/maintenance/DATA_CLEANUP_PLAN.md

@@ -0,0 +1,136 @@
+# DAARION Data Cleanup Plan
+
+> **Date:** 2025-11-28  
+> **Status:** In Progress  
+> **Goal:** Remove test/mock data, enforce "every agent has a MicroDAO"
+
+---
+
+## 0. Backup
+
+### Production Database Backup (NODE1)
+
+```bash
+# SSH to NODE1 and create backup
+ssh root@144.76.224.179
+
+# Create backup directory
+mkdir -p /opt/backups
+
+# Backup daarion database
+docker exec dagi-postgres pg_dump -U postgres -Fc daarion > /opt/backups/daarion_before_cleanup_$(date +%Y%m%d_%H%M%S).dump
+
+# Verify backup
+ls -la /opt/backups/
+```
+
+**Backup created:** `[TO BE FILLED]`
+
+---
+
+## 1. Database Schema Discovery
+
+### Key Tables
+
+#### agents
+```sql
+-- Columns discovered:
+-- id, display_name, kind, status, node_id, is_public, public_slug, 
+-- public_title, public_tagline, public_skills, public_district,
+-- avatar_url, capabilities, model, created_at, updated_at
+```
+
+#### microdaos
+```sql
+-- id, slug, name, description, district, base_node_id, created_at
+```
+
+#### microdao_agents
+```sql
+-- id, microdao_id, agent_id, role, is_core, created_at
+```
+
+#### node_cache
+```sql
+-- id, node_id, node_name, hostname, roles, environment, status, gpu, last_sync
+```
+
+---
+
+## 2. Orphan Agents Analysis
+
+### Agents without MicroDAO membership
+```sql
+SELECT a.id, a.display_name, a.kind, a.node_id
+FROM agents a
+LEFT JOIN microdao_agents ma ON ma.agent_id = a.id
+WHERE ma.agent_id IS NULL
+ORDER BY a.display_name;
+```
+
+**Results:** `[TO BE FILLED]`
+
+### Agents without node_id
+```sql
+SELECT id, display_name FROM agents WHERE node_id IS NULL OR node_id = '';
+```
+
+**Results:** `[TO BE FILLED]`
+
+### Agents not in router-config
+```sql
+-- Compare with router-config.yml agent_ids
+```
+
+**Results:** `[TO BE FILLED]`
+
+---
+
+## 3. MicroDAO Analysis
+
+### MicroDAO with 0 agents
+```sql
+SELECT m.id, m.slug, m.name, COUNT(ma.agent_id) AS agents_count
+FROM microdaos m
+LEFT JOIN microdao_agents ma ON ma.microdao_id = m.id
+GROUP BY m.id
+HAVING COUNT(ma.agent_id) = 0
+ORDER BY m.name;
+```
+
+**Results:** `[TO BE FILLED]`
+
+### MicroDAO without orchestrator
+```sql
+SELECT m.id, m.slug, m.name
+FROM microdaos m
+WHERE NOT EXISTS (
+    SELECT 1 FROM microdao_agents ma 
+    WHERE ma.microdao_id = m.id AND ma.role = 'orchestrator'
+);
+```
+
+**Results:** `[TO BE FILLED]`
+
+---
+
+## 4. Cleanup Actions Log
+
+| Step | Action | SQL/Script | Status |
+|------|--------|------------|--------|
+| 1 | Add is_archived to agents | migrations/023_agents_add_archived.sql | Pending |
+| 2 | Add is_archived to microdaos | migrations/023_agents_add_archived.sql | Pending |
+| 3 | Archive orphan agents | scripts/maintenance/archive_test_agents.py | Pending |
+| 4 | Archive empty microDAO | scripts/maintenance/archive_test_microdao.py | Pending |
+| 5 | Update API filters | repo_city.py | Pending |
+
+---
+
+## 5. Verification Checklist
+
+- [ ] `/agents` shows only non-archived agents with node badges
+- [ ] `/citizens` shows only public, non-archived agents
+- [ ] `/microdao` shows only non-archived DAOs with agents
+- [ ] `/nodes` shows only real nodes (NODE1, NODE2)
+- [ ] Creating agent without microDAO returns 400 error
+

migrations/023_agents_add_archived.sql

@@ -0,0 +1,28 @@
+-- Migration: Add is_archived flag to agents and microdaos
+-- Purpose: Allow soft-delete/archiving of test data without physical deletion
+-- Date: 2025-11-28
+
+-- Add is_archived to agents
+ALTER TABLE agents
+    ADD COLUMN IF NOT EXISTS is_archived BOOLEAN NOT NULL DEFAULT FALSE,
+    ADD COLUMN IF NOT EXISTS archive_reason TEXT;
+
+-- Add is_archived to microdaos
+ALTER TABLE microdaos
+    ADD COLUMN IF NOT EXISTS is_archived BOOLEAN NOT NULL DEFAULT FALSE,
+    ADD COLUMN IF NOT EXISTS archive_reason TEXT;
+
+-- Add requires_microdao flag for enforcement
+ALTER TABLE agents
+    ADD COLUMN IF NOT EXISTS requires_microdao BOOLEAN NOT NULL DEFAULT TRUE;
+
+-- Create index for faster filtering
+CREATE INDEX IF NOT EXISTS idx_agents_is_archived ON agents(is_archived) WHERE is_archived = FALSE;
+CREATE INDEX IF NOT EXISTS idx_microdaos_is_archived ON microdaos(is_archived) WHERE is_archived = FALSE;
+
+-- Comment for documentation
+COMMENT ON COLUMN agents.is_archived IS 'Soft-delete flag. Archived agents are hidden from UI but not deleted.';
+COMMENT ON COLUMN agents.archive_reason IS 'Reason for archiving (e.g., "orphan_no_microdao", "test_data")';
+COMMENT ON COLUMN agents.requires_microdao IS 'If TRUE, agent must belong to at least one MicroDAO';
+COMMENT ON COLUMN microdaos.is_archived IS 'Soft-delete flag for MicroDAO';
+

services/city-service/repo_city.py

@@ -293,13 +293,14 @@ async def get_rooms_for_map() -> List[dict]:
 # =============================================================================
 
 async def get_all_agents() -> List[dict]:
-    """Отримати всіх агентів"""
+    """Отримати всіх агентів (non-archived)"""
     pool = await get_pool()
     
     query = """
         SELECT id, display_name, kind, avatar_url, color, status, 
                current_room_id, capabilities, created_at, updated_at
         FROM agents
+        WHERE COALESCE(is_archived, false) = false
         ORDER BY display_name
     """
     
@@ -317,7 +318,7 @@ async def get_agents_with_home_node(
     pool = await get_pool()
     
     params: List[Any] = []
-    where_clauses = ["1=1"]
+    where_clauses = ["COALESCE(a.is_archived, false) = false"]
     
     if kind:
         params.append(kind)
@@ -656,7 +657,7 @@ async def get_public_citizens(
     pool = await get_pool()
     
     params: List[Any] = []
-    where_clauses = ["a.is_public = true", "a.public_slug IS NOT NULL"]
+    where_clauses = ["a.is_public = true", "a.public_slug IS NOT NULL", "COALESCE(a.is_archived, false) = false"]
     
     if district:
         params.append(district)
@@ -982,7 +983,7 @@ async def get_microdaos(district: Optional[str] = None, q: Optional[str] = None,
     
     params = []
     
-    where_clauses = ["m.is_public = true", "m.is_active = true"]
+    where_clauses = ["m.is_public = true", "m.is_active = true", "COALESCE(m.is_archived, false) = false"]
     
     if district:
         params.append(district)
@@ -1043,7 +1044,7 @@ async def get_microdao_by_slug(slug: str) -> Optional[dict]:
           a.display_name as orchestrator_display_name
         FROM microdaos m
         LEFT JOIN agents a ON m.owner_agent_id = a.id
-        WHERE m.slug = $1
+        WHERE m.slug = $1 AND COALESCE(m.is_archived, false) = false
     """
     
     dao_row = await pool.fetchrow(query_dao, slug)