Commit Graph

169 Commits

Author SHA1 Message Date
Apple
b2be937fbb feat(file-tool): add djvu conversion and extraction actions 2026-02-15 03:11:55 -08:00
Apple
3a565fd910 feat(file-tool): harden svg rendering and add rich pptx/pdf updates 2026-02-15 02:48:35 -08:00
Apple
aad5870e81 feat(file-tool): add image_bundle and svg actions 2026-02-15 02:33:42 -08:00
Apple
36314a871f feat(file-tool): add pptx ods parquet and image actions 2026-02-15 02:30:00 -08:00
Apple
cf6ac778bb feat(file-tool): add text markdown xml html actions 2026-02-15 02:24:11 -08:00
Apple
e91584246d feat(router): implement file_tool excel actions on NODE1 stack 2026-02-15 02:11:28 -08:00
Apple
21576f0ca3 node1: add universal file tool, gateway document delivery, and sync runbook 2026-02-15 01:50:37 -08:00
Apple
25e57d8221 feat: Add valid ComfyUI SD1.5 workflow to comfy-agent
- Replace placeholder workflow with complete SD1.5 pipeline
- Support dynamic prompt, negative_prompt, steps, seed, width, height
- Nodes: CheckpointLoader -> CLIP -> KSampler -> VAE -> SaveImage

Co-Authored-By: Warp <agent@warp.dev>
2026-02-10 04:39:40 -08:00
Apple
42599787a6 chore(helion): respond to direct mentions in groups
Clarify Helion group behavior: stay silent unless energy topic or direct mention, but answer operational questions when directly addressed.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-10 04:20:03 -08:00
Apple
7f3ee700a4 fix(router): guard DSML tool-call flows
Prevent DeepSeek DSML from leaking to users and avoid returning raw memory_search/web results when DSML is detected.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-10 04:19:57 -08:00
Apple
c41c68dc08 feat: Add Comfy Agent service for NODE3 image/video generation
- Create comfy-agent service with FastAPI + NATS integration
- ComfyUI client with HTTP/WebSocket support
- REST API: /generate/image, /generate/video, /status, /result
- NATS subjects: agent.invoke.comfy, comfy.request.*
- Async job queue with progress tracking
- Docker compose configuration for NODE3
- Update PROJECT-MASTER-INDEX.md with NODE2/NODE3 docs

Co-Authored-By: Warp <agent@warp.dev>
2026-02-10 04:13:49 -08:00
Apple
09dee24342 feat: MD pipeline — market-data-service hardening + SenpAI NATS consumer
Producer (market-data-service):
- Backpressure: smart drop policy (heartbeats→quotes→trades preserved)
- Heartbeat monitor: synthetic HeartbeatEvent on provider silence
- Graceful shutdown: WS→bus→storage→DB engine cleanup sequence
- Bybit V5 public WS provider (backup for Binance, no API key needed)
- FailoverManager: health-based provider switching with recovery
- NATS output adapter: md.events.{type}.{symbol} for SenpAI
- /bus-stats endpoint for backpressure monitoring
- Dockerfile + docker-compose.node1.yml integration
- 36 tests (parsing + bus + failover), requirements.lock

Consumer (senpai-md-consumer):
- NATSConsumer: subscribe md.events.>, queue group senpai-md, backpressure
- State store: LatestState + RollingWindow (deque, 60s)
- Feature engine: 11 features (mid, spread, VWAP, return, vol, latency)
- Rule-based signals: long/short on return+volume+spread conditions
- Publisher: rate-limited features + signals + alerts to NATS
- HTTP API: /health, /metrics, /state/latest, /features/latest, /stats
- 10 Prometheus metrics
- Dockerfile + docker-compose.senpai.yml
- 41 tests (parsing + state + features + rate-limit), requirements.lock

CI: ruff + pytest + smoke import for both services
Tests: 77 total passed, lint clean
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 11:46:15 -08:00
Apple
c50843933f feat: market-data-service for SenpAI trading agent
New service: real-time market data collection with unified event model.

Architecture:
- Domain events: TradeEvent, QuoteEvent, BookL2Event, HeartbeatEvent
- Provider interface: MarketDataProvider ABC with connect/subscribe/stream/close
- Async EventBus with fan-out to multiple consumers

Providers:
- BinanceProvider: public WebSocket (trades + bookTicker), no API key needed,
  auto-reconnect with exponential backoff, heartbeat timeout detection
- AlpacaProvider: IEX real-time data + paper trading auth,
  dry-run mode when no keys configured (heartbeats only)

Consumers:
- StorageConsumer: SQLite (via SQLAlchemy async) + JSONL append-only log
- MetricsConsumer: Prometheus counters, latency histograms, events/sec gauge
- PrintConsumer: sampled structured logging (1/100 events)

CLI: python -m app run --provider binance --symbols BTCUSDT,ETHUSDT
HTTP: /health, /metrics (Prometheus), /latest?symbol=XXX

Tests: 19/19 passed (Binance parse, Alpaca parse, bus smoke tests)

Config: pydantic-settings + .env, all secrets via environment variables.
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 11:19:00 -08:00
Apple
7887f7cbe9 fix: DSML fallback — 3rd LLM call for clean synthesis + think tag stripping
Router (main.py):
- When DSML detected in 2nd LLM response after tool execution,
  make a 3rd LLM call with explicit synthesis prompt instead of
  returning raw tool results to the user
- Falls back to format_tool_calls_for_response only if 3rd call fails

Router (tool_manager.py):
- Added _strip_think_tags() helper for <think>...</think> removal
  from DeepSeek reasoning artifacts

Gateway (http_api.py):
- Strip <think>...</think> tags before sending to Telegram
- Strip DSML/XML-like markup (function_calls, invoke, parameter tags)
- Ensure empty text after stripping gets "..." fallback

Deployed to NODE1 and verified services running.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 10:30:37 -08:00
Apple
990e594a1d feat: harden memory summary — fingerprint dedup, versioning, prompt injection defense
Summary hardening:
- SHA256 fingerprint of events content for deduplication
  (skips LLM call when events unchanged since last summary)
- Versioned summary storage: summary:agent:channel:vN keys
- Latest pointer: summary_latest:agent:channel for fast retrieval
- Prompt injection defense: sanitize event content before LLM,
  strip [SYSTEM]/[INTERNAL] markers, block "ignore instructions" patterns
- Anti-injection clause in SUMMARY_SYSTEM_PROMPT

Database fix:
- list_facts_by_agent: SQL filter by fact_prefix to only return chat_events
  (prevents summary/version facts from consuming LIMIT quota)
- Fixed NULL team_id issue in UNIQUE constraint (PostgreSQL NULL != NULL)
  using "__system__" sentinel for team_id in summary operations

Tested on NODE1: dedup works (same events → skipped), force=true bypasses.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 10:26:03 -08:00
Apple
0cfd3619ea feat: auto-summarize trigger for agent memory
- Memory Service: POST /agents/{agent_id}/summarize endpoint
  - Fetches recent events by agent_id (new db.list_facts_by_agent)
  - Generates structured summary via DeepSeek LLM
  - Saves summary to PostgreSQL facts + Qdrant vector store
  - Returns structured JSON (summary, goals, decisions, key_facts)

- Gateway memory_client: auto-trigger after 30 turns
  - Turn counter per chat (agent_id:channel_id)
  - 5-minute debounce between summarize calls
  - Fire-and-forget via asyncio.ensure_future (non-blocking)
  - Configurable via SUMMARIZE_TURN_THRESHOLD / SUMMARIZE_DEBOUNCE_SECONDS

- Database: list_facts_by_agent() for agent-level queries without user_id

Tested on NODE1: Helion summarize returns valid Ukrainian summary with 20 events.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 10:15:43 -08:00
Apple
acceac6929 fix: helion string literal + memory brief anti-echo in Router
- Fixed unquoted `helion` variable reference to string literal `"helion"`
  in tool_manager.py search_memories fallback
- Replaced `[Контекст пам'яті]` with `[INTERNAL MEMORY - do NOT repeat
  to user]` in all 3 injection points in main.py
- Verified: Senpai now responds without Helion contamination or memory
  brief leaking

Tested and deployed on NODE1.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 10:05:25 -08:00
Apple
b9f7ca8ecf fix(critical): Senpai using Helion's memory — 3 root causes fixed
1. YAML structure bug: Senpai was in `policies:` instead of `agents:`
   in router-config.yml. Router couldn't find Senpai config → no routing
   rule → fallback to local model.

2. tool_manager agent_id not passed: memory_search and graph_query
   tools were called without agent_id → defaulted to "helion" →
   ALL agents' tool calls searched Helion's Qdrant collections.
   Fixed: agent_id now flows from main.py → execute_tool → _memory_search.

3. Config not mounted: router-config.yml was baked into Docker image,
   host changes had no effect. Added volume mount in docker-compose.

Also added:
- Sofiia agent config + routing rule (was completely missing)
- Senpai routing rule: cloud_deepseek (was falling to local qwen3:8b)
- Anti-echo instruction for memory brief injection

Deployed and verified on NODE1: Senpai now searches senpai_* collections.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 10:00:08 -08:00
Apple
3b924118be fix: quarantine dead brand commands + implement Memory LLM summary
Brand commands (~290 lines):
- Code was trapped inside `if reply_to_message:` block (unreachable)
- Moved to feature flag: ENABLE_BRAND_COMMANDS=true to activate
- Zero re-indentation: 8sp code naturally fits as feature flag body
- Helper functions (_brand_*, _artifact_*) unchanged

Memory LLM Summary:
- Replace placeholder with real DeepSeek API integration
- Structured output: summary, goals, decisions, open_questions, next_steps, key_facts
- Graceful fallback if API key not set or call fails
- Added MEMORY_DEEPSEEK_API_KEY config
- Ukrainian output language

Deployed and verified on NODE1.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 09:42:44 -08:00
Apple
ef3473db21 snapshot: NODE1 production state 2026-02-09
Complete snapshot of /opt/microdao-daarion/ from NODE1 (144.76.224.179).
This represents the actual running production code that has diverged
significantly from the previous main branch.

Key changes from old main:
- Gateway (http_api.py): expanded from ~40KB to 164KB with full agent support
- Router: new /v1/agents/{id}/infer endpoint with vision + DeepSeek routing
- Behavior Policy: SOWA v2.2 (3-level: FULL/ACK/SILENT)
- Agent Registry: config/agent_registry.yml as single source of truth
- 13 agents configured (was 3)
- Memory service integration
- CrewAI teams and roles

Excluded from snapshot: venv/, .env, data/, backups, .tgz archives

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 08:46:46 -08:00
Apple
6b54e0da6d fix(router): Replace requests with urllib in healthcheck
- Use stdlib urllib.request instead of requests library
- requests was not installed in the router image, causing healthcheck
  to always fail with "ModuleNotFoundError: No module named 'requests'"
- Increase start_period to 30s and retries to 5 for stability

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-03 05:56:02 -08:00
Apple
a3923cd96f feat(P0/P1/P2): Add E2E agent prober, version pinning, prometheus fixes 2026-01-28 07:06:07 -08:00
Apple
0c8bef82f4 feat: Add Alateya, Clan, Eonarch agents + fix gateway-router connection
## Agents Added
- Alateya: R&D, biotech, innovations
- Clan (Spirit): Community spirit agent
- Eonarch: Consciousness evolution agent

## Changes
- docker-compose.node1.yml: Added tokens for all 3 new agents
- gateway-bot/http_api.py: Added configs and webhook endpoints
- gateway-bot/clan_prompt.txt: New prompt file
- gateway-bot/eonarch_prompt.txt: New prompt file

## Fixes
- Fixed ROUTER_URL from :9102 to :8000 (internal container port)
- All 9 Telegram agents now working

## Documentation
- Created PROJECT-MASTER-INDEX.md - single entry point
- Added various status documents and scripts

Tokens configured:
- Helion, NUTRA, Agromatrix (existing)
- Alateya, Clan, Eonarch (new)
- Druid, GreenFood, DAARWIZZ (configured)
2026-01-28 06:40:34 -08:00
Apple
5290287058 feat: implement TTS, Document processing, and Memory Service /facts API
- TTS: xtts-v2 integration with voice cloning support
- Document: docling integration for PDF/DOCX/PPTX processing
- Memory Service: added /facts/upsert, /facts/{key}, /facts endpoints
- Added required dependencies (TTS, docling)
2026-01-17 08:16:37 -08:00
Apple
38cb96dd68 🔐 Auth: інтеграція JWT в Memory Service + конфігурації
- Опціональна JWT auth в Memory Service endpoints
- get_current_service_optional для backward compatibility
- NATS auth config (nkeys) - шаблони
- Qdrant auth config (API keys) - шаблони
- Тестовий скрипт для повного потоку

TODO: Генерація реальних JWT/ключів та застосування конфігів
2026-01-10 10:46:25 -08:00
Apple
6c426bc274 🔐 Auth: базова реалізація JWT для Memory Service
- JWT middleware для FastAPI
- Генерація/перевірка JWT токенів
- Скрипти для генерації Qdrant API keys
- Скрипти для генерації NATS operator JWT
- План реалізації Auth

TODO: Додати JWT до endpoints, NATS nkeys config, Qdrant API key config
2026-01-10 10:43:14 -08:00
Apple
3478dfce5f 🔒 КРИТИЧНО: Видалено паролі/API ключі з документів + закрито NodePort
Some checks failed
Build and Deploy Docs / build-and-deploy (push) Has been cancelled
- Видалено всі паролі та API ключі з документів
- Замінено на посилання на Vault
- Закрито NodePort для Memory Service (тільки internal)
- Створено SECURITY-ROTATION-PLAN.md
- Створено ARCHITECTURE-150-NODES.md (план для 150 нод)
- Оновлено config.py (видалено hardcoded Cohere key)
2026-01-10 09:46:03 -08:00
Apple
90758facae 🧠 Add Agent Memory System with PostgreSQL + Qdrant + Cohere
Features:
- Three-tier memory architecture (short/mid/long-term)
- PostgreSQL schema for conversations, events, memories
- Qdrant vector database for semantic search
- Cohere embeddings (embed-multilingual-v3.0, 1024 dims)
- FastAPI Memory Service with full CRUD
- External Secrets integration with Vault
- Kubernetes deployment manifests

Components:
- infrastructure/database/agent-memory-schema.sql
- infrastructure/kubernetes/apps/qdrant/
- infrastructure/kubernetes/apps/memory-service/
- services/memory-service/ (FastAPI app)

Also includes:
- External Secrets Operator
- Traefik Ingress Controller
- Cert-Manager with Let's Encrypt
- ArgoCD for GitOps
2026-01-10 07:52:32 -08:00
Apple
744c149300 Add automated session logging system
Some checks failed
Build and Deploy Docs / build-and-deploy (push) Has been cancelled
- Created logs/ structure (sessions, operations, incidents)
- Added session-start/log/end scripts
- Installed Git hooks for auto-logging commits/pushes
- Added shell integration for zsh
- Created CHANGELOG.md
- Documented today's session (2026-01-10)
2026-01-10 04:53:17 -08:00
Apple
33fcb04f65 fix: Make Redis optional for city rooms online count
- Handle Redis connection errors gracefully
- Return rooms even if Redis is unavailable
- This fixes 500 error on /api/city/rooms endpoint
2025-12-05 03:18:58 -08:00
Apple
1f97586699 fix: Handle missing crew_team_key column in agents query
- Remove crew_team_key from SELECT (column doesn't exist)
- Use pop() to safely handle crew_team_key in data processing
- This fixes 500 error on /api/agents/list endpoint
2025-12-05 02:52:14 -08:00
Apple
72b76bf29f fix: Remove non-existent owner_type/owner_id columns from city_rooms queries
- Fix get_all_rooms() to not select owner_type/owner_id
- Fix get_city_rooms_for_list() to not select owner_type/owner_id
- Fix get_city_rooms_api() to use space_scope instead of owner_type
- This fixes 500 error on /api/city/rooms endpoint
2025-12-05 02:52:03 -08:00
Apple
7ac64c3183 fix: Add banner_url to MicrodaoDetail response
- Add missing banner_url field when creating MicrodaoDetail
- This fixes issue where banner_url was saved in DB but not returned by /api/microdao/{slug} endpoint
2025-12-02 09:19:36 -08:00
Apple
62f03f0dad fix: Use api_route for HEAD method support in assets proxy 2025-12-02 08:48:13 -08:00
Apple
d13115e3b0 fix: Fix Request import for HEAD method support in assets proxy 2025-12-02 08:47:45 -08:00
Apple
192631c2eb fix: Add HEAD method support to assets proxy endpoint
- Add HEAD method handler for browser preflight requests
- Use stat_object for HEAD requests (more efficient)
- Return proper headers for HEAD requests
- This fixes 405 errors when browser checks image availability
2025-12-02 08:47:02 -08:00
Apple
1ca6a4f55a feat: Complete assets proxy implementation with documentation
- Add comprehensive documentation in docs/ASSETS_PROXY.md
- Add contract comments in normalizeAssetUrl and proxy_asset
- Verify all components use normalizeAssetUrl
- Verify ENV variables are correctly set
- Add troubleshooting guide
2025-12-02 08:36:55 -08:00
Apple
517efc6a16 fix: Add API proxy for MinIO assets to work without assets.daarion.space DNS
- Add /api/assets/[...path] proxy route in Next.js
- Add /assets/proxy/{path} endpoint in city-service
- Update normalizeAssetUrl to convert assets.daarion.space URLs to /api/assets/...
- This allows assets to work even if DNS for assets.daarion.space is not configured
2025-12-02 07:43:36 -08:00
Apple
fca48b3eb0 feat(node2): Complete NODE2 setup - guardian, agents, swapper models
- Node-guardian running on MacBook and updating metrics
- NODE2 agents (Atlas, Greeter, Oracle, Builder Bot) assigned to node-2-macbook-m4max
- Swapper models displaying correctly (8 models)
- DAGI Router agents showing with correct status (3 active, 1 stale)
- Router health check using node_cache for remote nodes
2025-12-02 07:07:58 -08:00
Apple
240ceba2e8 debug(node2): Change logging to WARNING level for router_healthy 2025-12-02 07:05:54 -08:00
Apple
50c0ac97dd debug(node2): Add logging for agent status determination 2025-12-02 07:05:16 -08:00
Apple
08cb8c52cc fix(node2): Add detailed logging for router_healthy determination 2025-12-02 07:03:50 -08:00
Apple
90ebf32de3 fix(node2): Fix router_healthy initialization in get_dagi_router_agents
- Initialize router_healthy as None instead of False
- Use bool() to ensure proper boolean conversion
- Add info logging for debugging
- This ensures cached router_healthy=True is properly used
2025-12-02 07:03:01 -08:00
Apple
88188ed693 fix(node2): Use node_cache router_healthy for DAGI Router agents status
- Fix get_dagi_router_agents to use router_healthy from node_cache first
- Fallback to direct API call only if cache is unavailable
- This fixes NODE2 agents showing as 'stale' when router is actually healthy
- Fix CITY_SERVICE_URL in scripts (remove /api/city, use /api)
2025-12-02 07:02:08 -08:00
Apple
ace183e136 feat: Add MicroDAO Dashboard with activity feed and statistics
- Add microdao_activity table for news/updates/events
- Add statistics columns to microdaos table
- Implement dashboard API endpoints
- Create UI components (HeaderCard, ActivitySection, TeamSection)
- Add seed data for DAARION DAO
- Update backend models and repositories
- Add frontend types and API client
2025-12-02 06:37:16 -08:00
Apple
95c9a17a7a fix(models): Add MicrodaoActivity and related classes to models_city 2025-12-02 03:21:20 -08:00
Apple
973b58f08f fix(city-service): Fix logger initialization order 2025-12-02 03:15:27 -08:00
Apple
f95810e8a7 fix(nodes): Normalize Router/Swapper endpoints and fix NODE2 display
Major changes:
- Normalize get_node_endpoints to use ENV vars (ROUTER_BASE_URL, SWAPPER_BASE_URL)
- Remove node_id-based URL selection logic
- Add fallback direct API call in get_node_swapper_detail
- Fix Swapper API endpoint (/models instead of /api/v1/models)
- Add router_healthy and router_version to node_heartbeat fallback
- Add ENV vars to docker-compose for Router/Swapper URLs

Documentation:
- Add TASK_PHASE_NODE2_ROUTER_SWAPPER_FIX.md with full task description
- Add NODE2_GUARDIAN_SETUP.md with setup instructions

This fixes:
- Swapper models not showing for NODE1 and NODE2
- DAGI Router agents not showing for NODE2
- Router/Swapper showing as Down/Degraded when they're actually up
2025-12-02 03:13:01 -08:00
Apple
5f07a6b3ae fix(nodes): Fix Swapper models and DAGI Router agents display for NODE2
- Fix get_node_endpoints to correctly determine URLs for NODE2 (localhost instead of Docker service names)
- Fix swapper detail endpoint to return fallback data instead of 404 when metrics not found
- This allows UI to show pending state instead of error for NODE2

Fixes:
- Swapper Service models not showing for NODE2
- DAGI Router agents not showing for NODE2
2025-12-02 02:49:02 -08:00
Apple
d24a23ec96 fix(db-hardening): Add lib __init__.py and improve MinIO import error handling 2025-12-02 01:57:27 -08:00