3 Commits

Author	SHA1	Message	Date
Apple	c4b94a327d	P2.2+P2.3: NATS offload node-worker + router offload integration ... Node Worker (services/node-worker/): - NATS subscriber for node.{NODE_ID}.llm.request / vision.request - Canonical JobRequest/JobResponse envelope (Pydantic) - Idempotency cache (TTL 10min) with inflight dedup - Deadline enforcement (DEADLINE_EXCEEDED on expired jobs) - Concurrency limiter (semaphore, returns busy) - Ollama + Swapper vision providers Router offload (services/router/offload_client.py): - NATS req/reply with configurable retries - Circuit breaker per node+type (3 fails/60s → open 120s) - Concurrency semaphore for remote requests Model selection (services/router/model_select.py): - exclude_nodes parameter for circuit-broken nodes - force_local flag for fallback re-selection - Integrated circuit breaker state awareness Router /infer pipeline: - Remote offload path when NCS selects remote node - Automatic fallback: exclude failed node → force_local re-select - Deadline propagation from router to node-worker Tests: 17 unit tests (idempotency, deadline, circuit breaker) Docs: ops/offload_routing.md (subjects, envelope, verification) Made-with: Cursor	2026-02-27 02:44:05 -08:00
Apple	a92c424845	P2: Global multi-node model selection + NCS on NODA1 ... Architecture for 150+ nodes: - global_capabilities_client.py: NATS scatter-gather discovery using wildcard subject node.*.capabilities.get — zero static node lists. New nodes auto-register by deploying NCS and subscribing to NATS. Dead nodes expire from cache after 3x TTL automatically. Multi-node model_select.py: - ModelSelection now includes node, local, via_nats fields - select_best_model prefers local candidates, then remote - Prefer list resolution: local first, remote second - All logged per request: node, runtime, model, local/remote NODA1 compose: - Added node-capabilities service (NCS) to docker-compose.node1.yml - NATS subscription: node.noda1.capabilities.get - Router env: NODE_CAPABILITIES_URL + ENABLE_GLOBAL_CAPS_NATS=true NODA2 compose: - Router env: ENABLE_GLOBAL_CAPS_NATS=true Router main.py: - Startup: initializes global_capabilities_client (NATS connect + first discovery). Falls back to local-only capabilities_client if unavailable. - /infer: uses get_global_capabilities() for cross-node model pool - Offload support: send_offload_request(node_id, type, payload) via NATS Verified on NODA2: - Global caps: 1 node, 14 models (NODA1 not yet deployed) - Sofiia: cloud_grok → grok-4-1-fast-reasoning (OK) - Helion: NCS → qwen3:14b local (OK) - When NODA1 deploys NCS, its models appear automatically via NATS discovery Made-with: Cursor	2026-02-27 02:26:12 -08:00
Apple	89c3f2ac66	P1: NCS-first model selection + NATS capabilities + Grok 4.1 ... Router model selection: - New model_select.py: resolve_effective_profile → profile_requirements → select_best_model pipeline. NCS-first with graceful static fallback. - selection_policies in router-config.node2.yml define prefer order per profile without hardcoding models (e.g. local_default_coder prefers qwen3:14b then qwen3.5:35b-a3b). - Cloud profiles (cloud_grok, cloud_deepseek) skip NCS; on cloud failure use fallback_profile via NCS for local selection. - Structured logs: selected_profile, required_type, runtime, model, caps_age_s, fallback_reason on every infer request. Grok model fix: - grok-2-1212 no longer exists on xAI API → updated to grok-4-1-fast-reasoning across all 3 hardcoded locations in main.py and router-config.node2.yml. NCS NATS request/reply: - node-capabilities subscribes to node.noda2.capabilities.get (NATS request/reply). Enabled via ENABLE_NATS_CAPS=true in compose. - NODA1 router can query NODA2 capabilities over NATS leafnode without HTTP connectivity. Verified: - NCS: 14 served models from Ollama+Swapper+llama-server - NATS: request/reply returns full capabilities JSON - Sofiia: cloud_grok → grok-4-1-fast-reasoning (tested, 200 OK) - Helion: NCS → qwen3:14b via Ollama (caps_age=23.7s cache hit) - Router health: ok Made-with: Cursor	2026-02-27 02:17:34 -08:00