P3.1: GPU/Queue-aware routing — NCS metrics + scoring-based model selection
NCS (services/node-capabilities/metrics.py): - NodeLoad: inflight_jobs, queue_depth, concurrency_limit, estimated_wait_ms, cpu_load_1m, mem_pressure (macOS + Linux), rtt_ms_to_hub - RuntimeLoad: per-runtime healthy, p50_ms, p95_ms from rolling 50-sample window - POST /capabilities/report_latency for node-worker → NCS reporting - NCS fetches worker metrics via NODE_WORKER_URL Node Worker: - GET /metrics endpoint (inflight, concurrency, latency buffers) - Latency tracking per job type (llm/vision) with rolling buffer - Fire-and-forget latency reporting to NCS after each successful job Router (model_select v3): - score_candidate(): wait + model_latency + cross_node_penalty + prefer_bonus - LOCAL_THRESHOLD_MS=250: prefer local if within threshold of remote - ModelSelection.score field for observability - Structured [score] logs with chosen node, model, and score breakdown Tests: 19 new (12 scoring + 7 NCS metrics), 36 total pass Docs: ops/runbook_p3_1.md, ops/CHANGELOG_FABRIC.md No breaking changes to JobRequest/JobResponse or capabilities schema. Made-with: Cursor
This commit is contained in:
@@ -126,6 +126,7 @@ services:
|
||||
- CACHE_TTL_SEC=15
|
||||
- ENABLE_NATS_CAPS=true
|
||||
- NATS_URL=nats://dagi-nats:4222
|
||||
- NODE_WORKER_URL=http://node-worker:8109
|
||||
depends_on:
|
||||
- swapper-service
|
||||
- dagi-nats
|
||||
@@ -150,6 +151,7 @@ services:
|
||||
- NODE_DEFAULT_LLM=qwen3:14b
|
||||
- NODE_DEFAULT_VISION=llava:13b
|
||||
- NODE_WORKER_MAX_CONCURRENCY=2
|
||||
- NCS_REPORT_URL=http://node-capabilities:8099
|
||||
depends_on:
|
||||
- dagi-nats
|
||||
- swapper-service
|
||||
|
||||
Reference in New Issue
Block a user