P3.2+P3.3+P3.4: NODA1 node-worker + NATS auth config + Prometheus counters
P3.2 — Multi-node deployment: - Added node-worker service to docker-compose.node1.yml (NODE_ID=noda1) - NCS NODA1 now has NODE_WORKER_URL for metrics collection - Fixed NODE_ID consistency: router NODA1 uses 'noda1' - NODA2 node-worker/NCS gets NCS_REPORT_URL for latency reporting P3.3 — NATS accounts/auth (opt-in config): - config/nats-server.conf with 3 accounts: SYS, FABRIC, APP - Per-user topic permissions (router, ncs, node_worker) - Leafnode listener :7422 with auth - Not yet activated (requires credential provisioning) P3.4 — Prometheus counters: - Router /fabric_metrics: caps_refresh, caps_stale, model_select, offload_total, breaker_state, score_ms histogram - Node Worker /prom_metrics: jobs_total, inflight gauge, latency_ms histogram - NCS /prom_metrics: runtime_health, runtime_p50/p95, node_wait_ms - All bound to 127.0.0.1 (not externally exposed) Made-with: Cursor
This commit is contained in:
@@ -10,6 +10,7 @@ import config
|
||||
from models import JobRequest, JobResponse, JobError
|
||||
from idempotency import IdempotencyStore
|
||||
from providers import ollama, swapper_vision
|
||||
import fabric_metrics as fm
|
||||
|
||||
logger = logging.getLogger("node-worker")
|
||||
|
||||
@@ -95,17 +96,21 @@ async def _handle_request(msg):
|
||||
|
||||
global _inflight_count
|
||||
_inflight_count += 1
|
||||
fm.set_inflight(_inflight_count)
|
||||
try:
|
||||
async with _semaphore:
|
||||
resp = await _execute(job, remaining)
|
||||
finally:
|
||||
_inflight_count -= 1
|
||||
fm.set_inflight(_inflight_count)
|
||||
|
||||
_idem.put(idem_key, resp)
|
||||
_idem.complete_inflight(idem_key, resp)
|
||||
resp.latency_ms = int((time.time() - t0) * 1000)
|
||||
|
||||
fm.inc_job(job.required_type, resp.status)
|
||||
if resp.status == "ok" and resp.latency_ms > 0:
|
||||
fm.observe_latency(job.required_type, resp.model or "?", resp.latency_ms)
|
||||
buf = _latencies_llm if job.required_type in ("llm", "code") else _latencies_vision
|
||||
buf.append(resp.latency_ms)
|
||||
if len(buf) > _LATENCY_BUFFER:
|
||||
|
||||
Reference in New Issue
Block a user