P2.2+P2.3: NATS offload node-worker + router offload integration
Node Worker (services/node-worker/):
- NATS subscriber for node.{NODE_ID}.llm.request / vision.request
- Canonical JobRequest/JobResponse envelope (Pydantic)
- Idempotency cache (TTL 10min) with inflight dedup
- Deadline enforcement (DEADLINE_EXCEEDED on expired jobs)
- Concurrency limiter (semaphore, returns busy)
- Ollama + Swapper vision providers
Router offload (services/router/offload_client.py):
- NATS req/reply with configurable retries
- Circuit breaker per node+type (3 fails/60s → open 120s)
- Concurrency semaphore for remote requests
Model selection (services/router/model_select.py):
- exclude_nodes parameter for circuit-broken nodes
- force_local flag for fallback re-selection
- Integrated circuit breaker state awareness
Router /infer pipeline:
- Remote offload path when NCS selects remote node
- Automatic fallback: exclude failed node → force_local re-select
- Deadline propagation from router to node-worker
Tests: 17 unit tests (idempotency, deadline, circuit breaker)
Docs: ops/offload_routing.md (subjects, envelope, verification)
Made-with: Cursor
This commit is contained in:
53
services/node-worker/main.py
Normal file
53
services/node-worker/main.py
Normal file
@@ -0,0 +1,53 @@
|
||||
"""Node Worker — NATS offload executor for cross-node inference."""
|
||||
import logging
|
||||
import os
|
||||
|
||||
from fastapi import FastAPI
|
||||
|
||||
import config
|
||||
import worker
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger("node-worker")
|
||||
|
||||
app = FastAPI(title="Node Worker", version="1.0.0")
|
||||
|
||||
_nats_client = None
|
||||
|
||||
|
||||
@app.get("/healthz")
|
||||
async def healthz():
|
||||
connected = _nats_client is not None and _nats_client.is_connected if _nats_client else False
|
||||
return {
|
||||
"status": "ok" if connected else "degraded",
|
||||
"node_id": config.NODE_ID,
|
||||
"nats_connected": connected,
|
||||
"max_concurrency": config.MAX_CONCURRENCY,
|
||||
}
|
||||
|
||||
|
||||
@app.on_event("startup")
|
||||
async def startup():
|
||||
global _nats_client
|
||||
try:
|
||||
import nats as nats_lib
|
||||
_nats_client = await nats_lib.connect(config.NATS_URL)
|
||||
logger.info(f"✅ NATS connected: {config.NATS_URL}")
|
||||
await worker.start(_nats_client)
|
||||
logger.info(f"✅ Node Worker ready: node={config.NODE_ID} concurrency={config.MAX_CONCURRENCY}")
|
||||
except Exception as e:
|
||||
logger.error(f"❌ Startup failed: {e}")
|
||||
|
||||
|
||||
@app.on_event("shutdown")
|
||||
async def shutdown():
|
||||
if _nats_client:
|
||||
try:
|
||||
await _nats_client.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=config.PORT)
|
||||
Reference in New Issue
Block a user