P2.2+P2.3: NATS offload node-worker + router offload integration
Node Worker (services/node-worker/):
- NATS subscriber for node.{NODE_ID}.llm.request / vision.request
- Canonical JobRequest/JobResponse envelope (Pydantic)
- Idempotency cache (TTL 10min) with inflight dedup
- Deadline enforcement (DEADLINE_EXCEEDED on expired jobs)
- Concurrency limiter (semaphore, returns busy)
- Ollama + Swapper vision providers
Router offload (services/router/offload_client.py):
- NATS req/reply with configurable retries
- Circuit breaker per node+type (3 fails/60s → open 120s)
- Concurrency semaphore for remote requests
Model selection (services/router/model_select.py):
- exclude_nodes parameter for circuit-broken nodes
- force_local flag for fallback re-selection
- Integrated circuit breaker state awareness
Router /infer pipeline:
- Remote offload path when NCS selects remote node
- Automatic fallback: exclude failed node → force_local re-select
- Deadline propagation from router to node-worker
Tests: 17 unit tests (idempotency, deadline, circuit breaker)
Docs: ops/offload_routing.md (subjects, envelope, verification)
Made-with: Cursor
This commit is contained in:
47
services/node-worker/models.py
Normal file
47
services/node-worker/models.py
Normal file
@@ -0,0 +1,47 @@
|
||||
"""Canonical job envelope for cross-node inference offload."""
|
||||
from typing import Any, Dict, List, Literal, Optional
|
||||
from pydantic import BaseModel, Field
|
||||
import time
|
||||
import uuid
|
||||
|
||||
|
||||
def _ulid() -> str:
|
||||
return str(uuid.uuid4())
|
||||
|
||||
|
||||
class JobRequest(BaseModel):
|
||||
job_id: str = Field(default_factory=_ulid)
|
||||
trace_id: str = ""
|
||||
actor_agent_id: str = ""
|
||||
target_agent_id: str = ""
|
||||
required_type: Literal["llm", "vision", "stt", "tts"] = "llm"
|
||||
deadline_ts: int = 0
|
||||
idempotency_key: str = ""
|
||||
payload: Dict[str, Any] = Field(default_factory=dict)
|
||||
hints: Dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
def remaining_ms(self) -> int:
|
||||
if self.deadline_ts <= 0:
|
||||
return 30_000
|
||||
return max(0, self.deadline_ts - int(time.time() * 1000))
|
||||
|
||||
def effective_idem_key(self) -> str:
|
||||
return self.idempotency_key or self.job_id
|
||||
|
||||
|
||||
class JobError(BaseModel):
|
||||
code: str = "UNKNOWN"
|
||||
message: str = ""
|
||||
|
||||
|
||||
class JobResponse(BaseModel):
|
||||
job_id: str = ""
|
||||
trace_id: str = ""
|
||||
node_id: str = ""
|
||||
status: Literal["ok", "busy", "timeout", "error"] = "ok"
|
||||
provider: str = ""
|
||||
model: str = ""
|
||||
latency_ms: int = 0
|
||||
result: Optional[Dict[str, Any]] = None
|
||||
error: Optional[JobError] = None
|
||||
cached: bool = False
|
||||
Reference in New Issue
Block a user