feat: MD pipeline — market-data-service hardening + SenpAI NATS consumer

Producer (market-data-service):
- Backpressure: smart drop policy (heartbeats→quotes→trades preserved)
- Heartbeat monitor: synthetic HeartbeatEvent on provider silence
- Graceful shutdown: WS→bus→storage→DB engine cleanup sequence
- Bybit V5 public WS provider (backup for Binance, no API key needed)
- FailoverManager: health-based provider switching with recovery
- NATS output adapter: md.events.{type}.{symbol} for SenpAI
- /bus-stats endpoint for backpressure monitoring
- Dockerfile + docker-compose.node1.yml integration
- 36 tests (parsing + bus + failover), requirements.lock

Consumer (senpai-md-consumer):
- NATSConsumer: subscribe md.events.>, queue group senpai-md, backpressure
- State store: LatestState + RollingWindow (deque, 60s)
- Feature engine: 11 features (mid, spread, VWAP, return, vol, latency)
- Rule-based signals: long/short on return+volume+spread conditions
- Publisher: rate-limited features + signals + alerts to NATS
- HTTP API: /health, /metrics, /state/latest, /features/latest, /stats
- 10 Prometheus metrics
- Dockerfile + docker-compose.senpai.yml
- 41 tests (parsing + state + features + rate-limit), requirements.lock

CI: ruff + pytest + smoke import for both services
Tests: 77 total passed, lint clean
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Apple
2026-02-09 11:46:15 -08:00
parent c50843933f
commit 09dee24342
47 changed files with 3930 additions and 56 deletions

View File

@@ -19,6 +19,9 @@ class Settings(BaseSettings):
binance_ws_url: str = "wss://stream.binance.com:9443/ws"
binance_rest_url: str = "https://api.binance.com"
# ── Bybit (backup crypto — no key needed) ──────────────────────────
bybit_ws_url: str = "wss://stream.bybit.com/v5/public/spot"
# ── Alpaca (paper trading — free tier) ─────────────────────────────
alpaca_key: str = ""
alpaca_secret: str = ""
@@ -41,6 +44,11 @@ class Settings(BaseSettings):
http_port: int = 8891
metrics_enabled: bool = True
# ── NATS output adapter ─────────────────────────────────────────────
nats_url: str = "" # e.g. "nats://localhost:4222"
nats_subject_prefix: str = "md.events" # → md.events.trade.BTCUSDT
nats_enabled: bool = False
# ── Logging ────────────────────────────────────────────────────────
log_level: str = "INFO"
log_sample_rate: int = 100 # PrintConsumer: log 1 out of N events
@@ -49,5 +57,9 @@ class Settings(BaseSettings):
def alpaca_configured(self) -> bool:
return bool(self.alpaca_key and self.alpaca_secret)
@property
def nats_configured(self) -> bool:
return bool(self.nats_url and self.nats_enabled)
settings = Settings()