feat: MD pipeline — market-data-service hardening + SenpAI NATS consumer
Producer (market-data-service):
- Backpressure: smart drop policy (heartbeats→quotes→trades preserved)
- Heartbeat monitor: synthetic HeartbeatEvent on provider silence
- Graceful shutdown: WS→bus→storage→DB engine cleanup sequence
- Bybit V5 public WS provider (backup for Binance, no API key needed)
- FailoverManager: health-based provider switching with recovery
- NATS output adapter: md.events.{type}.{symbol} for SenpAI
- /bus-stats endpoint for backpressure monitoring
- Dockerfile + docker-compose.node1.yml integration
- 36 tests (parsing + bus + failover), requirements.lock
Consumer (senpai-md-consumer):
- NATSConsumer: subscribe md.events.>, queue group senpai-md, backpressure
- State store: LatestState + RollingWindow (deque, 60s)
- Feature engine: 11 features (mid, spread, VWAP, return, vol, latency)
- Rule-based signals: long/short on return+volume+spread conditions
- Publisher: rate-limited features + signals + alerts to NATS
- HTTP API: /health, /metrics, /state/latest, /features/latest, /stats
- 10 Prometheus metrics
- Dockerfile + docker-compose.senpai.yml
- 41 tests (parsing + state + features + rate-limit), requirements.lock
CI: ruff + pytest + smoke import for both services
Tests: 77 total passed, lint clean
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
40
services/senpai-md-consumer/docker-compose.senpai.yml
Normal file
40
services/senpai-md-consumer/docker-compose.senpai.yml
Normal file
@@ -0,0 +1,40 @@
|
||||
# SenpAI Market-Data Consumer + NATS
|
||||
# Usage: docker-compose -f docker-compose.senpai.yml up -d
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
nats:
|
||||
image: nats:2.10-alpine
|
||||
container_name: senpai-nats
|
||||
ports:
|
||||
- "4222:4222"
|
||||
- "8222:8222" # monitoring
|
||||
command: ["--js", "-m", "8222"]
|
||||
restart: unless-stopped
|
||||
|
||||
senpai-md-consumer:
|
||||
container_name: senpai-md-consumer
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
environment:
|
||||
- NATS_URL=nats://nats:4222
|
||||
- NATS_SUBJECT=md.events.>
|
||||
- NATS_QUEUE_GROUP=senpai-md
|
||||
- FEATURES_ENABLED=true
|
||||
- FEATURES_PUB_RATE_HZ=10
|
||||
- LOG_LEVEL=INFO
|
||||
- HTTP_PORT=8892
|
||||
ports:
|
||||
- "8892:8892"
|
||||
depends_on:
|
||||
- nats
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test:
|
||||
- CMD-SHELL
|
||||
- python -c "import urllib.request; urllib.request.urlopen('http://localhost:8892/health')"
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
Reference in New Issue
Block a user