feat: MD pipeline — market-data-service hardening + SenpAI NATS consumer
Producer (market-data-service):
- Backpressure: smart drop policy (heartbeats→quotes→trades preserved)
- Heartbeat monitor: synthetic HeartbeatEvent on provider silence
- Graceful shutdown: WS→bus→storage→DB engine cleanup sequence
- Bybit V5 public WS provider (backup for Binance, no API key needed)
- FailoverManager: health-based provider switching with recovery
- NATS output adapter: md.events.{type}.{symbol} for SenpAI
- /bus-stats endpoint for backpressure monitoring
- Dockerfile + docker-compose.node1.yml integration
- 36 tests (parsing + bus + failover), requirements.lock
Consumer (senpai-md-consumer):
- NATSConsumer: subscribe md.events.>, queue group senpai-md, backpressure
- State store: LatestState + RollingWindow (deque, 60s)
- Feature engine: 11 features (mid, spread, VWAP, return, vol, latency)
- Rule-based signals: long/short on return+volume+spread conditions
- Publisher: rate-limited features + signals + alerts to NATS
- HTTP API: /health, /metrics, /state/latest, /features/latest, /stats
- 10 Prometheus metrics
- Dockerfile + docker-compose.senpai.yml
- 41 tests (parsing + state + features + rate-limit), requirements.lock
CI: ruff + pytest + smoke import for both services
Tests: 77 total passed, lint clean
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
32
services/market-data-service/Dockerfile
Normal file
32
services/market-data-service/Dockerfile
Normal file
@@ -0,0 +1,32 @@
|
||||
# ── Market Data Service ─────────────────────────────────────────────────
|
||||
# Multi-stage build: slim Python 3.11+ image
|
||||
FROM python:3.11-slim AS base
|
||||
|
||||
# Prevent Python from writing bytecode and enable unbuffered output
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# System dependencies (none needed for this service)
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# ── Dependencies ───────────────────────────────────────────────────────
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
# ── Application code ──────────────────────────────────────────────────
|
||||
COPY app/ ./app/
|
||||
COPY pyproject.toml .
|
||||
|
||||
# ── Health check ──────────────────────────────────────────────────────
|
||||
HEALTHCHECK --interval=15s --timeout=5s --start-period=10s --retries=3 \
|
||||
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8891/health')" || exit 1
|
||||
|
||||
# ── Default command ───────────────────────────────────────────────────
|
||||
# Override with docker-compose command or CLI args
|
||||
EXPOSE 8891
|
||||
|
||||
ENTRYPOINT ["python", "-m", "app"]
|
||||
CMD ["run", "--provider", "binance", "--symbols", "BTCUSDT,ETHUSDT"]
|
||||
Reference in New Issue
Block a user