# Market Data Service (SenpAI)

Real-time market data collection and normalization for the SenpAI/Gordon trading agent.

## Quick Start

### 1. Install

```bash
cd services/market-data-service
pip install -r requirements.txt
```

### 2. Copy config

```bash
cp .env.example .env
```

### 3. Run (Binance — no keys needed)

```bash
python -m app run --provider binance --symbols BTCUSDT,ETHUSDT
```

### 4. Run (Alpaca — paper trading)

First, get free paper-trading API keys:
1. Sign up at https://app.alpaca.markets
2. Switch to **Paper Trading** in the dashboard
3. Go to API Keys → Generate New Key
4. Add to `.env`:
   ```
   ALPACA_KEY=your_key_here
   ALPACA_SECRET=your_secret_here
   ALPACA_DRY_RUN=false
   ```
5. Run:
   ```bash
   python -m app run --provider alpaca --symbols AAPL,TSLA
   ```

Without keys, Alpaca runs in **dry-run mode** (heartbeats only).

### 5. Run both providers

```bash
python -m app run --provider all --symbols BTCUSDT,AAPL
```

## HTTP Endpoints

Once running, the service exposes:

| Endpoint | Description |
|---|---|
| `GET /health` | Service health check |
| `GET /metrics` | Prometheus metrics |
| `GET /latest?symbol=BTCUSDT` | Latest trade + quote from SQLite |

Default port: `8891` (configurable via `HTTP_PORT`).

## View Data

### SQLite
```bash
sqlite3 market_data.db "SELECT * FROM trades ORDER BY ts_recv DESC LIMIT 5;"
```

### JSONL Event Log
```bash
tail -5 events.jsonl | python -m json.tool
```

### Prometheus Metrics
```bash
curl http://localhost:8891/metrics
```

Key metrics:
- `market_events_total` — events by provider/type/symbol
- `market_exchange_latency_ms` — exchange-to-receive latency
- `market_events_per_second` — throughput gauge

## Architecture

```
Provider (Binance/Alpaca)
    │ raw WebSocket messages
    ▼
Adapter (_parse → domain Event)
    │ TradeEvent / QuoteEvent / BookL2Event
    ▼
EventBus (asyncio.Queue fan-out)
    ├─▶ StorageConsumer  → SQLite + JSONL
    ├─▶ MetricsConsumer  → Prometheus counters/histograms
    └─▶ PrintConsumer    → structured log (sampled 1/100)
```

## Adding a New Provider

1. Create `app/providers/your_provider.py`
2. Subclass `MarketDataProvider`:

```python
from app.providers import MarketDataProvider
from app.domain.events import Event, TradeEvent

class YourProvider(MarketDataProvider):
    name = "your_provider"

    async def connect(self) -> None:
        # Establish connection
        ...

    async def subscribe(self, symbols: list[str]) -> None:
        # Subscribe to streams
        ...

    async def stream(self) -> AsyncIterator[Event]:
        # Yield normalized events, handle reconnect
        while True:
            raw = await self._receive()
            yield self._parse(raw)

    async def close(self) -> None:
        ...
```

3. Register in `app/providers/__init__.py`:
```python
from app.providers.your_provider import YourProvider

registry = {
    ...
    "your_provider": YourProvider,
}
```

4. Run: `python -m app run --provider your_provider --symbols ...`

## Tests

```bash
pytest tests/ -v
```

## TODO: Future Providers

- [ ] CoinAPI (REST + WebSocket, paid tier)
- [ ] IQFeed (US equities, DTN subscription)
- [ ] Polygon.io (real-time + historical)
- [ ] Interactive Brokers TWS API