microdao-daarion/docs/RUNBOOK_NODE1_RECOVERY_SAFETY.md

# Runbook: NODE1 Recovery & Safety

## Purpose
Швидко відновити роботу NODE1 після збоїв (Telegram webhook 500, router DNS, NATS/worker, Grafana crash-loop) і уникнути випадкового зупинення не того стеку.

## Quick links / aliases
- `./stack-node1 ps|up|down|logs` (node1 stack)
- `./stack-staging ps|up|down|logs` (staging stack)
- NODE1 Docker network: `dagi-network` (для `nats-box`)

## Scope (NODE1 stack)
- dagi-gateway-node1 (9300)
- dagi-router-node1 (router API)
- dagi-nats-node1 (4222, JetStream enabled)
- crewai-nats-worker
- dagi-memory-service-node1 (8000)
- dagi-qdrant-node1 (6333)
- dagi-postgres (5432)
- dagi-redis-node1 (6379)
- dagi-neo4j-node1 (7474/7687)
- prometheus (9090)
- grafana
- dagi-crawl4ai-node1 (11235)
- control-plane (9200)
- other node1 services as defined in docker-compose.node1.yml

## Safety rules (DO THIS FIRST)
1) Always set project name for NODE1:
   - `export COMPOSE_PROJECT_NAME=dagi_node1`
2) Always use the correct compose file:
   - `-f docker-compose.node1.yml`
3) Never run `docker compose down` without verifying target:
   - `docker compose -f docker-compose.node1.yml ps`
4) If staging exists, it MUST have a different `COMPOSE_PROJECT_NAME` and networks.

## Quick status
- `docker compose -f docker-compose.node1.yml ps`
- `docker compose -f docker-compose.node1.yml logs --tail=80 dagi-gateway-node1 dagi-router-node1 dagi-nats-node1 crewai-nats-worker grafana`

## Standard restart order (most incidents)
1) NATS (foundation)
2) Router (dependency for Gateway routing)
3) Gateway (webhooks)
4) Worker (async jobs)
5) Grafana (observability only)

Commands:
- `docker compose -f docker-compose.node1.yml up -d dagi-nats-node1`
- `docker compose -f docker-compose.node1.yml up -d dagi-router-node1`
- `docker compose -f docker-compose.node1.yml up -d dagi-gateway-node1`
- `docker compose -f docker-compose.node1.yml up -d crewai-nats-worker`
- `docker compose -f docker-compose.node1.yml up -d grafana`

## Incident playbooks

### A) Telegram webhook returns 500 (e.g. /greenfood/telegram/webhook)
Symptoms:
- 500 responses from gateway
- gateway logs show router request failures

Check:
- `docker logs --tail=200 dagi-gateway-node1 | grep -E "webhook|Router request failed|GREENFOOD"`
- `docker compose -f docker-compose.node1.yml ps | grep -E "dagi-gateway-node1|dagi-router-node1"`

Fix:
1) Ensure router is healthy:
   - `docker logs --tail=120 dagi-router-node1`
   - `docker inspect --format '{{json .State.Health}}' dagi-router-node1`
2) Ensure gateway can resolve router (Docker DNS):
   - `docker exec -it dagi-gateway-node1 getent hosts router || true`
3) Restart router + gateway:
   - `docker restart dagi-router-node1`
   - `docker restart dagi-gateway-node1`

Root cause examples:
- router container crash-loop → DNS name `router` not resolvable
- ROUTER_URL points to non-existing host/service in node1 network

### B) Router crash-loop on startup (Pydantic / config errors)
Symptoms:
- router restarting
- traceback in `docker logs dagi-router-node1`

Fix:
1) Read the first error in logs:
   - `docker logs --tail=200 dagi-router-node1`
2) Hotfix then rebuild/recreate if needed:
   - code fix (example previously: `temperature: float = 0.2`)
   - `docker compose -f docker-compose.node1.yml up -d --build --force-recreate dagi-router-node1`

### C) NATS worker shows Subscription failed / NotFoundError
Symptoms:
- worker logs mention `NotFoundError`
- worker cannot subscribe / consume tasks

Check:
- `docker logs --tail=200 crewai-nats-worker`
- `docker logs --tail=200 dagi-nats-node1 | grep -i jetstream`

Fix (JetStream):
1) Ensure JetStream enabled (NATS started with `-js`).
2) Ensure required stream exists (example used on NODE1):
   - Stream: `STREAM_AGENT_RUN`
   - Subjects: `agent.run.>`
3) Using nats-box (inside node1 network):
   - `docker run --rm -it --network <NODE1_NETWORK> natsio/nats-box:latest sh`
   - create stream/consumer as required by worker subjects
4) Restart worker:
   - `docker restart crewai-nats-worker`

### D) Grafana crash-loop due to provisioning alert rule
Symptoms:
- grafana restarting
- logs mention invalid alert rule / relative time range `From: 0s, To: 0s`

Fix:
1) Identify failing rule file:
   - `docker logs --tail=200 grafana`
2) Fix provisioning yaml (example path used on NODE1):
   - `/opt/microdao-daarion/monitoring/grafana/provisioning/alerting/alerts.yml`
   - Ensure rule has valid `relativeTimeRange`
3) Restart grafana:
   - `docker restart grafana`

## Post-recovery verification checklist
1) Core health:
- `docker compose -f docker-compose.node1.yml ps | grep -E "Up|healthy"`
2) Router reachable from gateway:
- `docker exec -it dagi-gateway-node1 getent hosts router`
3) NATS OK:
- `docker logs --tail=80 dagi-nats-node1 | grep -i "JetStream\|Server is ready"`
4) Worker subscribed:
- `docker logs --tail=120 crewai-nats-worker | grep -E "Subscribed|Subscription OK|NotFoundError" || true`
5) GREENFOOD policy sanity:
- рекламне оголошення → ігнор
- пряме питання → відповідь ≤ 3 речень

## Known configuration anchors (update when changed)
- GREENFOOD торговa група: `t.me/+SPm1OV-pDJZhZGFi`
- ROUTER_URL used by gateway: `http://router:8000` (must resolve inside node1 network)
- NATS_URL: `nats://nats:4222`
- JetStream Stream: `STREAM_AGENT_RUN` (`agent.run.>`)
- Grafana alerts provisioning file: `monitoring/grafana/provisioning/alerting/alerts.yml`

## Appendix: common commands
- `docker compose -f docker-compose.node1.yml ps`
- `docker compose -f docker-compose.node1.yml logs -f <service>`
- `docker restart <container>`
- `docker compose -f docker-compose.node1.yml up -d --build --force-recreate <service>`
- `docker system df`